SAS

Mastering Directory Management in SAS: A Guide to Copying Directories

by  • 

Efficient Directory Management in SAS: Copying Directories

Mastering Directory Management in SAS: A Guide to Copying Directories

In data management and processing, efficiently handling directories is crucial. Whether you’re consolidating project files or reorganizing data storage, copying directories from one folder to another can streamline your workflow. In this blog post, we’ll explore a powerful SAS script that automates this task, ensuring you can manage your directories with ease and precision.

Objective

The goal of this SAS script is to copy all directories from a source folder to a target folder. This can be particularly useful for tasks such as archiving, backup, or restructuring data storage. Below, we provide a comprehensive breakdown of the SAS code used to achieve this.

SAS Code for Copying Directories

%let source=/data/projects/2024/Research/Files ;
%let target=/data/projects/2024/Research/Backup ;

data source ;
  infile "dir /b ""&source/"" " pipe truncover;
  input fname $256. ;
run; 

data target ;
  infile "dir /b ""&target/"" " pipe truncover;
  input fname $256. ;
run; 

proc sql noprint ;
  create table newfiles as
    select * from source
    where not (upcase(fname) in (select upcase(fname) from target ));
quit;

data _null_;
   set newfiles ;
  cmd = catx(' ','copy',quote(catx('/',"&source",fname)),quote("&target"));
   infile cmd pipe filevar=cmd end=eof ;
   do while (not eof);
     input;
     put _infile_;
   end;
run;

How It Works

This SAS script performs several key operations to ensure that directories are copied effectively from the source folder to the target folder:

  1. Define Source and Target Folders: The script begins by specifying the source and target folder paths using macro variables. This makes it easy to adjust the paths as needed.
  2. List Directories in Source and Target: Two data steps are used to list all directories in the source and target folders. This is done using the infile statement with a pipe command that executes the dir /b command to retrieve directory names.
  3. Identify New Directories: A PROC SQL step compares the directory names in the source and target folders. It creates a new dataset newfiles containing directories that are present in the source but not in the target folder.
  4. Copy Directories: Finally, a data step constructs and executes a command to copy each new directory from the source to the target folder. The catx function is used to build the copy command, and the infile statement with a pipe executes the command.

Usage Example

To use this script, replace the source and target paths with your desired directories. The script will automatically handle the rest, ensuring that all directories in the source that do not already exist in the target are copied over.

%let source=/path/to/source/folder ;
%let target=/path/to/target/folder ;
/* Run the script as shown above */

Conclusion

Efficiently managing directories is essential for data organization and project management. This SAS script provides a robust solution for copying directories from one folder to another, helping you keep your data well-structured and accessible. By incorporating this script into your workflow, you can automate the process of directory management and focus on more critical aspects of your projects.

Feel free to customize the script to fit your specific needs, and happy coding!

Read more →

SAS

Efficient Directory Management in SAS: A Custom Macro

by  • 

SAS Macro for Directory Management

Efficient Directory Management in SAS: A Custom Macro

Managing directories effectively is crucial for organizing and handling large volumes of files in SAS. In this article, we’ll walk through a custom SAS macro that helps you identify all folders within a specified directory. This macro is particularly useful for managing directory structures in complex projects.

Macro Overview

The get_folders macro is designed to list all folders present in a specified directory. It verifies the existence of the directory, retrieves the names of all items within it, and outputs this information in a readable format. Below is the complete SAS code for this macro:

%macro get_folders(dir);
    /* 
       Macro: get_folders
       Purpose: Identifies all folders available within a specified directory location.
       Source: Custom macro developed for directory management in SAS.
       Date: September 2024
    */
    
    /* CHECK FOR EXISTENCE OF DIRECTORY PATH */
    %if %sysfunc(fileexist(&dir)) %then %do;
    
    /* ASSIGNS THE FILEREF OF MYDIR TO THE DIRECTORY AND OPENS THE DIRECTORY */
    %let filrf=mydir;
    %let rc= %sysfunc(filename(filrf,&dir));
    %let did= %sysfunc(dopen(&filrf));
    
    /* RETURNS THE NUMBER OF MEMBERS IN THE DIRECTORY */
    %let memcnt= %sysfunc(dnum(&did));
    
    %put rc=&rc;
    %put did=&did;
    %put memcnt=&memcnt;
    
    data Dir_Contents;
    length member_name $ 32;
    /* LOOPS THROUGH ENTIRE DIRECTORY */
    %do i = 1 %to &memcnt;
        member_name="%qsysfunc(dread(&did,&i))";
        put 'member_name ' member_name;
        output;
    %end;
    run;
    
    TITLE "CONTENTS OF FOLDER &DIR";
    proc print data=dir_contents;
    run;
    
    /* CLOSES THE DIRECTORY */
    %let rc= %sysfunc(dclose(&did));
    %end;
    %else %do;
    %put ERROR: Folder &dir Not Found;
    %end;
    
    %mend get_folders;
    
    /* Example usage of the macro */
    %get_folders('/example/directory/path');

How It Works

Here’s a step-by-step breakdown of the macro:

  1. Check Directory Existence: The macro first checks if the specified directory exists using the %sysfunc(fileexist) function. If the directory does not exist, an error message is displayed.
  2. File Reference and Directory Opening: If the directory exists, a file reference is assigned, and the directory is opened using the %sysfunc(filename) and %sysfunc(dopen) functions.
  3. Count Directory Members: The macro retrieves the number of items (folders or files) in the directory with %sysfunc(dnum).
  4. Retrieve and Output Folder Names: Using a data step, the macro loops through each item in the directory, retrieves its name with %qsysfunc(dread), and outputs this information to a dataset.
  5. Display Contents: The contents of the dataset are printed using PROC PRINT.
  6. Close the Directory: Finally, the directory is closed with %sysfunc(dclose).

Usage Example

To use this macro, simply call it with the directory path you want to scan:

%get_folders('/example/directory/path');

This will list all folders within the specified directory, making it easier to manage and organize your files.

Conclusion

The get_folders macro is a powerful tool for directory management in SAS. By incorporating this macro into your workflow, you can streamline the process of identifying and organizing folders within your projects. Feel free to modify and adapt the macro to suit your specific needs.

Happy coding!

Read more →

SAS

SAS Functions: SOUNDEX, COMPGED, and Their Alternatives

by  • 

SAS Functions: SOUNDEX, COMPGED, and Their Alternatives

SAS Functions: SOUNDEX, COMPGED, and Their Alternatives

Introduction

In SAS, the SOUNDEX and COMPGED functions are powerful tools for text comparison, particularly when dealing with names or textual data that may have variations. In addition to these, SAS offers other functions like DIFFERENCE and SPEDIS that provide additional ways to measure similarity and distance between strings. This article explores these functions, provides examples, and compares their uses.

The SOUNDEX Function

The SOUNDEX function converts a character string into a phonetic code. This helps in matching names that sound similar but may be spelled differently. The function generates a four-character code based on pronunciation.

Syntax

SOUNDEX(string)

Where string is the character string you want to encode.

Example

data names;
    input name $20.;
    soundex_code = soundex(name);
    datalines;
John
Jon
Smith
Smythe
;
run;

proc print data=names;
run;

In this example, “John” and “Jon” have the same SOUNDEX code, reflecting their similar pronunciation, while “Smith” and “Smythe” have different codes.

The COMPGED Function

The COMPGED function measures the similarity between two strings using the Generalized Edit Distance algorithm. This function is useful for fuzzy matching, especially when dealing with misspelled or slightly varied text.

Syntax

COMPGED(string1, string2)

Where string1 and string2 are the strings to compare.

Example

data comparisons;
    string1 = 'John';
    string2 = 'Jon';
    distance = compged(string1, string2);
    datalines;
;
run;

proc print data=comparisons;
run;

The COMPGED function returns a numerical value representing the edit distance between “John” and “Jon”. Lower values indicate higher similarity.

Alternative Functions

The DIFFERENCE Function

The DIFFERENCE function returns the difference between the SOUNDEX values of two strings. This function is useful for comparing the phonetic similarity of two strings directly.

Syntax

DIFFERENCE(string1, string2)

Where string1 and string2 are the strings to compare.

Example

data soundex_comparison;
    input name1 $20. name2 $20.;
    diff = difference(name1, name2);
    datalines;
John Jon
Smith Smythe
;
run;

proc print data=soundex_comparison;
run;

In this example, the DIFFERENCE function compares the SOUNDEX values of “John” and “Jon”, and “Smith” and “Smythe”. Lower values indicate more similar phonetic representations.

The SPEDIS Function

The SPEDIS function measures the similarity between two strings based on the Soundex encoding and a variant of the Generalized Edit Distance. This function is useful for matching names with variations in spelling.

Syntax

SPEDIS(string1, string2)

Where string1 and string2 are the strings to compare.

Example

data spedisp_comparison;
    string1 = 'John';
    string2 = 'Jon';
    spedis_score = spedis(string1, string2);
    datalines;
;
run;

proc print data=spedisp_comparison;
run;

The SPEDIS function returns a score reflecting the similarity between “John” and “Jon”. A lower score indicates higher similarity, similar to COMPGED, but with a different approach to similarity measurement.

Comparison of Functions

Here’s a quick comparison of these functions:

  • SOUNDEX: Encodes a string into a phonetic code. Useful for phonetic matching, but limited to sounds and does not consider spelling variations.
  • COMPGED: Uses the Generalized Edit Distance algorithm to measure string similarity. Suitable for fuzzy matching with spelling variations.
  • DIFFERENCE: Compares the phonetic similarity of two strings based on their SOUNDEX values. Provides a direct measure of phonetic similarity.
  • SPEDIS: Measures similarity using a combination of Soundex and Edit Distance. Useful for matching names with spelling variations and phonetic differences.

Conclusion

The SOUNDEX and COMPGED functions are valuable tools for text comparison in SAS. By understanding their characteristics and how they compare to other functions like DIFFERENCE and SPEDIS, you can choose the most appropriate method for your specific text matching needs. Each function offers unique advantages depending on the nature of the text data and the type of comparison required.

Read more →

SAS

Using SUPPQUAL for Specifying Natural Key Variables in Define.XML

by  • 

Using SUPPQUAL for Specifying Natural Key Variables in Define.XML

Using SUPPQUAL for Specifying Natural Key Variables in Define.XML

Author: Sarath

Introduction

Define.XML plays a critical role in specifying dataset metadata, particularly in the context of clinical trial data. One important aspect of define.xml is the identification of natural keys, which ensure the uniqueness of records and define the sort order for datasets.

Using SUPPQUAL for Natural Keys

SUPPQUAL, or Supplemental Qualifiers, is a structure used in SDTM/SEND datasets to capture additional attributes related to study data that are not part of the standard domains. In certain cases, the standard SDTM/SEND variables may not be sufficient to fully describe the structure of collected study data. In these cases, SUPPQUAL variables can be utilized as part of the natural key to ensure complete and accurate dataset representation.

Example Scenarios

Consider a scenario where multiple records exist for a single subject in a dataset, with additional details captured in SUPPQUAL. If the standard variables (e.g., USUBJID, VISITNUM, --TESTCD) do not uniquely identify a record, SUPPQUAL variables such as QNAM or QVAL can be incorporated to achieve uniqueness.

Strategies for Incorporating SUPPQUAL Variables

When incorporating SUPPQUAL variables into the natural key, it is important to:

  • Select SUPPQUAL variables that are consistently populated and relevant to the uniqueness of the records.
  • Ensure that the selected SUPPQUAL variables contribute to the overall sort order and are aligned with the study’s data structure.

Documenting SUPPQUAL Natural Keys in Define.XML

Documenting SUPPQUAL variables in define.xml requires careful attention to detail. Here is a step-by-step guide:

  1. Identify the SUPPQUAL variables that need to be included in the natural key.
  2. In the ItemGroupDef section of define.xml, ensure that these variables are listed as part of the Keys attribute.
  3. Provide clear documentation in the ItemDef section, describing the role of each SUPPQUAL variable in the natural key.

Example XML snippet:

<ItemGroupDef OID="IG.SUPPQUAL" Name="SUPPQUAL" Repeating="Yes" IsReferenceData="No" Purpose="Tabulation">
    <!-- Define the key variables -->
    <ItemRef ItemOID="IT.USUBJID" OrderNumber="1" KeySequence="1"/>
    <ItemRef ItemOID="IT.RDOMAIN" OrderNumber="2" KeySequence="2"/>
    <ItemRef ItemOID="IT.IDVARVAL" OrderNumber="3" KeySequence="3"/>
    <ItemRef ItemOID="IT.QNAM" OrderNumber="4" KeySequence="4"/>
</ItemGroupDef>

Conclusion

Using SUPPQUAL variables as part of the natural key in define.xml can be a powerful strategy for ensuring accurate and comprehensive dataset documentation. By carefully selecting and documenting these variables, you can enhance the quality and integrity of your clinical trial data.

References

  1. CDISC Define-XML Specification, Version 2.0. Available at: https://www.cdisc.org/standards/foundational/define-xml
  2. CDISC SDTM Implementation Guide, Version 3.2. Available at: https://www.cdisc.org/standards/foundational/sdtm
  3. FDA Study Data Technical Conformance Guide. Available at: https://www.fda.gov/media/130878/download
  4. SAS Support – Define-XML 2.0: Generating XML Content with SAS. Available at: https://support.sas.com/resources/papers/proceedings15/3273-2015.pdf
  5. How to use SUPPQUAL for specifying natural key variables in define.xml? Available at: https://www.lexjansen.com/phuse/2019/si/SI07.pdf

Read more →

SAS

Optimizing Data Processing with Multi-Threaded Processing in SAS

by  • 

Optimizing Data Processing with Multi-Threaded Processing in SAS

Optimizing Data Processing with Multi-Threaded Processing in SAS

Author: Sarath

Date: August 31, 2024

Introduction

Multi-threaded processing in SAS leverages the parallel processing capabilities of modern CPUs to optimize data handling and analytical tasks. This approach is particularly beneficial when working with large datasets or performing computationally intensive operations. By distributing the workload across multiple threads, SAS can process data more efficiently, leading to reduced runtime and better utilization of available resources.

Why Use Multi-Threaded Processing?

As datasets grow in size and complexity, traditional single-threaded processing can become a bottleneck, leading to longer runtimes and inefficient resource utilization. Multi-threaded processing addresses these issues by:

  • Distributing tasks across multiple CPU cores, allowing for parallel execution.
  • Reducing overall processing time, particularly for tasks like sorting, merging, and data summarization.
  • Enhancing scalability, enabling SAS to handle larger datasets and more complex analyses.
  • Improving resource efficiency by making full use of modern multi-core processors.

Setting Up Multi-Threaded Processing in SAS

To take advantage of multi-threaded processing in SAS, you need to configure your environment correctly. The following steps outline the process:

  1. Enable Multi-Threading: Start by setting the THREADS and CPUCOUNT options. The THREADS option enables multi-threading, while CPUCOUNT specifies the number of CPU cores to use. For example:

    
options threads cpucount=4;

    This configuration enables multi-threaded processing on 4 CPU cores.

  2. Use Multi-Threaded Procedures: SAS offers several procedures optimized for multi-threading, such as SORT, MEANS, SUMMARY, and SQL. Ensure that you’re using these procedures where appropriate.
  3. Optimize Data Structure: Organize your data to minimize dependencies between operations, which can hinder parallel processing. For example, avoid excessive sorting and merging operations, as these can create bottlenecks.
  4. Monitor and Tune Performance: Use SAS options like PROC SQL _METHOD and STIMER to monitor performance and identify potential bottlenecks. Tuning these options can help optimize your multi-threaded processes further.

Example 1: Multi-Threaded Data Sorting

Sorting large datasets is one of the most common tasks that can benefit from multi-threaded processing. The following example demonstrates how to use multi-threading to sort a large dataset: 


options threads cpucount=4;

proc sort data=large_dataset out=sorted_dataset;
    by key_variable;
run;

In this example, the sorting operation is distributed across 4 CPU cores, significantly reducing the time required to sort the dataset.

Example 2: Multi-Threaded Summary Statistics

Calculating summary statistics on large datasets can be time-consuming. Here’s how multi-threading can speed up the process using the PROC MEANS procedure: 


options threads cpucount=6;

proc means data=large_dataset mean stddev maxdec=2;
    var numeric_variable;
    class categorical_variable;
run;

This example uses 6 CPU cores to calculate mean, standard deviation, and other statistics for a large dataset. The PROC MEANS procedure is optimized for multi-threading, making it well-suited for this type of task.

Example 3: Multi-Threaded SQL Processing

SQL operations, such as joining large tables, can be optimized using multi-threaded processing. Here’s an example: 


options threads cpucount=8;

proc sql;
    create table joined_dataset as
    select a.*, b.variable2
    from large_table1 as a
    inner join large_table2 as b
    on a.key = b.key;
quit;

In this example, the join operation between two large tables is distributed across 8 CPU cores, reducing the time required to complete the process.

Best Practices for Multi-Threaded Processing

To get the most out of multi-threaded processing in SAS, consider the following best practices:

  • Match CPU Count to Workload: Use the CPUCOUNT option to specify the appropriate number of CPU cores based on your server’s capabilities and the complexity of the task.
  • Minimize I/O Bottlenecks: Ensure that your storage system can handle the increased I/O demands of multi-threaded processing, particularly when working with large datasets.
  • Balance Load: Distribute your workload evenly across threads to avoid overloading individual cores. Consider breaking down large tasks into smaller, parallelizable components.
  • Test and Tune: Regularly monitor performance using SAS options and system tools, and adjust your settings as needed to optimize performance.

Challenges and Considerations

While multi-threaded processing offers significant benefits, it also presents some challenges:

  • Complexity: Configuring and optimizing multi-threaded processing can be complex, particularly in environments with limited resources or when dealing with highly interdependent tasks.
  • Resource Contention: Running too many threads simultaneously can lead to resource contention, where multiple processes compete for the same CPU or I/O resources, potentially reducing overall performance.
  • Hardware Limitations: The effectiveness of multi-threaded processing is heavily dependent on the underlying hardware. Systems with fewer CPU cores or slower I/O subsystems may see limited benefits.

Conclusion

Multi-threaded processing in SAS is a powerful technique for optimizing data processing, particularly for large and complex datasets. By leveraging the parallel processing capabilities of modern CPUs, you can achieve significant performance improvements, reducing runtime and improving resource utilization. However, careful configuration and monitoring are essential to maximize the benefits and avoid potential challenges. By following best practices and continuously tuning your approach, you can make the most of multi-threaded processing in your SAS environment.

© 2024 Sarath. All rights reserved.

Read more →