How to download and convert CSV files for use in SAS

This post was kindly contributed by The SAS Dummy - go there to comment and to read the full post.

In his recent article Perceptions of probability, Rick Wicklin explores how vague statements about “likeliness” translate into probabilities that we can express numerically. It’s a fun, informative post — I recommend it! You’ll “Almost Certainly” enjoy it.

To prepare the article, Rick first had to download the source data from the study he cited. The data was shared as a CSV file on GitHub. Rick also had to rename the variables (column names) from the data table so that they are easier to code within SAS. Traditionally, SAS variable names must adhere to a few common programming rules: they must be alphanumeric, begin with a letter, and contain no spaces or special characters. The complete rules are documented in the SAS Language Reference Guide. In his final program, Rick includes a DATA step with everything you need to reproduce his nifty plots, so you don’t need to do any additional data prep.

Rick challenged me to devise a method to pull the data directly from GitHub and prep it in SAS. To solve his requirements, the SAS program must connect to GitHub and download the CSV file, import the data into SAS, change the column names to comply with SAS naming rules, but retain the original column names as descriptive labels.

Step 1. Download the data file with PROC HTTP

In 2012 I first shared this method for reading data from a cloud service like DropBox and GitHub. It’s still my favorite technique for reading data from the Internet. You’ll find lots of papers and examples that use FILENAME URL for the same job in fewer lines of code, but PROC HTTP is more robust. It runs faster, and it allows you to separate the step of fetching the file from the subsequent steps of processing that file.

You can see the contents of the CSV file at this friendly URL: https://github.com/zonination/perceptions/blob/master/probly.csv. But that’s not the URL that I need for PROC HTTP or any programmatic access. To download the file via a script, I need the “Raw” file URL, which I can access via the Raw button on the GitHub page.

In this case, that’s https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv. Here’s the PROC HTTP step to download the CSV file into a temporary fileref.

/* Fetch the file from the web site */
filename probly temp;
proc http
 url="https://raw.githubusercontent.com/zonination/perceptions/master/probly.csv"
 method="GET"
 out=probly;
run;

A note for SAS University Edition users: this step won’t work for you, as the free software does not support access to secure (HTTPS) sites. You’ll have to manually download the file via your browser and then continue with the remaining steps.

Step 2. Import the data into SAS with PROC IMPORT

SAS can process data with nonstandard variable names, including names that contain spaces and special characters. You simply have to use the VALIDVARNAME= system option to put SAS into the right mode (oops, almost wrote “mood” there, but it’s sort of the same thing).

With VALIDVARNAME=ANY in effect, you can use variable names like “My special name” or “#CoolestNameEver”, but you have to enclose those names in a special literal syntax that most SAS programmers find to be inconvenient. Or maybe even a “crime against nature.” (Or ‘crime against nature’n.)

For this step, I’ll set VALIDVARNAME=ANY to allow PROC IMPORT to retain the original column names from the CSV file. The same trick would work if I was importing from an Excel file, or any other data source that was a little more liberal in its naming rules.

/* Tell SAS to allow "nonstandard" names */
options validvarname=any;
 
/* import to a SAS data set */
proc import
  file=probly
  out=work.probly replace
  dbms=csv;
run;

Step 3. Create RENAME and LABEL statements with PROC SQL

This is one of my favorite SAS tricks. You can use PROC SQL SELECT INTO to create SAS programming statements for you, based on the data you’re processing. Using this technique, I can build the parts of the LABEL statement and the RENAME statement dynamically, without knowing the variable names ahead of time.

The LABEL statement is simple. I’m going to build a series of assignments that look like this:

  'original variable name'n = 'original variable name'

I used the SELECT INTO clause to build a label assignment for each variable name. I used the CAT function to assemble the label assignment piece-by-piece, including the special literal syntax, the variable name, the assignment operator, and the label value within quotes. I’m fetching the variable names from SASHELP.VCOLUMN, one of the built-in dictionary tables that SAS provides to surface table and column metadata.

  select cat("'" , trim(name), "'n", "=", "'", trim(name), "'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";

Here’s part of the value of &labelStmt:

'Almost Certainly'n='Almost Certainly' 
'Highly Likely'n='Highly Likely' 
'Very Good Chance'n='Very Good Chance' 
'Probable'n='Probable' 
'Likely'n='Likely' 
'Probably'n='Probably' 
'We Believe'n='We Believe'

Because I have to calculate a new valid variable name, building the RENAME statement is a little trickier. For this specific data source that’s easy, because the only SAS “rule” that these column names violate is the ban on space characters. I can create a new name by using the COMPRESS function to remove the spaces. To be a little safer, I used the “kn” modifier on the COMPRESS function to keep only English letters, numbers, and underscores. That should cover all cases except for variable names that are too long (greater than 32 characters) or that begin with a number (or that don’t contain any valid characters to begin with).

Some of the column names are one-word names that are already valid. If I include those in the RENAME statement, SAS will generate an error (you cannot “rename” a variable to its current name). I used the NVALID function to test whether the variable name contains characters that need attention, and process only those that aren’t valid to begin with.

Here’s part of the value of &renameStmt:

'Almost Certainly'n=AlmostCertainly 
'Highly Likely'n=HighlyLikely 
'Very Good Chance'n=VeryGoodChance 
'We Believe'n=WeBelieve

Here’s the complete label/rename segment of the program:

/* Generate new names to comply with SAS rules.                          */
/* Assumes names contain spaces, and can fix with COMPRESS               */
/* Other deviations (like special chars, names that start with a number) */
/* would need different adjustments                                      */
/* NVALID() function can check that a name is a valid V7 name           */
proc sql noprint;
 
  /* retain original names as labels */
  select cat("'", trim(name), "'n", "=", "'", trim(name), "'") 
     into :labelStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK";
 
  select cat("'", trim(name), "'n", "=", compress(name,,'kn')) 
     into :renameStmt separated by ' '  
  from sashelp.vcolumn where memname="PROBLY" and libname="WORK"
  /* exclude those varnames that are already valid */
  AND not NVALID(trim(name),'V7');
quit;

Step 4. Modify the data set with new names and labels using PROC DATASETS

With the body of the LABEL and RENAME statements built, it’s time to plug them into a PROC DATASETS step. PROC DATASETS can change data set attributes such as variable names, labels, and formats without requiring a complete rewrite of the data — it’s a very efficient operation.

I include the LABEL statement first, since it references the original variable names. Then I include the RENAME statement, which changes the variable names to their new V7-compliant values.

Finally, I reset the VALIDVARNAME= option to the normal V7 sanity. (Unless you’re running in SAS Enterprise Guide, in which case the option is already set to ANY by default. Check this blog post for a less disruptive method of setting/restoring options.)

proc datasets lib=work nolist ;
  modify probly / memtype=data;
  label &labelStmt.;
  rename &renameStmt.;
  /* optional: report on the var names/labels */
  contents data=probly nodetails;
quit;
 
/* reset back to the old rules */
options validvarname=v7;

Here’s the CONTENTS output from the PROC DATASETS step, which shows the final variable attributes. I now have easy-to-code variable names, and they still have their descriptive labels. My data dictionary dreams are coming true!