Using SAS to Read and Write EXCEL Workbooks Which Are · PDF fileUsing SAS to Read and Write...

Using SAS to Read and Write

EXCEL Workbooks Which Are

Not Set Up In Columns

SCSUG Educational Forum

Presented by: Carl Raish

Nov 5 - 6, 2012 Houston, Texas

2

Agenda

1. Background Information on ERCOT Emergency Response Service

2. Reading Excel ERS Submission Forms

3. Writing Processed Excel ERS Submission Forms

4. Additional Tips

5. Q & A

3

North American Interconnected Grids

4

Electric Grid Operations

• Grid operations

– ERCOT is one of 10 North

American independent system

operators (ISOs)

• ISOs serve 67% of U.S. population

• ERCOT directs traffic on the grid to maintain reliability

– ‘Air traffic controller’ of the electric supply

• Coordinates scheduling of power by market participants

• Analyzes grid conditions continuously in real-time

• Dispatches generation to ensure power production matches Load at all times

• Secures extra generation capacity to meet reliability requirements

• Coordinates planned outages of generators and transmission lines

• Relieves transmission system congestion – Nodal Market Dec 2010

• Coordinates emergency actions & recovery

• Operates markets to meet regional energy & capacity requirements not met

through bilateral arrangements

Not

pictured:

New

Brunswick

System

Operator

5

Emergency Response Service (ERS)

• Provide ERCOT Operations with an additional emergency tool to

lessen the likelihood of involuntary firm Load shedding (a.k.a. rolling

blackouts)

• ERCOT started procuring Feb. 1, 2008

• ERS is provided by

– Loads (customers) willing to interrupt during an electric

grid emergency in exchange for a payment

– Distributed generators not participating in ERCOT markets

• Last resort prior to firm Load shedding (rotating outages)

• Deployed ONLY in the late stages of a grid emergency

“Controlled interruption of prepared customers vs.

uncontrolled interruption of unprepared customers”

http://www.fundraw.com/clipart/clip-art/00003250/Compact-Fluorescent-Lightbulb/

6

Excel Form Description

• ERCOT uses an Excel form

for resource identification

and offer submission

• Three different tab types

– Identification tab has

submitter information

– Input data is part of an

Excel named range,

id_input_area

– Range has a column of

descriptive information,

merged cells and input

cells

7


• ‘Alt’ tab has site level information

– ERID Number assigned and filled in by ERCOT when processing the form to return to

submitter

– Tab name is pre-defined

8


• Resource tabs

– Specific resource information provided by submitters

– ERID Number assigned and filled in by ERCOT when processing the form to return to

submitter

– Tab names arbitrarily assigned by submitter

9

Reading the Excel Submission Form

• Code to generate a SAS dataset of Excel file names in the input

folder and a count of the number of files

10


• Code to begin reading the workbooks

– Macro do-loop to cycle through all file names

– Macro variable &filename. to specify the next workbook being read

– Libname option ‘mixed = yes’ treats all fields as character, ‘header = no’ treats first row as data instead of

variable names

– SAS dataset tabnames contains a list of all tabs (end with $) and named ranges within the workbook

11


• Tabnames dataset just created

– This example workbook has many tab names with special characters

– If tab name has space, &, parenthesis SAS adds a single quote at beginning and end

of tab name

– If tab name contains a single quote, SAS makes it two single quotes

12


• Executing the libname statement works, but, if you double click on

a member name containing a single quote in the explorer window

to view it, SAS issues and error

• Proc contents on the library also results in an error

13


• Data step creates tab_names_1 and bad_tab_names

– Translates tab names that originally contained a single quote back to a single quote

– Tab names with single quote and ‘&’ written to bad_tab_names

– Excludes tabs with reserved tab names

14


• Data set tab_names_1 with modified tab names

• Data set bad_tab_names for tab names with single quotes and

ampersands

15


• This part of the code checks for missing Identification tabs and

writes error message to log

• Reads form version number on first row of Identification tab using

Excel libname engine

16


• This part of the code inputs data in the id_input named range and merges with version number

• Each Excel row is treated as an observation, data is in variable named f2

• Logic skips rows with no data to input and retains all variables until last row is read

• Creates a macro variable, &numtabs., for the number of tabs to be read

17


• This part of the code creates the sheet command for a proc import as the macro variable &sheet.

• If the tab name has a single quote, a double quote is added at the beginning and end of the tab

name in the sheet command

• If not already included, the sheet command has single quotes added before and after the tab name

18


• This part of the code parses and converts the imported data using the f1 – f27 variables and the _n_ variable to

specify the row and column where the specific data is located

• The input function is used to convert to numeric variables when necessary

• Input variables are retained until the last non-blank row on the tab is processed

19


• This part of the code merges identification and resource data and creates combine_id_resource

• Any resources submitted with incorrect version numbers are output to bad_version

20


• The final part of the code accumulates resource, bad version and bad tab name data across all

workbooks.

21

Writing the Processed Excel Submission Form

• After forms are successfully read the submitted data is run through validation and historical usage data analysis.

• SAS datasets with these results and are summarized in tabs on the processed workbooks returned to the

submitters.

• The returned workbooks are used as the basis for submitting offers back to ERCOT to provide the service.

• Note the options, ‘noxwait’ and ‘noxsync’ which are needed for subsequent DDE processing

22


• The code below begins the process of creating tab names

– Tab name is based on the resource name used by the submitter

– Special characters are removed and other formatting is done on the tab names

– The tab name is ‘upcased’ because Excel does not treat tab names as case sensitive

23


• The code below finishes the process of creating tab names

– Tab names are modified with a sequence number as necessary to ensure no duplication exists

– The SAS dataset qse_erid_list is created containing a list of all resource tabs that need to be written to the workbooks

– Multiple workbooks are created, one for each submitter (QSE), so the macro variable, &numqses. is created as a count

of the submitters with workbooks being processed

24


• The remainder of the code is embedded in a macro do-loop which produces the workbook for each submitter

– Blank submission form templates are used to create each workbook

– The macro variable, &numerids., is set up as the number of tabs to be written for the submitter being processed

– Depending on the form submitted one of two templates is used , ID_Template, or ID_Alt_Template, and specified in the

macro variable &id_template.

25


• The code below issues system commands to make a copy of the ID_Template being used to avoid the risk of

corrupting the original template

• Then proc export is used to create place-holder tabs on the copied template for each resource tab that is needed for

the submitter being processed

• The SAS dataset, ‘fill’, is used with the proc export because a non-empty dataset is required

26


• The code below launches Excel using an ‘X’ command … the path specified is for the Excel executable

• The filename, ‘ddecmds’ is set up to enable SAS to send DDE commands to Excel

• The three data steps issue DDE commands to open three existing Excel workbooks (sleeps help Excel keep up with SAS)

– Input data.xlsx is the workbook to which SAS will write data to populate the workbook being created

– Rtab_template.xlsx is a template for resource tabs

– &ID_Template_copy.xlsx is a template for the Identification tab and an Alt tab, if needed in the submitters workbook

27


• A portion of ‘Input data.xlsx’ is shown below with Excel’s name manager information

– Identification tab data is written by SAS to the ‘id_data’ tab to a range named ‘id_inputs’

– Resource level data is written to the ‘load_data’ tab to a range named ‘load_inputs’

– Site level data is written to the ‘site_data’ tab to a range named ‘site_inputs’ (up to 1,000 rows)

28


• A portion of ‘id_alt_template.xlsx’ is shown below

– Cells to be populated are actually links to the second row of the ‘id_data’ tab of ‘input data.xlsx’

– For cosmetic purposes the links are Excel ‘if’ statements which populate the cell with a blank or actual

data

– Since SAS has opened both workbooks, the links in ‘id_alt_template.xlsx’ are resolved based on

whatever SAS writes to ‘input data.xlsx’

29


• A portion of ‘rtab_template.xlsx’ is shown below

– Cells in rows 7 – 1000 are actually links to the ‘site_data’ tab of ‘input data.xlsx’

– For cosmetic purposes the links are Excel ‘if’ statements which populate the cell with a blank or actual data

– Since SAS has opened both workbooks, the links in ‘rtab_template.xlsx’ are resolved based on whatever SAS writes to

‘input data.xlsx’

30


• A portion of ‘rtab_template.xlsx’ is shown below

– Cells in rows 1 – 6 are actually links to the second row of the ‘load_data’ tab of ‘input data.xlsx’

– For cosmetic purposes the links are Excel ‘if’ statements which populate the cell with a blank or actual data

– Since SAS has opened both workbooks, the links in ‘rtab_template.xlsx’ are resolved based on whatever SAS writes to

‘input data.xlsx’

– The baseline cell in C4 is a link back to cell E2 on the ‘load_data’ tab

– The baseline cell is also associated with a drop down list set up in cells AD1 – AD4 on ‘Rtab_template.xlsx’ … these

four cells are also links to four cells in row 2 of the ‘load_data’ tab … with this setup SAS can control the options

appearing in the drop-down list for the cell as well as the cell itself

– This is how ERCOT communicates the baseline options available back to the submitter

31


• The code below writes the data for the submitter being processed out to the ‘id_inputs’ named range on the ‘input

data.xlsx’ workbook

• The output uses the Excel libname engine … which outputs variable names as titles on the first row of the range

and actual data in the second row … note: ‘id_inputs’ range must be blank before output occurs

• The links set up on the opened template workbook are updated as SAS writes out the data

• A drop-down list on the ‘Identification’ tab is populated using the sub_type1 and sub_type2 variables

32


• The code below issues DDE commands to activate various tabs on the ‘id_alt_template_copy.xlsx’ workbook and

then run selected Excel macros stored in ‘personal.xls’ to affect the activated tab

33


• The actual Excel macro code for the three modules is shown below … the code was created by recording

keystrokes (I’m not a VBA programmer)

• The copy_paste_values_id_tab macro turns the links on the Identification tab into actual values … its done with

two sets of instructions, one to deal with version number and drop-down cell values, and one to deal with the

submitter information

• The expand_cols macro does an auto fit on all columns on the active tab

• The clear_lines macro eliminates cell outlines, which, in this application, mysteriously appear on tabs produced

with proc export

34


• The code below writes the data for the submitter being processed out to the ‘load_inputs’ and ‘site_inputs’ named

ranges on the ‘input data.xlsx’ workbook

• The macro do-loop cycles through all the tab names that need to be written

• The output uses the Excel libname engine … which outputs variable names as titles on the first row of the range

and actual data in the second row … note: ‘id_inputs’ range must be blank before output occurs

• The links set up on the opened template workbook are updated as SAS writes out the data

35


• The code below issues a DDE command to activate the placeholder resource tab created earlier by proc export and

then runs an Excel macro to copy the contents of the Rtab_template.xlsx to that activated tab on the

‘ID_Alt_Template_Copy.xlsx’ workbook

36


• The code then runs a copy-paste –values macro similar to the one shown earlier to eliminate the links on the

copied tab

• The code also runs one of a set of macros to delete unneeded rows at the bottom of the resource tab

37


• When all tabs for a submitter have been created, the code then issues DDE commands to save and close all three of the open

Excel workbooks

– The saves occur in reverse of the order the workbooks were opened, so the first save.as command saves the workbook to be

returned to the submitter with a filename set up for our file distribution system

– The second two saves are included to allow Excel to close down without any user intervention.

• The quit() command shuts down Excel

• The last data step issues system commands to delete unneeded workbooks created during the outermost macro do-loop to

set the stage for another iteration or the end of the SAS session

38

Additional Tips

• Our initial code did all input and output to Excel using DDE commands; use of the Excel libname engine has greatly improved the speed, accuracy and reliability. Our latest iteration of the code has eliminated as much as possible use of DDE commands.

• We have always run a verify step after writing to Excel workbooks with SAS. Though communication errors largely have been eliminated with this version of the code, we still, from time to time, find other ‘logic’ errors when we read the workbook back in and run proc compare to compare the resulting SAS dataset to the one used for the output code.

• The code is fun to watch because both SAS and Excel are actively doing things on your desktop (we run interactively); it’s helpful to keep an eye on the Excel, if the code appears to be hung up. In many cases it’s because Excel has encountered a condition, usually an error, requiring user input via a dialog box. In these cases, you need to cancel submitted code in SAS and also shut down Excel manually.

• Not surprisingly, diagnosing errors, particularly distinguishing between SAS errors and Excel macro errors is a bit mysterious … the error messages are usually very uninformative.

• We use an Excel add-in to clear excess formats and it usually greatly reduces the size of files.

http://www.microsoft.com/en-us/download/details.aspx?id=21649

• The paper has been updated … if you’re interested in the latest version, give me a business card and I’ll email it to you




39

Q & A

Date post:	07-Mar-2018
Category:	Documents
Upload:	lamthuan
View:	228 times
Download:	4 times

Using SAS to Read and Write EXCEL Workbooks Which Are · PDF fileUsing SAS to Read and Write...

Documents