Introduction to the LOGPARSE Macro GASUG Atlanta, GA July 18 th, 2007.

Introduction to the LOGPARSE MacroIntroduction to the LOGPARSE Macro

GASUG Atlanta, GA

July 18th, 2007

© Copyright ERMAS Corporation 2007 LOGPARSE Page 2

Integration Consulting Marketing Automation and Optimization Marketing Campaign Management, Fulfillment, and

Response Analysis Risk & Analytics Analytical Modeling Data Warehousing Business Intelligence Solutions CRM Database Solutions Web Development Services

Our ServicesOur ServicesOur ServicesOur Services


Our Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + InternationalOur Locations: 5 Main Hubs + International

5. UK & Europe

We have completed projects in Korea, Singapore, Malaysia, United Kingdom, Netherlands, Sweden, Germany, and Peru.

3. New York

2. Charlotte

4. Dallas 1. Atlanta

International

Corporate Location:Corporate Location: 4080 McGinnis Ferry Rd4080 McGinnis Ferry Rd

Bldg 100, Suite 101Bldg 100, Suite 101Alpharetta, GA 30005Alpharetta, GA 30005

Corporate Location:Corporate Location: 4080 McGinnis Ferry Rd4080 McGinnis Ferry Rd

Bldg 100, Suite 101Bldg 100, Suite 101Alpharetta, GA 30005Alpharetta, GA 30005


Our Atlanta LabOur Atlanta LabOur Atlanta LabOur Atlanta Lab

Always Seeking TalentAlways Seeking TalentAlways Seeking TalentAlways Seeking Talent

Send inquiries to:

[email protected]


Today’s TopicToday’s TopicToday’s TopicToday’s Topic

Introduce the LOGPARSE macro to capture FULLSTIMER statistics

Demonstrate a technique for profiling very large SAS applications

Provide code for creating a Performance Dashboard

…Capture Tuning Statistics

…Not Tune SAS Applications


LOGPARSE MACROLOGPARSE MACROLOGPARSE MACROLOGPARSE MACRO

Provided by SAS, experimential with SAS 9.1

Captures FULLSTIMER statistics from the SAS Log and stores them in a SAS Dataset• For every SAS Step

Available on the SAS Scalability & Performance Community Site

www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html

www.support.sas.com/rnd/scalability/tools/fullstim/logparse.zip


What’s Inside logparse.zipWhat’s Inside logparse.zipWhat’s Inside logparse.zipWhat’s Inside logparse.zip

Four files in logparse.zip

%LOGPARSE()• Extracts the statistics from the SAS LOG(s)

%PASSINFO()• Optional, provides session information to the %LOGPARSE()

macro.

%MVSNAME()• Called by %PASSINFO() for MVS only

Readme.txt• A good place to start

logparse.sas passinfo.sas mvsname.sas readme.txt


Getting Started with %LOGPARSEGetting Started with %LOGPARSEGetting Started with %LOGPARSEGetting Started with %LOGPARSE

OPTIONS FULLSTIMER; as the first line for the program you wish to collect statistics

%PASSINFO; optionally on the second line

Run the program to generate the log(s)

Create another program to use %LOGPARSE and collect the statistics

%include logparse; %logparse( yourProgram.log, yourPerfData, OTH ); proc print data=yourPerfData; run;


Big PictureBig PictureBig PictureBig Picture

LOGFILE(S)

YOURLOGPARSEPROGRAM

StatisticsDataset

%LOGPARSE( saslog, outds, system, pdsloc, append=no )


%LOGPARSE Syntax%LOGPARSE Syntax%LOGPARSE Syntax%LOGPARSE Syntax


saslog is the name of the SAS LOG to parse, requires the .log extension

outds is the output SAS dataset for the parsed LOG results

system is the 3-CHAR code for the system on which %LOGPARSE is run. If not specified, the default value is the system %LOGPARSE is executing on.

• MVS - z/OS, OS/390, or MVS• ALP - OpenVMS Alpha• VMS - VMS• OTH - All other OSs, like Windows

pdsloc identifies the PDS member where the SAS LOG is stored on MVS system. See the readme.txt file.

append=no specifies whether to append the results to an existing file or overwrite

• NO – creates a new SAS file based on the outds parameter• YES – appends the output to file specified by outds or creates a new file if

the file does not exist


The STATISTICS DatasetThe STATISTICS DatasetThe STATISTICS DatasetThe STATISTICS Dataset

The statistics captured by %LOGPARSE are dependent on the NOTES: produced by FULLSTIMER and vary by system.

Important non-statistics columns to understand• logfile – name of the log file

• stepname – DATA, SORT, FORMAT, PRINTO, SAS, initialization.

NOTE: DATA statement used (Total process time):

NOTE: The SAS System used:

• stepcnt – sequence number applied to the order of the step within the log.

Windows Statistics Columns• realtime• usertime• systemtime

• obsin• obsout• varsout

• memused


Big PictureBig PictureBig PictureBig Picture

LOGFILE(S)

YOURLOGPARSEPROGRAM

StatisticsDataset



ExampleExampleExampleExampleoptions fullstimer;

%let defaultPath=C:\Documents and Settings\Administrator\My Documents\My SAS Code;

%include "&defaultPath\timerUtils\passinfo.sas";%passinfo;

%include "&defaultPath\census\ipumsFormats.inc";%include "&defaultPath\census\convertToSAS.inc";%include "&defaultPath\census\splitData.inc";%include "&defaultPath\census\subsetMetro.inc";%include "&defaultPath\census\hhRollup.inc";

libname ipumsdat "&defaultPath\census";/*----------------------------------------------------------------------------+ Main Program+----------------------------------------------------------------------------*/%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);%splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);%subsetMetro("Atlanta", 2003 2004 2005 2006);%hhRollup(2003 2004 2005 2006)


The Bad NewsThe Bad NewsThe Bad NewsThe Bad News

A dataset of statistics is produced – map that back to your program/log…. OR

stepcnt stepname realtime usertime systime cputime obsin obsout varsout memused0 initialization 00:00.5 . . 00:00.2 . . . .1 DATA 00:00.0 00:00.0 00:00.0 . . . . 1612 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 133 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 134 OPTIONS 00:00.0 00:00.0 00:00.0 . . . . 135 FORMAT 00:00.7 00:00.0 00:00.0 . . . . 1826 FORMAT 00:00.0 00:00.0 00:00.0 . . 3052 21 887 DATA 01:28.6 00:32.1 00:03.5 . 3052 848875 220 3338 DATA 00:55.0 00:15.8 00:03.7 . 848875 848875 880 5069 DATA 00:01.6 00:01.3 00:00.2 . 1564 1564 9 203

10 SORT 00:00.0 00:00.0 00:00.0 . 1564 1564 9 28211 DATA 00:01.4 00:01.3 00:00.2 . 1651 1651 9 20312 SORT 00:00.0 00:00.0 00:00.0 . 1651 1651 9 28213 DATA 00:01.5 00:01.3 00:00.2 . 1922 1922 9 20314 SORT 00:00.0 00:00.0 00:00.0 . 1922 1922 9 31615 DATA 00:01.4 00:01.2 00:00.1 . 2427 2427 9 20316 SORT 00:00.0 00:00.0 00:00.0 . 2427 2427 9 36717 DATA 00:00.2 00:00.0 00:00.0 . 1564 600 10 20918 DATA 00:00.0 00:00.0 00:00.0 . 1651 619 10 20919 DATA 00:00.0 00:00.0 00:00.0 . 1922 742 10 20920 DATA 00:00.0 00:00.0 00:00.0 . 2427 921 10 20921 SAS 02:31.3 00:53.0 00:08.3 . . . . 3382


The Good NewsThe Good NewsThe Good NewsThe Good News


Use PRINTTO to Write Separate LogsUse PRINTTO to Write Separate LogsUse PRINTTO to Write Separate LogsUse PRINTTO to Write Separate Logs

filename fmtLog "&defaultPath\census\ipumsFormtas.timingDemo.log";filename cvrtLog "&defaultPath\census\convertToSAS.timingDemo.log";filename splitLog "&defaultPath\census\splitData.timingDemo.log";filename substLog "&defaultPath\census\subsetMetro.timingDemo.log";filename hhrupLog "&defaultPath\census\hhRollup.timingDemo.log";/*----------------------------------------------------------------------------+ Main Program+----------------------------------------------------------------------------*/proc printto log=fmtLog new; run;%ipumsFormats(ipumsdat.fmmiller_ermas_com_001);

proc format cntlin=ipumsdat.fmmiller_ermas_com_001; run;

proc printto log=cvrtLog new; run;%convertToSAS(asciidat, ipumsdat.mmiller_ermas_com_001);

proc printto log=splitLog new; run; %splitData(ipumsdat, ipumsdat.mmiller_ermas_com_001);

proc printto log=substLog new; run;%subsetMetro("Atlanta", 2003 2004 2005 2006);

proc printto log=hhrupLog new; run;%hhRollup(2003 2004 2005 2006);run; proc printto; run;

MODULE.PROGRAM.LOG

Use PROC PRINTTO to WriteSeparate Log Files


Use our toolkit package to create a visual performance dashboard (WINDOWS ONLY)

Name each log in the format module.program.log for each subset of the program to be analyzed

Use the perfDash.sas program to analyze the logs

ERMAS Performance ToolkitERMAS Performance ToolkitERMAS Performance ToolkitERMAS Performance Toolkit

/*----------------------------------------------------------------------------+

User Parameters

rootDir - location of the SAS utilities, see installation instructions below

logDir - directory where the SAS LOGs are located

program - name of the SAS program for which to generate the dashboard

+----------------------------------------------------------------------------*/

%let rootDir = %str(C:\GASUG2007);

%let logDir = %str(C:\GASUG2007\census);

%let program = timingDemo; /* case sensitive */


How to get the Ermas Performance Toolkit

• GASUG2007.ZIP file contains four directories: timerUtils – location of the perfDash.sas program census – log files to analyze myLibrary – utility library utils – more utilities.

SAS Scalability & Performance Community

Great SUGI Paper on the LOGPARSE Macro

ResourcesResourcesResourcesResources

www.support.sas.com/rnd/scalability/tools/fullstim/fullstim.html

http://www.ermas.com/downloads.htm

http://www2.sas.com/proceedings/sugi30/219-30.pdf

Date post:	16-Jan-2016
Category:	Documents
Upload:	logan-ferguson
View:	219 times
Download:	0 times