MXG Tools and Usage

transcript

MXG Tools and Usage

Chuck Hopf

Agenda – Building the PDB

VMXGALOC\VGETALOC UTILBLDP BLDSMPDB UTILWORK READDB2 VMXGSUM

Agenda - Analysis

ANALID ANALGRID VMXGPRNT VMXGFIND VMXGSRCH ANALCNCR ANALCAPD

VMXGALOC – Pseudo GDGs

ASCII ONLY – Windows or LINUX Allocates directories and assigns

LIBNAMEs using a date based structure Allows you to ‘keep’ as many

generations as you wish of each type of data – daily, weekly, trend, spin, db2acct, cicstran, monthly

VMXGALOC – Parameters

BASEDIR=C:\MXG – where do you want to put the directories? Can be any valid location so long as it is connected to the system executing SAS/MXG

FORCEDAY= used in the event of a rerun or the need to report for some given day - can be a SAS date value – 27AUG12 or a relative value – today()-2

VMXGALOC - Parameters

WEEKSTRT=MON – the day of the week on which your week starts. MON is the MXG default

Number of generations -– WEEKKEEP=12 – keep 12 weeks– DAYSKEEP=14 – keep 14 days– MNTHKEEP=15 – keep 15 months– CICSKEEP=15 – keep 15 days of CICSTRAN– DB2KEEP=14 – keep 14 days of DB2ACCT

VMXGALOC Parameters

RUNWTD=NO – change to yes to run week to date logic

RUNMTD=NO – change to yes to run month to date logic

TRENDING=daily or weekly – how often to update TREND databases

READONLY=yes/no if NO the aging of old generations is suppressed

CLEARALL=YES clears the normal default LIBNAMEs from AUTOEXEC

VMXGALOC Parameters

DATEFMT= can be any valid DATE format…– date date7 date9– mmddyy6 8 10– ddmmyy 6 8 10– yymmdd 6 8 10– julian 5 7

If the format (mmddyy8. for example) contains / then the equivalent mmddyyd8. is substituted

An invalid datefmt will result in an ABEND

VGETALOC

VGETALOC will fetch a ‘range’ of dates for daily/weekly/monthly PDBs and pass that information to VMXGSET so that you could say something like:– %vgetaloc(getdaterange=12jul12 23jul12,

typeofdata=daily,basedir=c:\mxg); data jobs; set %vmxgset(dataset=jobs);

VGETALOC

Can only be used on ASCII systems where VMXGALOC has been used to create pseudo-GDGs

If a date in the date range does not exist it is skipped

VGETALOC - Parameters

GETDATERANGE – the range of dates in the form of SAS date values to be searched

TYPEOFDATA – DAILY WEEKLY MONTHLY?

DATEFORMAT – the DATE format used in VMXGALOC

BASEDIR – the directory as specified in VMXGALOC

UTILBLDP Normally the code to read an SMF record is:

– %INCLUDE SOURCLIB(TYPE30); And to read two types you might code:

– %INCLUDE SOURCLIB(TYPE30);– %INCLUDE SOURCLIB(TYPE1415);

But that would cause two passes of the SMF dataset which can be very large and make this an expensive and time consuming process.

With UTILBLDP this becomes:– %UTILBLDP(USERADD=30 1415,

BUILDPDB=NO,SORTOUT=NO,OUTFILE=INSTREAM);– %INCLUDE INSTREAM;

UTILBLDP

UTILBLDP is a macro designed to simplify adding records to the normal MXG PDB (performance data base.) The coding in exits is not difficult if you understand it all but can be arcane to the uninitiated.

It can also be used to read multiple kinds of SMF data in a single pass of the SMF data and create the SAS datasets in WORK or in a PDB.

UTILBLDP

For documentation on all parameters and usage see the member in the MXG SOURCLIB

For our purposes there are only a few important parameters

SORTOUT=NO – suppresses sorting and writing of the data to the PDB DD. You may want to use the sort (just add a PDB DD to your JCL) as it will remove any duplicate records.

USERADD= a list of the record types you wish to read – 30 6 1415 64 70 etc.

UTILBLDP

OUTFILE= INSTREAM writes the data to the temporary dataset defined by the INSTREAM DD. You can then simply %INCLUDE INSTREAM to execute the code. If you want to STORE the code for future use (or just to see what the generated code looks like) route to a PDB member or a sequential dataset.

BUILDPDB=NO – suppresses the logic that builds the full MXG PDB.

BLDSMPDB

Build the daily/weekly/monthly/trend databases from a single job on ASCII platforms (the JCL just would not work on zOS – could be done using DYNALLOC and LIBNAME statements but that would preclude the use of GDGs.)

BLDSMPDB

There are numerous parameters – too many to mention here but all are documented in the member of SOURCLIB

– Allows for reruns– User code– Run daily/weekly/monthly– Run WTD MTD– Run TRENDing daily/weekly– Read DCOLLECT and Tape management data– And much much more…–

Combine these to tailor your PDB Use UTILBLDP to add/subtract record

types and specify things to run after BUILDPDB

Use BLDSMPDB to control the execution of BUILDPDB

Example

Suppress CICSTRAN and DB2ACCT but process statistics datasets for both CICS and DB2

Add TYPE6156 and TYPE42 data to the PDB

Suppress TYPE74 data

ANALID

New MACRO to create an SMF Audit dataset and report– READSMF=NO– PRINT=YES– PDBOUT=PDB– PERCENTS=YES– ODS parameters

ANALID – READSMF

READSMF=YES will read an SMF dataset. The default of NO is used in BUILDPDB to read the ID dataset already being created.

Driven by the value of the SMFAUDIT macro variable in VMXGINIT. If set to NO with a %LET the older style report is created with fewer variables.

ANALID – PRINT/PDBOUT/PERCENTS

PRINT=YES – prints SMF Audit report. To suppress the report specify NO.

PDBOUT=PDB – the destination of the new SMFRECNT dataset.

PERCENTS=YES – calculates the percentage of the data for each system represented by a single type/subtype.

ANALID – ODS Parameters

ODSTYPE= if you want to create HTML output specify HTML or specify some other valid ODS value. If blank ODS is not used.

ODSPATH= the pathname for the ODS output – typically a directory on ASCII or a PDSE or zFS directory on zOS

ODSFILE= the name of the output that will be created

ANALID - Example

%ANALID( READSMF=YES, PDBOUT=PDB, PRINT=YES, ODSTYPE=HTML, ODSPATH=E:\. ODSFILE=ANALID.HTML);

ANALID – Sample

ANALID - Sample

ANALID – Sample

ANALID - Sample

ANALGRID

Creates a dense color coded grid of values using PROC REPORT

Does not require SAS/GRAPH Works on all SAS versions 9.1.3 and

ANALGRID

Example 1– Read ASUM70LP and for the specified

system create a grid of CPU busy for a day.

– This is the default with addition of an INCODE to select a specific LPAR

%ANALGRID(INCODE=IF LPARNAME=SYSG;);

ANALGRID

Example 2 – compare year to year same month excluding weekdays and holidays

– %ANALGRID(– INDATA=RMFINTRV,– SORTBY=SYSTEM MONTH,– SYSTEM=SYSG,– INCODE=MONTH=DATEPART(STARTIME)-DAY(DATEPART(STARTIME))+1;– FORMAT MONTH MONYY.;– if 1 lt weekday(datepart(startime)) lt 7;– if month(datepart(startime))=1;– if datepart(startime) not in('26dec11'd,'24nov11'd,'25nov11'd,– '05sep11'd,'04jul11'd,'30may11'd,'21feb11'd,'17jan11'd,'24dec10'd,– '25nov10'd,'26nov10'd,'16jan12'd,'02jan12'd,'16jan12'd,'20feb12'd);,– TITLE1=% CPU Busy,– VARIABLE=pctcpuby,VARLABEL=% CPU,varformat=5.2,– ROWVARIABLE=DATE,ROWLABEL=DATE,ROWFORMAT=DATE.,– ODSPATH=e:,ODSFILE=april.html);

ANALGRID

You have complete control of– Colors and levels– Column and row variables– Column and row labels– Column and row formats

ANALGRID

%ANALGRID( SYSTEM=SYSG, INDATA=RMFINTRV, SORTBY=SYSTEM, VARFORMAT=TIME12.2, dates=lastweek, BKT1='01:00'T/BLUE/WHITE, BKT2='02:00'T/GREEN/WHITE, BKT3='03:00'T/CYAN/BLACK, BKT4=, WEIGHT=, SORTLABEL=System, STAT=SUM, VARIABLE=CPUTM, odspath=e:, odsfile=cputime.html, VARLABEL=CPU TIME, COLVARIABLE=TIME,COLLABEL=TIME,COLFORMAT=TIME5.,

ROWVARIABLE=DATE,ROWLABEL=DATE,ROWFORMAT=DATE.

ANALGRID

VMXGPRNT

Utility to print any SAS dataset with labels modified to include the variable name and/or create a comma delimited output (CSV).

VMXGPRNT – Parameters

SP_DSET – dataset to be printed – defaults to _LAST_

SP_NOBS – number of OBS to be printed – defaults to 20

SP_REMV – remove * from labels in CSV file – defaults to NO

TMPPRNT – destination for a temporary dataset – on zOS it will be constructed and dynalloc’ed as a temporary dataset but on ASCII will be placed in your SASUSER directory. Defaults to TMPPRNT.SAS

BYLST – list of BY variables – defaults to a null string

VARLST – list of variables to be printed. Default is a null string which will print all variables

NOEXIMSG – suppresses various warnings/notes – default is YES

SP_OPNS – PROC PRINT options default is SPLIT=‘*’

VMXGPRNT – Example 1

%VMXGPRNT(SP_DSET=PDB.DB2ACCT,SP_NOBS=3);

Print PDB.DB2ACCT

Create a CSV file– Filename csv ‘h:\mxg\vmxgprnt.csv’;– ods csvall file=csv;

– %vmxgprnt(SP_DSET=PDB.DB2ACCT,SP_NOBS=3,sp_remv=Y);

– run;

– ods csvall close;

– run;

VMXGFIND

Utility that will find every OBS in every dataset where some condition is satisfied and make a copy/print the observations.

For example:– Find all obs where JOB=:’CICS’

VMXGFIND – Parameters

PDB= LIBNAME to be searched – default is PDB – can be 1 or many

PDBOUT= where to put the output datasets – datasets here will be named DDNAME_dataset where DDNAME is the libname where they were found

KEEPIN= a list of variables that are used in the comparison

FIND= the comparison – for example…– Job=:’CICS’

– KEEPIN=STARTIME STRTTIME INTBTIME,– FIND= IF ('31JAN2010:10:11:12'DT LE STARTIME LE '31JAN2010:22:23:24'DT )

OR ('31JAN2010:10:11:12'DT LE STRTTIME LE '31JAN2010:22:23:24'DT ) – OR ('31JAN2010:10:11:12'DT LE INTBTIME LE '31JAN2010:22:23:24'DT ) ;,

PRINT= default is NO– YES – print all the observations– NO – no print– xxx – print xxx observations

VMXGFIND

If PRINT=YES or xxx then VMXGPRNT is used to do the printing

Example 1:– %VMXGFIND(FIND=QWHSSSID=DBTB,PRINT=3);

VMXGFIND

VMXGSRCH

Utility that will find every observation in every dataset in every allocated SAS data library where the value of the observation contains some string.– Note: libraries must have been allocated

either explicitly (LIBNAME statement) or by a DATA/PROC step.

VMXGSRCH – Parameters

LIBNAME= the libname to be searched. Default is a NULL string. _ALL_ will search all allocated SAS data libraries (they don’t have to be MXG) and anything else will search that specific LIBNAME. Only LIBNAMEs that have been opened will be found!!!!! You may need to insert a LIBNAME on zOS.

VMXGSRCH - Parameters

COPYTO= copy the datasets and observations that match to this LIBNAME

NOBS= the number of OBS to print – default is MAX

LOG= a large number of lines may be generated – LOG=NO suppresses them. Default is YES

VMXGSRCH - Parameters

VALUE – the value to search for Results= what you want us to do

– PRINT – just print the obs/datasets that match– COPYONLY – copy the datasets but don’t print– COUNT – just produce a count of datasets/obs/variables

that match– LABEL – produce a list of variables/datasets where the

value is in the label– FORMAT – produce a list of variables/datasets where the

value is in the format

VMXGSRCH – Example 1

%VMXGSRCH( LOG=NO,RESULTS=COUNT, VALUE=D2DD,LIBNAME=PDB);

VMXGSRCH- Example 1

%VMXGSRCH( LOG=NO,RESULTS=PRINT,NOBS=2, VALUE=D2DD,LIBNAME=PDB);

%VMXGSRCH( LOG=NO,RESULTS=PRINT,NOBS=2, VALUE=D2DD,LIBNAME=PDB, COPYTO=WORK);

%VMXGSRCH( LOG=NO,RESULTS=COPYONLY, VALUE=D2DD,LIBNAME=PDB, COPYTO=WORK);

%VMXGSRCH(VALUE=CPU,RESULTS=LABEL);

NOTE: Values are case sensitive

VMXGSRCH(VALUE=TIME,RESULTS=FORMAT);

UTILWORK

Don’t understand the documentation on defining your workloads to RMFINTRV? This utility will build you a skeleton RMFINTRV member based on your TYPE72GO records.

UTILWORK - Parameters

PDB= may be either SMF or some libname that contains a TYPE72GO dataset. SMF is preferred since the normal _ETY72GO exit will suppress service classes with no activity in an interval. You only need to use a single RMF interval.

UTILWORK – Parameters

USEREPRT= YES/NO do you want to use report classes or service classes to define workloads. Strongly recommended that you use report classes since there can be many many more at no real cost.

UTILWORK - Example

%UTILWORK(PDB=PDB, OUTFILE=RMFINTRV, USERPRT=YES, INTERVAL=QTRHOUR)

UTILWORK - Example

READDB2

MXG supplied macro that generates the code to read all of the different types of DB2 SMF data (all IFCIDs). It has been ‘enhanced’ to make a copy of the SMF data and allow for selection based on reading the record headers only which makes it very fast.

READDB2

For a full list of parameters and usage see READDB2 member in the MXG SOURCLIB

Concentration here will be on selection parameters and copying of SMF data

READDB2

SMFOUT= DDNAME to which SMF data will be copied – if blank no copy is made

COPYONLY= YES/NO – only copy SMF data do not format SAS datasets– Useful to make mini-SMF files to feed to

DB2PM or send off to vendors PDBOUT= DDNAME to which SAS datasets

are written (WORK is default if left blank)

READDB2 - Parameters

SYSTEM – list of systems PLAN – list of plan names AUTHID – list of authorization IDs CORRID – list of correlation IDs CONNID – list of connection IDs DB2 – list of DB2 subsystems CONNTYPE – list of connect types

READDB2 - Parameters

TRANNAME – list of end-user transaction names

PACKAGE – list of package names SMFBEGIN =SAS datetime constant –

starting point of data SMFEND – SAS datetime constant –

end point of data– SAS datetime constants are of the form

01sep10:01:30:00 – no quotes are needed

READDB2

All values in lists separated by spaces All parameters separated by commas (except the last

one) All values are automatically wild carded – that is,

however many bytes are in the value is the length of the compare

SMFBEGN= earliest time in form ddmmmyy:hh:mm:ss or 10OCT08:15:00:00

SMFEND= latest time in same form

READDB2

%READDB2(TRANNAME=OLB_DISP, COPYONLY=YES,SMFOUT=SMFOUT);– Copy records where TRANNAME starts with

OLB_DISP to SMFOUT DD but do not create SAS datasets

%READDB2(TRANNAME=OLB,PDB=WORK, SMFOUT=SMFOUT);– Copy records where TRANNAME starts with OLB

and also place them in SAS datasets in the WORK dataset

VMXGSUM

Generalized summarization of ANY SAS dataset

– Uses PROC MEANS to do summarization– SORTs data– Allows for changes in input and output data– Optimizes variables kept – Carries labels and formats thru summarization– Allows for long variable names– Allows for normalization of variables and changing time intervals

VMXGSUM

Common in reporting:– DATA xxxx;– SET yyyy;– PROC SORT DATA=xxxx;– PROC MEANS DATA=XXXX OUT=zzzz;– DATA final;– SET zzzz;

VMXGSUM

VMXGSUM is a short-hand way of coding a repetitive set of commands.

Used extensively internally in many MXG members but especially common in ASUM**** and TRND**** members.

VMXGSUM - SYNTAX

%VMXGSUM(– INDATA= input dataset(s) name– OUTDATA= output dataset name– SUMBY= list of variables by which data should

be sorted– INCODE= a stub of SAS code executed during

the first data step– OUTCODE= a stub of SAS code executed

during the final data step

VMXGSUM - SYNTAX

– INTERVAL= how to change the time interval. Valid values are:

QTRHOUR HALFHOUR HOUR THREEHR MINUTE WEEK MONTH MYTIME

– DATETIME= the variable name of the variable containing the datetime value on which INTERVAL= will be applied

– SYNC59= if your time is synched to 59 minutes, will add 60 seconds before calculating interval if set to YES

VMXGSUM - SYNTAX

ID= list of variables that will be carried forward as ID values

AUTONAME=YES/NO AUTONAME = YES says to use the autonaming functions of SAS V8 to name the output variables.

– This allows the specification of the same variable name in multiple lists but changes the output variable name to variable_suffix where suffix is the name of the function performed on the variable.

VMXGSUM - SYNTAX

SUM= list of variables to be summed MAX= list of variables to be maxxed MIN= list of variables to be minned MEAN= list of variables to be meaned P1= list of variables to get percentile 1 P5= 5th percentile variables P10= 10th percentile variables

VMXGSUM - SYNTAX

– P25 P50 P75 P90 P95 P99 - percentile values

– STD - Standard Deviation– VAR - variance– CV - coefficient of variance– STDERR - Standard error– KURTOSIS - Kurtosis– T - T value

VMXGSUM - Syntax

NORM1-NORM99 - normalization of data. Maintaining rates as rates and not averages of averages. On the front-end, the rate has to be multiplied by the duration and on the back end divided again to recalculate the correct rate.

VMXGSUM - SYNTAX

– NORM1-NORM99 - syntax rate1 rate2 rate3…ratex/duration

List the variables to be normalized followed by a / then the variable to be used to do the normalization.

VMXGSUM - SYNTAX

There are other parameters. See the documentation in the member for usage and the member ADOCSUM.

VMXGSUM - Example 1

Summarize the dataset TYPETMNT by DEVICE and TMNTTIME calculating average mount delay and the total number of mounts per quarter hour.

%vmxgsum( indata=pdb.typetmnt, outdata=tapemnts, sumby=device tmnttime, interval=qtrhour, datetime=tmnttime, mean=tapmnttm, freq=mounts);

VMXGSUM - Example 2

Summarize the Goal Mode type 72 records for the TSO service class calculating the average response time, the number of transactions at one hour intervals by period.

VMXGSUM - Example 2

%VMXGSUM( INDATA=PDB.TYPE72GO, OUTDATA=TSOSUM, SUMBY=STARTIME PERIOD, INCODE= IF SRVCLASS=‘TSO’;, SUM=RESPAVG NUMTRAN, NORM1=RESPAVG/NUMTRAN, INTERVAL=HOUR, DATETIME=STARTIME);

VMXGSUM Usage Notes

NORMx operands must be contiguous starting at 1. That is, you cannot have NORM1 and NORM3 without a NORM2.

VMXGSUM Usage Notes

The first data step is almost always converted to a VIEW rather than a real data step.

KEEPALL=NO is resource intensive and not really needed except in odd cases. KEEPALL=YES is much preferred. The keep lists on all output datasets are optimized regardless of KEEPALL setting.

Why VMXGSUM?

So why not just use PROC MEANS with CLASS operands?

VMXGSUM in tests is usually much more efficient and in some cases will do the summarization where using PROC MEANS or PROC SUMMARY with CLASS operands runs out of memory.

This is especially true with the current release of SAS (9.1.3 SP4) on zOS which is defaulting to using THREADS.

ANALCNCR

Counts concurrent events. How many of something were happening at the same time.

ANALCNCR - History

Method used in original release of MXG:– DO TIME=BEGIN TO END BY 5;– OUTPUT;– END;– Then add up all the observations with a

given value of TIME. Created a HUGE number of observations and was cumbersome.

ANALCNCR - History

Method used with ANALCNCR:– TIME=BEGIN;COUNT=1;OUTPUT;– TIME=END;COUNT=-1;OUTPUT;

– Now add up the counts by time and you are done (basically.) Many many fewer observations.

ANALCNCR - History

If there are three tape allocations:

– Allocation 1 begins at 08:00 ends at 08:30– Allocation 2 begins at 08:15 ends at 08:25– Allocation 3 begins at 08:20 ends at 08:45

ANALCNCR - History MAX of 3 concurrent allocations

– 15 minutes of 1 – 5 minutes of 2 – 5 minutes of 3 – 5 minutes of 2 – 15 minutes of 1

Old method– Allocation 1 - 1800/5=360 obs– Allocation 2 - 600/5=120 obs– Allocation 3 - 1500/5=300 obs– Total = 780 obs

New Method– Each allocation is 2 OBS– Total = 6

ANALCNCR - Example 1

How many jobs are running concurrently in class A average and max.

%ANALCNCR(INDATA=PDB.JOBS, OUTSUMRY=RUNTIME, SUMBY=JOBCLASS, INCODE=IF TYPETASK=‘JOB’;, INTERVAL=QTRHOUR, STARTIME=JINITIME, ENDTIME=JTRMTIME, OTCODESM= AVGRUN=CONCURNT/DURATM; RENAME MAXCNCR=MAXRUN;);PROC PRINT;ID JOBCLASS TIMESTMP;VAR AVGRUN MAXRUN;

Now suppose you want the INPUT QUEUE time for the same job class.

%ANALCNCR(INDATA=PDB.JOBS, OUTSUMRY=QUETIME, SUMBY=JOBCLASS, INCODE=IF TYPETASK=:’JOB’;, INTERVAL=QTRHOUR, STARTIME=READTIME, ENDTIME=JINITIME, OTCODESM= AVGQUE=CONCURNT/DURATM; RENAME MAXQUE=MAXRUN;);PROC PRINT;ID JOBCLASS TIMESTMP;VAR AVGQUE MAXQUE;

Now put the two outputs together

DATA JOBSTAT;MERGE RUNTIME QUETIME;BY JOBCLASS TIMESTMP;PROC PRINT;ID JOBCLASS TIMESTMP;VAR AVGQUE AVGRUN MAXQUE MAXRUN;

ANALCAPD Can you save money by capping the MSU’s consumed? Billing is based on the peak of the rolling 4 hour MSU average Rolling average will (almost) always lag behind actual usage So, you can set a cap lower than the actual peak and possibly

reduce software billing ANALCAPD will let you ‘play’ with values to find a happy MSU

value that allows work to run while reducing the peak MSU value

ANALCAPD

Uses the ASUMCEC dataset in the PDB as input

Best granularity is when you match CECINTRV to INTERVAL in ASUM70PR

ANALCAPD – Parameters

PDB=PDB – where is the ASUMCEC data GRAPHICS=YES – use SAS/GRAPH (it will

detect if it is not there) DEFCAP= the MSU value you want to ‘model’ CECINTRV=HOUR – the CECINTRV value in

use – QTRHOUR HALFHOUR etc

ANALCAPD - Results

ANALCAPD – Results

Black line is current capacity Cyan line is current cap (in this case

there is not one) Blue line is actual usage Green line is rolling 4 hour average Red * are the intervals where the CEC

would have been capped

MXG Tools and Usage

Documents