Getting Yourself Sorted With DFSORT · Sample DFSORT JCL //EXAMP JOB A492,PROGRAMMER //SORT EXEC...

Getting Yourself Sorted With DFSORTRahul Bajaj

Broadcom

November 2019

Session AD

https://twitter.com/gseukc

https://www.linkedin.com/groups/1741877

https://www.facebook.com/pages/Guide-Share-Europe-UK-Region/174907039241415

Agenda

• Introduction to DFSORT• What can you do with DFSORT• DFSORT Control Statements

• Introduction to ICETOOL• Invoking ICETOOL through JCL• ICETOOL Operators

• Application Programming Basics• Invoking DFSORT from a program• Improving efficiency

• Smart DFSORT examples

• Q & A

Introduction to DFSORT

What is DFSORT?

• DFSORT is IBM's high-performance sort, merge, copy, analysis, and reporting product for z/OS

• DFSORT gives you versatile data handling capabilities at the record, field and bit level

• DFSORT delivers both performance improvements as well as programmer’s productivity improvements

What else can you do with DFSORT?

While sorting, merging, or copying data sets, you can also perform other tasks such as:

• Select a subset of records from an input data set

• Reformat records in a variety of ways

• Sum the values in selected records while sorting or merging (but not while copying)

• Create multiple output data sets and simple or complex reports from a single pass over an input dataset

• Convert VB data sets to FB data sets and vice versa

and much more..

Invoking DFSORT

• With a JCL EXEC statement, using the name of the program or the name of the cataloged procedure

• Within programs written in COBOL, PL/I, or Assembler language

• Invoked from a CLIST or REXX

• Called by the ICETOOL utility

Sample DFSORT JCL

//EXAMP JOB A492,PROGRAMMER

//SORT EXEC PGM=SORT

//SYSOUT DD SYSOUT=A

//SORTIN DD DSN=A123456.SORT.SAMPIN,DISP=SHR <Input data set>

//SORTOUT DD DSN=A123456.SORT.SAMPOUT,DISP=OLD <Output data set>

//SYSIN DD *

<DFSORT control statements>

SORT FORMAT=CH,

FIELDS=(110,10,A,145,17,A,1,75,A)

/*

<Additional JCL statements go here>

DFSORT Control Statements and SYMBOLS

Most important DFSORT control statements

• Selects records to be included based on a logical criteriaINCLUDE

• Selects records to be omitted based on a logical criteriaOMIT

• Describes how fields are to be summed after sorting or mergingSUM

• Describes how records are to be reformatted after (or before) they are sorted, copied or mergedOUTREC and INREC

• Describes various types of processing to be performed for one or more output data sets after sorting, copying or mergingOUTFIL

Description of sample bookstore data set

INCLUDE and OMIT statements

• These statements provide powerful filtering capabilities for DFSORT

• Multiple logical conditions can be joined by OR or AND for complex tests

• Using these statements, you can significantly increase the speed of the sort, copy or merge

• INCLUDE and OMIT both offer powerful substring search capabilities

Note: You cannot use both statements together.

INCLUDE and OMIT examples

INCLUDE/OMIT COND=(location,length,format,comparison operator,location,length,format)

INCLUDE COND=(166,4,BI,GT,162,4,BI)

OMIT COND=(110,5,CH,EQ,C' ')

INCLUDE COND=(42,10,CH,GE,DATE1(/)-30,AND,42,10,CH,LE,DATE1(/)+30)

INCLUDE COND=(22,6,PD,EQ,NUM,AND,28,6,PD,EQ,NUM)

INCLUDE COND=(106,5,SS,EQ,C'BIOL ,HIST ,BUSIN,PSYCH')

SUM statement

• On this statement, you can specify one or more numeric fields that are to be summed (called ‘summary fields’) whenever records have same control fields (specified on the SORT statement)

• In simple words, contents of the summary fields are summed for "duplicate" records

• The data formats you can specify on the SUM statement are binary (BI), fixed-point (FI), packed decimal (PD), zoned decimal (ZD) and floating-point (FL)

Note: When a sum becomes larger than the space available for it, overflow occurs and records are kept unchanged.

SUM examplesSUM FIELDS=(location,length,format)

INCLUDE COND=(110,5,CH,EQ,C'ENGL')

SORT FIELDS=(110,5,CH,A)

SUM FIELDS=(170,4,BI)


SUM FIELDS=(162,4,166,4),FORMAT=BI


SUM FIELDS=NONE

OUTREC and INREC statements

• These statements can be used to reformat records with fixed fields or variable fields

• A wide variety of tasks can be performed while sorting, merging or copying such as:• Delete/Reorder fields.• Insert separators (blanks, zeros, strings, current date,….)• Translate data from uppercase to lowercase and vice versa• Convert numeric values to many different types of printable formats• Perform arithmetic using numeric values and decimal constants• Perform arithmetic using date fields• Perform various operations on groups of records

OUTREC and INREC statements (continued)

Records can be reformatted in one of the following three ways:

• Using BUILD, reformat each record by specifying all of its items one by one

• Using OVERLAY, reformat each record by specifying just the items that overlay specific columns

• Using IFTHEN clauses, reformat different records in different ways by specifying how BUILD or OVERLAY items are applied to records that meet given criteria

OUTREC and INREC examplesOUTREC/INREC FIELDS=(location,length,..)


SUM FIELDS=(162,4,BI,166,4,BI)

OUTREC FIELDS=(106,4,162,4,166,4)

OUTREC FIELDS=(1,80,C'Timestamp: ',DATE1(/),X,TIME2(:))

OPTION COPY

OUTREC FIELDS=(11:C'Publisher is ',106,4,31:C'Author is ',91,15,X,76,15)

INREC FIELDS=(110,5,170,4)


SUM FIELDS=(6,4,BI)

OUTFIL statement

• Using this statement, you can create one or more output data sets and reports from a single pass over sorted, merged, or copied input

• Different OUTFIL parameters can be used on different OUTFIL statements

• OUTFIL processing is performed at the end after all other statements are processed

• Can be used with any other DFSORT statements (for example, INCLUDE, OMIT, INREC, SORT, MERGE, OPTION, SUM and OUTREC)

OUTFIL examples

//COPY JOB A492,PROGRAMMER

//S1 EXEC PGM=SORT


//SORTIN DD DSN=A123456.SORT.SAMPIN,DISP=SHR

//BACKUP DD DSN=A123456.BOOKS.BACKUP,DISP=OLD

//NEWYORK DD DSN=BOOKS,UNIT=3490,DISP=(,KEEP),VOL=SER=REMOT1,LABEL=(,SL)

//SANJOSE DD DSN=BOOKS,UNIT=3490,DISP=(,KEEP),VOL=SER=REMOT2,LABEL=(,SL)

//SYSIN DD *

OPTION COPY

OUTFIL FNAMES=(BACKUP,NEWYORK,SANJOSE)

/*

OUTFIL examples (continued)

//COPY JOB A492,PROGRAMMER

//S1 EXEC PGM=SORT



//SORTOF01 DD DSN=A123456.BOOKS.ENGL,DISP=OLD

//SORTOF02 DD DSN=A123456.BOOKS.HIST,DISP=OLD

//SORTOF03 DD DSN=A123456.BOOKS.PSYCH,DISP=OLD

//SYSIN DD *


OUTFIL FILES=01,INCLUDE=(110,5,CH,EQ,C'ENGL')

OUTFIL FILES=02,INCLUDE=(110,5,CH,EQ,C'HIST')

OUTFIL FILES=03,INCLUDE=(110,5,CH,EQ,C'PSYCH')

/*

SYMBOLS

• A symbol is a name (preferably something meaningful) you can use to represent a field or a constant

• DFSORT's symbol processing feature gives you a powerful, simple and flexible way to create symbol mappings for your own frequently used data

• Symbols turn DFSORT's syntax into a high level language

• Symbols can help to standardize your DFSORT applications

SYMBOLS examples//SYM1 JOB A492,PROGRAMMER

//SORTIT EXEC PGM=SORT

//SYSOUT DD SYSOUT=*

//SYMNAMES DD DSN=A123456.JCL.DFSORT(SYMNAMES),DISP=SHR

//SYMNOUT DD SYSOUT=*


//SORTOUT DD DSN=A123456.SORT.SAMPOUT,DISP=SHR

//SYSIN DD *

INCLUDE COND=(COURSE_NUMBER,EQ,C'00032',OR,

COURSE_NUMBER,EQ,C'10347')

SORT FIELDS=(TITLE,A,

INSTRUCTOR_LAST_NAME,A,INSTRUCTOR_INITIALS,A,

PRICE,A)

/*

* Symbols for fieldsTitle,1,75,CHAuthor_Last_Name,*,15,CHAuthor_First_Name,*,15,CHPublisher,*,4,CHCourse_Department,*,5,CHCourse_Number,*,5,CHCourse_Name,*,25,CHInstructor_Last_Name,*,15,CHInstructor_Initials,*,2,CHNumber_in_Stock,*,4,BINumber_Sold_YTD,*,4,BIPrice,*,4,BI

DFSORT Limitations and Considerations

• Control statement positions must be adjusted accordingly for Variable-length records and Fixed-length records

• You must describe all data sets in DD statements

• Variable-length records are limited to 32756 bytes

• VSAM variable-length records are limited to 32752 bytes

• Fixed-length records are limited to 32760 bytes

• DFSORT truncates fixed-length records on the right when the SORTOUT LRECL is smaller than the SORTIN/SORTINnn LRECL

• DFSORT pads fixed-length records with binary zeros on the right when the SORTOUT LRECL is larger than the SORTIN LRECL

Introduction to ICETOOL Utility

What is ICETOOL?

• ICETOOL is a multipurpose DFSORT utility that uses the capabilities of DFSORT to perform multiple operations on one or more data sets in a single step.

• ICETOOL is a batch front-end utility

• ICETOOL can be called directly or from a program.

• ICETOOL includes 17 operators

• Usage: • Reporting

• Statistical Analysis

Sample ICETOOL JCL


//TOOL EXEC PGM=ICETOOL

//TOOLMSG DD SYSOUT=A

//DFSMSG DD SYSOUT=A

//TOOLIN DD *

<ICETOOL statements go here>

/*

//xxxxCNTL DD *

<DFSORT control statements>

/*

//indd DD <Input data set>

//outdd DD <Output data set>

<Additional JCL statements go here>

ICETOOL Operators

Operator Function

COPY Copies a data set to one or more output data sets

COUNT Prints a message containing the count of records in a data set

DISPLAY Prints the values or characters of specified numeric or character fields in a separate list data set

SORT Sorts a data set to one or more output data sets

MODE STOP/CONTINUE/SCAN mode for operator(s)

OCCUR Prints each unique value for specified numeric or character fields and how many times it occurs in a separate list data set

DATASORT Sorts data records between header and trailer records in a data set to an output data set

MERGE Merges one or more data sets to one or more output data sets

DEFAULTS Prints the DFSORT installation defaults in a separate list data set

ICETOOL OperatorsOperator Function

RANGE Prints a message containing the count of values in a specified range for a specified numeric field in a data set

RESIZE Creates a larger record from multiple shorter records, or creates multiple shorter records from a larger record, that is, resizes fixed length records

SELECT Selects records from a data set for inclusion in an output data set based on meeting criteria

SPLICE Splices together specified fields from records that have the same specified field values, but different information

STATS Prints messages containing the minimum, maximum, average, and total for specified numeric fields in a data set

SUBSET Selects records from a data set by keeping or removing header records, relative records ortrailer records

UNIQUE Prints a message containing the count of unique values for a specified numeric or character field

VERIFY Examines specified decimal fields in a data set and prints a message identifying each invalid valuefound for each field

ICETOOL Operator : COPY

• DFSORT is called to copy the indd data set to the outdd data sets

//STEP0100 EXEC PGM=ICETOOL

//TOOLMSG DD SYSOUT=*

//DFSMSG DD SYSOUT=*

//INDD DD DSN=SORT.SAMPIN,DISP=SHR

//DEPT1 DD DSN=SORT.DEPT1,DISP=…

//DEPT2 DD DSN=SORT.DEPT2,DISP=…

//TOOLIN DD *

COPY FROM(IN) TO(DEPT1) USING(DPT1)

COPY FROM(IN) TO(DEPT2) USING(DPT2)

//DPT1CNTL DD *

INCLUDE COND=(5,3,CH,EQ,C'D01’)

//DPT2CNTL DD *


ICETOOL Operator : COPY – Alternative Approach

//STEP0100 EXEC PGM=ICETOOL



//TOOLIN DD *

COPY FROM(IN) USING(ALL3)

//ALL3CNTL DD *

OUTFIL FNAMES=DEPT1,INCLUDE=(5,3,CH,EQ,C'D01')


OUTFIL FNAMES=DEPT3,INCLUDE=(5,3,CH,EQ,C'D03’)

ICETOOL Operator : SORT• Sorts a data set to one or more output data sets

• DFSORT is called to sort the indd data set to the outdd data sets using the DFSORT control statements in xxxxCNTL which must be supplied

* Method 1

SORT FROM(IN) TO(DEPT1) USING(DPT1)

SORT FROM(IN) TO(DEPT2) USING(DPT2)

DPT1CNTL

SORT FIELDS=(51,2,BI,A,18,5,CH,A,58,4,BI,A)


* Method 2

SORT FROM(IN) USING(ALL2)

ALL2CNTL

SORT FIELDS=(51,2,BI,A,18,5,CH,A,58,4,BI,A)



ICETOOL Operator : DISPLAY

• Prints the values or characters of specified numeric fields or character fields in a separate list data set

• Simple, tailored, and sectioned reports can be produced

DISPLAY FROM(SOURCE) LIST(FIELDS) ON(NUM) ON(40,12,CH) -

ON(20,8,PD)

RECORD NUMBER (40,12,CH) (20,8,PD)

000000000000001 SAN JOSE 000000000003745

000000000000002 MORGAN HILL 000000000016502

. . .

. . .

ICETOOL Operator : OCCUR• Prints each unique value for specified numeric fields or character fields, and how

many times it occurs, in a separate list data set

OCCUR FROM(IN) LIST(LIST1) -

TITLE(' 3090 Distribution ') -

PAGE -

HEADER('Data Centers') ON(VALCNT) -

HEADER('State') ON(1,16,CH) -

HEADER('3090s') ON(25,3,PD) -

BLANK

3090 Distribution - 1 -

Data Centers State 3090s

--------------- ---------------- ------

12 Alabama 1

6 Alabama 2

. . .

. . .

ICETOOL Operator : MODE

• Specifies one of three modes to control error checking and actions after error detection• STOP or CONTINUE processing operators after a return code of 8, 12 or 16• SCAN to check for errors in ICETOOL statements, but do not call DFSORT

MODE SCAN

RANGE ...

UNIQUE ...

MODE STOP

VERIFY ...

DISPLAY ...

MODE CONTINUE

COPY ...

SORT ...

STATS ...

ICETOOL Operator : COUNT

• Prints a message containing the count of records in a data set

• Can also be used to subtract a value from the count or add a value to the count, to create an output data set containing text and the count

COUNT FROM(IN3) WRITE(CT3) DIGITS(6) SUB(2) –

TEXT('Count is ‘)

Count is 008123

Application Programming Basics

Invoking DFSORT from a program

In addition to processing your DFSORT program control statements directly with PGM=SORT or PGM=ICEMAN, you can call DFSORT from COBOL, PL/I, Assembler, or other programs

Pass the optional control statement(s)


.

.

//SORTCNTL DD *

INCLUDE

COND=(110,5,CH,EQ,C'ENGL')

/*

Invoke DFSORT from a COBOL program using SORT verb

PROCEDURE DIVISION.

.

.

SORT-ROUTINE SECTION.

SORT SD-FILE

ASCENDING KEY TITLE-IN

USING MASTER-FILE

GIVING SORTED-MASTER-FILE.

IF SORT-RETURN > 0

DISPLAY "SORT FAILED".

.

.

Execute the COBOL program from JCL


//BOOKS EXEC PGM=COBOLPGM

//STEPLIB DD

DSN=USER.PGMLIB,DISP=SHR


//MASTIN DD

DSN=A123456.MASTER,DISP=…

//MASTOUT DD

DSN=A123456.OUTB,DISP=…

//PRINTFL DD SYSOUT=A

Improving Efficiency : Tips and Tricks

• Directly invoke DFSORT processing

• Specify input/output data set characteristics accurately

• Use options that enhance performance

• Avoid options that degrade performance

• Use ICEGENER instead of IEBGENER

• Consolidate records with SUM

• Create multiple output data sets with OUTFIL

• Replace program logic with DFSORT control statements

Smart DFSORT examples

Keep dropped duplicate records

//DOIT EXEC PGM=ICETOOL



//IN DD DSN=... input data set

//OUT DD DSN=... first record with each key

//SORTXSUM DD DSN=... subsequent records with each key

//TOOLIN DD *

SELECT FROM(IN) TO(OUT) ON(1,3,CH) FIRST DISCARD(SORTXSUM)

/*

Using the translation feature

//Change all low values (X'00') to spaces (X'40'), in an FB data set with an LRECL of 60:

ALTSEQ CODE=(0040)

OUTREC FIELDS=(1,60,TRAN=ALTSEQ)

//Make your changes to specified fields instead of to the entire record.

ALTSEQ CODE=(0040)

OUTREC FIELDS=(1,2,TRAN=ALTSEQ, CH - change zeros to spaces

21,5, PD field - no change

26,55,TRAN=ALTSEQ) CH - change zeros to spaces

Sampling the records

//S1 EXEC PGM=ICEMAN


//SORTIN DD DSN=... input file

//SORTOUT DD DSN=... output file

//SYSIN DD *

OPTION COPY

OUTFIL STARTREC=100,SAMPLE=100

/*

Squeezing out blanks or other characters

Input Records (LRECL=48)

Kevin James R

Douglas Philips K

Smith John L

Expected Output Records

Kevin James R

Douglas Philips K

Smith John L





//SYSIN DD *

OPTION COPY

INREC BUILD=(1,48,SQZ=(SHIFT=LEFT,MID=C' '))

/*

Splitting an input data set//SPLIT1R EXEC PGM=ICEMAN


//SORTIN DD DSN=A123456.INPUT1,DISP=OLD

//OUT1 DD DSN=A123456.SPLITR1,DISP=(NEW,CATLG),

// SPACE=(CYL,(5,5)),UNIT=SYSDA





//SYSIN DD *

SORT FIELDS=(21,5,FS,A)

OUTFIL FNAMES=(OUT1,OUT2,OUT3),SPLIT1R=4

/*

Checking for a numeric string

Records to keep123456

000001

987654

003218

Records to be discardedA23456

000XY2

0ABCDE

75218R


//SYSOUT DD SYSOUT=



//SYSIN DD

OPTION COPY

INCLUDE COND=(1,6,FS,EQ,NUM)

/*

FB to VB and VB to FB file type conversion

//FBVB JOB A92,PROGRAMMER

//S1 EXEC PGM=SORT


//SORTIN DD DSN=A123456.SORT.SAMPIN,DISP...

//VBOUT DD DSN=A123456.SORT.VSAMP,DISP...

//SYSIN DD *

OPTION COPY

OUTFIL FNAMES=VBOUT,FTOV

/*

//VBFB JOB A92,PROGRAMMER

//S1 EXEC PGM=SORT


//SORTIN DD DSN=A123456.VBIN,DISP...

//FBOUT DD DSN=A123456.FBOUT,DISP... //SYSIN DD *

OPTION COPY

OUTFIL FNAMES=FBOUT,VTOF,OUTREC=(5,100)

/*

References

• The DFSORT home page on the World Wide Web at URL: http://www.ibm.com/storage/dfsort

• IBM Manuals• z/OS DFSORT: Getting Started SC23-6880-30• z/OS DFSORT Application Programming Guide SC23-6878-30

• DFSORT ICETOOL Papers and Examples at URL: https://www.ibm.com/support/pages/dfsort-icetool-papers-and-examples• Beyond Sorting by Frank Yaeger

• Smart DFSORT Tricks at URL: https://www.ibm.com/support/pages/smart-dfsort-tricks

http://www.ibm.com/storage/dfsort

https://www.ibm.com/support/pages/dfsort-icetool-papers-and-examples

https://www.ibm.com/support/pages/smart-dfsort-tricks

Please submit your session feedback!

• Do it online at http://conferences.gse.org.uk/2019/feedback/AD

• This session is AD

Date post:	03-Aug-2020
Category:	Documents
Upload:	others
View:	23 times
Download:	2 times

Getting Yourself Sorted With DFSORT · Sample DFSORT JCL //EXAMP JOB A492,PROGRAMMER //SORT EXEC...

Documents