JCL REFERENCE - Pepstock.org - JCL Reference.pdf · JEM is able to manage different JCL by a JEM...

JCL REFERENCEVersion 1.0 (ACACIA)

Job Entry Manager, the Batch Execution EnvironmentVersion 1.0 (ACACIA)

1 What's JCL?......................................................................................................................................42 JCL JEM Properties..........................................................................................................................43 Datasets and Data descriptions.........................................................................................................74 Data Sources...................................................................................................................................105 Locks..............................................................................................................................................106 ANT as JCL....................................................................................................................................11

6.1 Properties definition................................................................................................................116.2 Data Descriptions and Datasets in ANT.................................................................................136.3 Data sources in ANT...............................................................................................................136.4 Locks in ANT..........................................................................................................................146.5 JEM ANT extensions..............................................................................................................14

6.5.1 STEP-JAVA task..............................................................................................................146.5.2 STEP-EXEC task............................................................................................................17

6.6 Utilities....................................................................................................................................196.6.1 NULL task.......................................................................................................................196.6.2 WAIT task.......................................................................................................................206.6.3 COPY task.......................................................................................................................206.6.4 SORT task.......................................................................................................................216.6.5 ShellScript task...............................................................................................................216.6.6 Windows task..................................................................................................................226.6.7 Procedure and Procedure Definition task........................................................................23

7 ANT advanced utilities (Enterprise Edition Only).........................................................................257.1 GDG Utility............................................................................................................................25

7.1.1 Defining GDG.................................................................................................................267.1.2 Rebuilding GDGs............................................................................................................277.1.3 Cleaning GDGs...............................................................................................................28

7.2 Nodes Utility...........................................................................................................................297.3 Roles Utility............................................................................................................................30

7.3.1 CREATE and REMOVE roles........................................................................................307.3.2 GRANT and REVOKE permissions to roles..................................................................317.3.3 ADD and DELETE users to roles...................................................................................31

7.4 Common-resource Utility.......................................................................................................327.4.1 Getting common-resources.............................................................................................337.4.2 Remove common-resource..............................................................................................357.4.3 Setting common-resources..............................................................................................36

7.5 Statistics Utility.......................................................................................................................388 Spring Batch as JCL.......................................................................................................................42

8.1 Properties definition................................................................................................................428.2 Data Descriptions and Datasets in SpringBatch.....................................................................448.3 Data sources in SpringBatch...................................................................................................468.4 Locks in SpringBatch..............................................................................................................468.5 Tasklets...................................................................................................................................478.6 Chunks....................................................................................................................................488.7 Utilities....................................................................................................................................49

8.7.1 NULL tasklet...................................................................................................................498.7.2 WAIT tasklet...................................................................................................................50

2/51


8.7.3 COPY tasklet...................................................................................................................50

3/51


1 What's JCL?Job Control Language (JCL) is a scripting language to instruct the system on how to run a batch job. The modern JCL are XML slang, as ANT or SpringBatch that we'll analyze on next chapters.

In JCL the unit of work is the job. A job consists of one or several steps, each of which is a request to run one specific program. JCL is the way how a job can be configured and can declare all needed resources to use. JCL declares some run-time attributes as well, necessary to execute correctly the job.

2 JCL JEM PropertiesJEM is able to manage different JCL by a JEM factory but in all JCL JEM will try to read some important (and in some cases mandatory) information for job execution.These information are usually defined by properties in the JCL. There is not any length restriction and as following it's allowed definition of characters which could be used:

'"' ([^<&"] | Reference)* '"' | "'" ([^<&'] | Reference)* "'"

So, you can't have:

• the same character that opens/closes the attribute value (either ' or " )

• a naked ampersand (& must be &)

• a left angle bracket (< must be <)

You should also not being using any characters that are outright not legal anywhere in an XML document.

All information, that JEM reads, are following:

• Job Name

is mandatory string property which is used to assign the name to the job.

• Environment

is mandatory string property with the name of cluster which the job must be executed in.

• Domain

is optional string property that represents a subset of nodes (by a tag) which job must be executed in. Default is ***. When a job has got the default value can be executed in whatever inactive node.

• Affinity

is optional string property that represents a subset of nodes of a domain (by one or more tags, comma separated) which job must be executed in. Default is ***. When a job has got the default value can be executed in whatever inactive node.

• User

JCL JEM Properties 4/51

http://www.w3.org/TR/xml/#NT-Char


is optional string property that represents the user to use to execute the job. Default is null. Usually the job must use the user who submitted it during the execution. This is used for authorization phases to access to resources. Indicating a different user in JCL, you can override the submitter's user and to have different permissions. To use this feature you must be authorize to the surrogate permission.

• Locking Scope

is optional string property what is used to decide at which level the job must request the locks for defined resources (data description and/or lock). There are 3 possible scopes:

• job: JEM must collect all the declared resources definitions and try to lock all them. The lock is maintained till the end of job. This kind of locking scope blocks concurrent jobs, reducing the parallel executions when jobs ask for same resources.

• step: JEM locks the resources at the beginning of step and releases them at the end of each step. This scope increases jobs concurrency but be careful with the job business logic, because not always is acceptable that another job uses the resources of another during either executions.

• task: JEM locks the resources at the beginning of each task (if JCL implements task concept) and releases them at the end of task. This scope increases jobs concurrency but be careful with the job business logic, because not always is acceptable that another job uses the resources of another during either executions.

Default is job.

• Hold

is optional boolean property which is used to maintain the job in the input queue and wait a release command. Default is false. This is helpful when you don't want to execute the job immediately but you want to decided when this job can be manually executed.

• Priority

is optional integer property which is used to assign a priority in input queue to job. Default is 10. This property is used by nodes to decide which job must be executed. The logic is check the job with highest priority. With same priorities, node chooses the oldest one, using the submitted timestamp.

• Memory

is optional integer property which is used to assign the right amount of memory, in mega bytes (MB). Default is 256. Every node of JEM knows how much memory can use and decides to execute the job if the amount of memory indicated is less than available one for node.

• Classpath

is optional string property which is used to assign the right classpath before starting job. All different paths must be semicolon ( ; ) separated. Default is null. This property becomes



mandatory when the job must execute business logic, contained in jars not in JEM classpath. It's helpful for JCL extensions (by java implementation) as well.

• Emails Notification

is optional string property which is used to assign a list of email addresses to notify when a job ends. All different addresses must be semicolon ( ; ) separated. Default is null. A Job life-cycle listener is implemented out-of-the-box and when the ended job contains this property, a formatted mail is sent to defined mail addresses.



3 Datasets and Data descriptionsOne of most important goal of a job is create, read and write files, usually named datasets.

A dataset represents a file that a task must use during the job execution to implement its business logic. There is not any length name restriction and as following it's allowed definition of characters which could be used:


So, you can't have:




You should also not being using any characters that are outright not legal anywhere in an XML document.

There are 4 kinds of datasets, as following:

• File

is a normal file, can be indicated by complete absolute or relative path (even using variables to compose it). The name can not start with *. or @@ ,

Example: My/data/info.txt

• GDG

is version file which is addressable by a relative position starting from last version, can be indicated by complete absolute or relative path (even using variables to compose it). This kind of dataset is helpful when you create and use the same relative version every periodical execution. The name can not start with *. or @@ ,

Example: My/data/info(0) or My/data/info(+1) o My/data/info(-1)

• Reference

is a link to dataset previously defined in other tasks. With this kind of dataset you don't have to define the same dataset twice or more. Using GDGs and different locking scope, you are sure to use the previous file, at the same version. The name must start with *. .

E xample : *.step1.dd1

• Temporary

is a file created in temporary directory and deleted on ended of job. The temporary files are used to park some data for further operations in next steps of the same job. The name must start with @@ and will be use as prefix of temporary file name.

Example: @@mytemp

• Inline

Datasets and Data descriptions 7/51



are data put directly inside JCL, usually to configure the program executed in task or step.

• Data Source

is a link to data source previously defined in same task or step. Currently the unique data sources, usable as dataset, are FTP resources. The file name can not start with *. or @@,

The data description is a container of datasets (one or more), addressable by name, both a java code (JNDI) and executable command (by environment variables). It could be defined without datasets when job wants to writes data on a system output stream (called sysout). There is not any length name restriction and as following it's allowed definition of characters which could be used:


So, you can't have:




You should also not being using any characters that are outright not legal anywhere in an XML document.There are 3 kinds of data description.

• Sysout

is a system output stream, used by job to write information without creating any common dataset. A sysout is physically saved in output folder. This is very useful during test phase to check the result of business logic, seeing what the program writes , before creating any dataset.

• Single dataset

is a data description which contains only a single file, used by a job. The type of access (read or write) is defined by disposition attribute

• Multiple datasets

is a data description which contains more than one dataset , used by a job. The type of access must be only in read because is not possible to read more than one file on the same output stream.

Inside the data description, you must define the disposition that means how you want to access to data (read or write). This attribute is really important to lock the resources in right way, compliant with business logic.The possible dispositions are the following:

• NEW: declares that the job is asking for a new dataset. If exists, an exception will be throw. This is also mandatory for sysout data description.

• OLD: declares that the job is asking for an existing dataset in write mode, overriding the previous




content. If dataset doesn't exists, an exception will be throw.

• MOD: declares that the job is asking for an existing dataset in write mode, in append mode to previous content. If dataset doesn't exists, an exception will be throw.

• SHR: declares that the job is asking for an existing dataset in read mode. If dataset doesn't exists, an exception will be throw.



4 Data SourcesData sources are common resources defined centrally and addressable by jcl from any job (if the user of the job is authorized to use them). All resources must have a name, the key used by job to have it. There is not any length name restriction and as following it's allowed definition of characters which could be used:


So, you can't have:




You should also not being using any characters that are outright not legal anywhere in an XML document.Currently this implementation works only for JAVA main programs, because data sources are provided by JNDI (Java Naming and Directory Interface).

The great benefit of data sources is you could access to a database or ftp server using logical name, without caring about authentication stuff (as user-id or password) because JEM will prepare a connection already ready to use.All data sources are stored inside a Hazelcast in a specific map, called common-resources. The map is configured to be persistent. All resources can be defined by a specific component of web application (if authorized) or by batch Apache Ant utilities (only for Enterprise Edition).

5 LocksLocks are virtual resources that you could use when you need a traffic light inside of whole JEM cluster. With a lock, you can serialize the executions of jobs on several nodes by a name. There is not any length name restriction and as following it's allowed definition of characters which could be used:


So, you can't have:




You should also not being using any characters that are outright not legal anywhere in an XML document.A lock generates a global lock by name in exclusive mode, so when more jobs use the same lock will be serialized, respecting the locking scope.

10/51


http://ant.apache.org/

http://www.hazelcast.com/



6 ANT as JCLApache Ant is a Java library and command-line tool whose mission is to drive processes described in build files as targets and extension points dependent upon each other. The main known usage of Apache Ant is the build of Java applications. Apache Ant supplies a number of built-in tasks allowing to compile, assemble, test and run Java applications. Apache Ant can also be used effectively to build non Java applications, for instance C or C++ applications. More generally, Ant can be used to pilot any type of process which can be described in terms of targets and tasks.

Apache Ant is written in Java. Users of Apache Ant can develop their own "antlibs" containing Apache Ant tasks and types.

Apache Ant is extremely flexible and does not impose coding conventions or directory layouts to the Java projects which adopt it as a build tool.

JEM uses Apache Ant as main job control language, developing all management utilities on it (only for Enterprise Edition).

6.1 Properties definitionTo use Apache Ant inside JEM, ANT XML JCL file must contain the mandatory and optional properties that JEM needs, described above. as following:

• Job Name

is optional string property, called for ANT jem.job.name .

If missing, name attribute of project element will be used. If boths are missing, an exception occurs.

• Environment

is optional string property, called for ANT jem.job.environment .

If missing, the JEM node environment definition is used.

• Domain

is optional string property, called for ANT jem.job.domain .

If missing, default value ( *** ) will be used.

• Affinity

is optional string property, called for ANT jem.job.affinity .


• Useris optional string property, called for ANT jem.job.user .

ANT as JCL 11/51












If missing, default value ( null ) will be used. An exception occurs if the user, who submitted the job, is not authorized to change the user job execution.

• Locking Scope

is optional string property, called for ANT jem.job.lockingScope .

If missing, default value ( job ) will be used. If the value is not equals to one the possible values (job, step , task), an exception occurs.

• Hold

is optional boolean property, called for ANT jem.job.hold .

If missing, default value ( false ) will be used.

• Priority

is optional integer property, called for ANT jem.job.priority .

If missing, default value ( 10 ) will be used.

• Memory

is optional integer property, called for ANT jem.job.memory .

If missing, default value ( 256 ) will be used. Be aware the unit is MegaBytes.

• Classpath

is optional string property, called for ANT jem.job.classPath .

Defining a classpath, it will be able to import ANT task definitions of 3rd parties. The value is a string and the files are separated by ; . If the file doesn't represent an absolute path, the JEM will use relative position from jem.classpath folder. You could use variables that JEM substitutes in string value. If missing, default value ( null ) will be used.


is optional string property, called for ANT jem.job.emailsNotification . If missing, default value ( null ) will be used.

Here is the sample about the ANT properties definition:

<property name="jem.job.name" value="JOB1"></property><property name="jem.job.environment" value="ENV1"></property><property name="jem.job.domain" value="domain"></property><property name="jem.job.affinity" value="classA"></property><property name="jem.job.priority" value="99"></property><property name="jem.job.user" value="newUSER"></property><property name="jem.job.lockingScope" value="task"></property><property name="jem.job.hold" value="true"></property><property name="jem.job.memory" value="1024"></property><property name="jem.job.classPath" value="/test/ant-taskdefs-lib.jar"></property><property name="jem.job.emailsNotification" value="[email protected];[email protected]"/>

ANT as JCL 12/51


6.2 Data Descriptions and Datasets in ANTData descriptions and datasets are implemented by specific tags. Data descriptions could be defined as following:

<datadescription name="name-of-data-description" sysout="true/false" disposition="NEW/MOD/OLD/SHR">

</datadescription>

Data description needs a mandatory name attribute. This name is used inside the business logic to access to datasets by JNDI so this name must be unique in the task definition. The sysout attribute is optional one that if there is means that the data description is a sysout.

The disposition attribute is optional and must be a valid value, one of the following string enumeration: NEW, MOD, OLD or SHR.

Datasets are children of a data description definition and identifies a file or data necessary for business logic. Datasets could be defined as following:

<dataset name="file-name" resource="datasurce-name">.......</dataset>

Datasets could have the name attribute which could represent:

• file name (absolute or relative path), composed by ANT properties if necessary. If it's a relative path, JEM adds the content of jem.data variable, used to identify the global file system with all data.

• GDG file name, following the same rules of a normal file name (see previous item).

• Reference to another data set in another data description, not in the same task as well.

• Temporary prefix file name

If the name attribute is missing, a temporary file is created, putting the content dataset element.

Datasets could have the resource attribute which could represent a relation with a data source, previously defined. It works only the FTP resources and the name of dataset represents the file to manage by FTP.

6.3 Data sources in ANTData sources are implemented by specific tags. They could be defined as following:

<datasource name="my-ftp" resource="resource-ftp"> <property name="binary">true</property>

ANT as JCL 13/51


</datasource>

Data source needs a mandatory name attribute. This name is used inside the business logic to access to data source (for every kind of data source, as jdbc or ftp) by JNDI so this name must be unique in the task definition. Data source needs a mandatory resource attribute. This name is used to locate the common-resource, defined previously in JEM. If resource is not defined, an exception occurs.

Data source element can contain one or more property element to override at run-time the value of some properties defined to create a data source. The property element needs a mandatory name attribute. An exception could occur if you try to override a property created in JEM with override="true" typically used for password, user-id or URL.

6.4 Locks in ANTLocks are implemented by specific tags. They could be defined as following:

<lock name="lock-name"/>

Lock needs a mandatory name attribute. This name is used to create a lock inside the JEM environment in exclusive. This name must be unique in the task definition.

6.5 JEM ANT extensionsJEM extends ANT providing some custom tasks that you could use to benefit of special JEM features, as data descriptions, datasets or locks. It extends java and exec tasks.

6.5.1 STEP-JAVA task

<step-java> is a custom Ant task (extends Java Task) adding a set of data descriptions, data sources and locks functions. Using this way, the programmer is able to develop own job batch programs, avoiding strong references with physical files but logical ones, by JNDI context.

<taskdef name="step-java" classname="org.pepstock.jem.ant.tasks.StepJava"/>… …<step-java classname="org.pepstock.test.Main">

… …</step-java>

Inside of <step-java> task you could add all data descriptions, data sources or locks necessary to complete the business logic. Here is a JCL sample:

<taskdef name="step-java" classname="org.pepstock.jem.ant.tasks.StepJava"/>

<step-java classname="org.pepstock.test.Main">  <datadescription name="my-single" disposition="SHR">

ANT as JCL 14/51


<dataset name="my-file.dat"></dataset> </datadescription></step-java>

and the java main program which can read and display the file content:

Hashtable<String, String> env = new Hashtable<String, String>();env.put(Context.INITIAL_CONTEXT_FACTORY,

"org.pepstock.jem.node.tasks.jndi.JemContextFactory");

try {InitialContext context = new InitialContext(env);/** single file input stream **/Object object = (Object) context.lookup("my-single");FileInputStream inputStream = (FileInputStream) object;/** reads input stream **/Scanner sc = new Scanner(inputStream);sc.useDelimiter("\n");while (sc.hasNext()) {

String record = sc.next().toString();System.out.println(record);

}sc.close();

} catch (Exception ex){ex.printStackTrace();

}Here is the complete sample with many possible implementation (JCL and java main program):

JCL<step-java classname="org.pepstock.test.Main">

<classpath> <pathelement path="${java.class.path}"></pathelement> </classpath>

 <datasource name="my-db" resource="resource-db"></datasource>

 <datasource name="my-ftp" resource="resource-ftp"> <property name="binary">true</property> </datasource>

 <datadescription name="my-sysout" sysout="true" disposition="NEW"></datadescription>

 <datadescription name="my-single" disposition="SHR"> <dataset name="cnf.xml"></dataset> </datadescription>

 <datadescription name="my-ftp-data" disposition="SHR"> <dataset name="remote-file.data" resource="my-ftp"></dataset>

ANT as JCL 15/51


</datadescription>

 <datadescription name="my-multi" disposition="SHR"> <dataset name="my.data.gdg(0)"></dataset> <dataset name="cnf.xml"></dataset> </datadescription>

 <datadescription name="my-inline"> <dataset> Here I am This is a DD </dataset> </datadescription>

 <datadescription name="my-temp" disposition="NEW"> <dataset name="@@mytemp"></dataset> </datadescription>

 <datadescription name="my-reference" disposition="SHR"> <dataset name="*.main.step-java.single"></dataset> </datadescription>

</step-java>

MAIN Program Hashtable<String, String> env = new Hashtable<String, String>();env.put(Context.INITIAL_CONTEXT_FACTORY,


try {InitialContext context = new InitialContext(env);/** * database reference */Object object1 = (Object) context.lookup("my-db");DataSource datasource = (DataSource) object1;/** * syslog output stream */Object object2 = (Object) context.lookup("my-sysout");FileOutputStream fos = (FileOutputStream) object2;/** * single file input stream */Object object3 = (Object) context.lookup("my-single");FileInputStream fis = (FileInputStream) object3;/** * FTP input stream */Object object4 = (Object) context.lookup("my-ftp-data");

ANT as JCL 16/51


InputStream ftpis = (InputStream) object4;/** * sequence input stream (multi input stream) */Object object5 = (Object) context.lookup("my-multi");SequenceInputStream sis = (SequenceInputStream) object5;/** * Inline input stream */Object object6 = (Object) context.lookup("my-inline");FileInputStream fis1 = (FileInputStream) object6;/** * Referback file input stream */Object object7 = (Object) context.lookup("my-reference");FileInputStream fis2 = (FileInputStream) object7;/** * Temporary file output stream */Object object8 = (Object) context.lookup("my-temp");FileOutputStream fis3 = (FileOutputStream) object8;

} catch (NamingException e) {e.printStackTrace();

}

As sample explains, to use the JNDI context, you should set as context factory a JEM context implementation, by org.pepstock.jem.node.tasks.jndi.JemContextFactory class.

6.5.2 STEP-EXEC task

<step-exec> is a custom Ant task (extends Exec task) adding a set of data descriptions and locks functions (not data sources, only for java). Using this way, the programmer is able to develop own job batch programs, avoiding strong references with physical files but logical ones, by environment variables. To reduce the risk to use variables already use on system, all variables names have DD_ prefix. <step-exec> is able to execute every kind of executable program, without taking care about programming language (for instance COBOL, Perl, C or C++, Python, Bash scripts).

Here is a simple sample how to execute a COBOL program which can read a data description copying records to another data description:

JCL<taskdef name="step-exec" classname="org.pepstock.jem.ant.tasks.StepExec"></taskdef>

<step-exec executable="cmd" dir="CobolDirectory"> <arg value="/c"></arg> <arg value="myCobol.exe"></arg>

<datadescription name="FILEOUT" disposition="NEW"> <dataset name="gdg.jemtest(1)"></dataset>

ANT as JCL 17/51


</datadescription>

<datadescription name="FILEIN" disposition="SHR"> <dataset name="gdg.jemtest(0)"></dataset> </datadescription></step-exec>

COBOL main program

IDENTIFICATION DIVISION.PROGRAM-ID. MYCOBOL.AUTHOR.

ENVIRONMENT DIVISION.CONFIGURATION SECTION.SPECIAL-NAMES. C01 IS CANALE1 DECIMAL-POINT IS COMMA.SOURCE-COMPUTER. IBM.INPUT-OUTPUT SECTION.FILE-CONTROL.

SELECT FILEIN ASSIGN DD_FILEIN. SELECT FILEOUT ASSIGN DD_FILEOUT.DATA DIVISION.FILE SECTION. FD FILEIN LABEL RECORD STANDARD RECORDING MODE IS F BLOCK 0. 01 REC-FILEIN PIC X(100). FD FILEOUT LABEL RECORD STANDARD RECORDING MODE IS F BLOCK 0. 01 REC-FILEOUT PIC X(100). 01 SW-EOF PIC X(01) VALUE SPACE. 88 EOF VALUE SPACE. 88 NOT-EOF VALUE 'X'. PROCEDURE DIVISION. OPEN INPUT FILEIN OUTPUT FILEOUT. READ FILEIN AT END DISPLAY 'EMPTY FILE' SET EOF TO TRUE NOT AT END SET NOT-EOF TO TRUE END-READ.

ANT as JCL 18/51


PERFORM UNTIL EOF WRITE REC-FILEOUT FROM REC-FILEIN READ FILEIN AT END SET EOF TO TRUE NOT AT END SET NOT-EOF TO TRUE END-READ END-PERFORM.FINE.

CLOSE FILEIN. CLOSE FILEOUT. STOP RUN.

6.6 UtilitiesWith JEM, you could find some common tasks which are helpful for your JCLs. ANT has got a lot of tasks already prepared and ready to use for many purposes but all of them doesn't have any integration with data descriptions, data sources and locks.For these reasons, JEM provides:

•NullTask : do nothing! Just exiting in RC=0!

•WaitTask : waits for seconds (passed as argument) and ends in RC=0

•CopyTask : copies one or more files into another one

•SortTask : sorts a file into another one

•ShellScriptTask : executes the script commands defined in task. JEM provides some OOTB language based task, extending the generic ShellScriptTask, as:

•BashScriptTask : executes BASH script commands

•WindowsScriptTask : executes WINDOWS script code

•PerlScriptTask : executes Perl code

•PythonScriptTask : executes Python code

•Procedure and ProcedureDefinition : definitions of a task which contains other tasks. It can be imported and override from task reference in the target

6.6.1 NULL task

NullTask is a ANT task that doesn't do anything, without any statement in its main method. It could be useful because it performs all locks for datasets defined in the task, as <step-java> does.

Here is a sample:

ANT as JCL 19/51



<taskdef name="iefbr14" classname="org.pepstock.jem.ant.tasks.utilities.NullTask"/>

<target name="null"> <iefbr14/></target>

6.6.2 WAIT task

WaitTask is a ANT task that wait for some seconds before to end in return code 0. The amount of seconds to wait must be passed as argument to main program. If the parameter is missing, WaitTask waits for forever and just canceling it could end. It could be useful because it performs all locks for datasets defined in the task, as step-java does.

Here is a sample:

<taskdef name="wait" classname="org.pepstock.jem.ant.tasks.utilities.WaitTask" /><target name="wait"> <wait> <arg value="600" /> </wait></target>

6.6.3 COPY task

CopyTask is a ANT task that copies one or more datasets into another one. The input datasets must be defined in a data description called INPUT and the output dataset must be defined in a data description called OUTPUT.

Here is a sample:

<taskdef name="icegener" classname="org.pepstock.jem.ant.tasks.utilities.CopyTask"/>

<target name="icegener"> <icegener>

ANT as JCL 20/51


<datadescription name="OUTPUT" disposition="NEW"> <dataset name="gdg/jemtest(1)"></dataset> </datadescription>

<datadescription name="INPUT" disposition="SHR"> <dataset name="gdg/jemtest(0)"></dataset> </datadescription> </icegener></target>

6.6.4 SORT task

SortTask is a ANT task that sorts the content of one or more datasets coping into another one. The input datasets must be defined in a data description called INPUT and the output dataset must be defined in a data description called OUTPUT.

Here is a sample:

<taskdef name="sort" classname="org.pepstock.jem.ant.tasks.utilities.SortTask"/> <target name="sort"> <sort> <datadescription name="OUTPUT" disposition="NEW"> <dataset name="${jem.data}/gdg1/jemtest(1)"></dataset> </datadescription> <datadescription name="INPUT" disposition="SHR"> <dataset name="${jem.data}/gdg1/jemtest(0)"></dataset>

</datadescription></sort>

</target>

6.6.5 ShellScript task

ShellScript is a ANT task that is able to execute code in script language, put in task definition..

Here is a sample:

<taskdef name="shellscript" classname="org.pepstock.jem.ant.tasks.utilities.ShellScriptTask"/> <target name="step"> <shellscript shell="cmd.exe" suffix=".bat"> <arg value="/c"></arg> <arg value="call"></arg> dir ${jem.data} </shellscript></target>

6.6.6 Windows task

Windows is a ANT task that is able to execute windows scripting code, put in task definition..

Here is a sample:

<taskdef name="windows" classname="org.pepstock.jem.ant.tasks.utilities.scripts.WindowsScriptTask"/> <target name="step1"> <windows> <datadescription name="OUTPUT" disposition="NEW"> <dataset name="gdg/jemtest(1)"></dataset> </datadescription> <datadescription name="INPUT" disposition="SHR"> <dataset name="gdg/jemtest(0)"></dataset> </datadescription> copy %DD_INPUT% %DD_OUTPUT% </windows></target><target name="step2" depends="step1"> <windows> <datadescription name="OUTPUT" disposition="NEW" sysout="true"> </datadescription> <datadescription name="INPUT" disposition="SHR"> <dataset name="*.step1.windows.OUTPUT"></dataset> </datadescription> sort %DD_INPUT% /O %DD_OUTPUT% /+10

ANT as JCL 22/51


</windows></target>

6.6.7 Procedure and Procedure Definition task

Procedure Definition is a ANT task that is able to define a task inside of ANT project (by ANT import task as well). This task is not executed but only defined and it can be related to a task in a target to be executed. It can have a task definition so if more than one task is necessary, is enough to define all tasks inside of ANT Sequential task .

The Procedure Definition must have a name attribute, unique inside of ANT project. This is used by Procedure to create the link with definitions.

Procedure is a ANT task that is related to a Procedure Definition (by name ) and is able to execute all tasks defined in. It can also override data description definition, where the tasks are data description container (StepJava or StepExec and their extensions).

Here is a sample of Procedure Definition:

<taskdef name="procdef" classname="org.pepstock.jem.ant.tasks.ProcedureDefinition"></taskdef> <procdef name="wait"> <wait> <arg value="15"></arg> <lock name="block"></lock> </wait></procdef> <procdef name="echoandcopy"> <sequential> <echo> Embed another:${user.name} </echo> <echo> Embed another:${jem.classpath} </echo> <windows name="first"> <datadescription name="OUTPUT" disposition="NEW"> <dataset name="gdg/jemtest(1)"></dataset> </datadescription> <datadescription name="INPUT" disposition="SHR"> <dataset name="${jem.data}/gdg/jemtest(0)"></dataset> </datadescription>

ANT as JCL 23/51

http://ant.apache.org/manual/Tasks/sequential.html

https://ant.apache.org/manual/Tasks/import.html

https://ant.apache.org/manual/Tasks/import.html


copy %DD_INPUT% %DD_OUTPUT% </windows> <windows id="second"> <datadescription name="OUTPUT" disposition="NEW"> <dataset name="gdg/jemtest(1)"></dataset> </datadescription> <datadescription name="INPUT" disposition="SHR"> <dataset name="${jem.data}/gdg/jemtest(0)"></dataset> </datadescription> copy %DD_INPUT% %DD_OUTPUT% </windows> </sequential></procdef>

Here is a sample of Procedure, which is related to previous sample:

<taskdef name="procedure" classname="org.pepstock.jem.ant.tasks.Procedure"></taskdef> <target name="step1"> <procedure name="wait"></procedure></target>  <target name="step2" depends="step1"> <procedure name="echoandcopy"> <windows name="first"> <datadescription name="INPUT" disposition="SHR"> <dataset name="gdg/jemtest(-1)"></dataset> </datadescription> </windows> <windows id="second"> <datadescription name="INPUT" disposition="SHR"> <dataset name="gdg/jemtest(-1)"></dataset> </datadescription> </windows> </procedure></target>

ANT as JCL 24/51


7 ANT advanced utilities (Enterprise Edition Only)

JEM provides a set of utilities to manage internal entities using a batch, with a great benefit to perform many statements and then handle massive changes with a unique step.

ANT advanced utilities are not free to use but a license is necessary. All tasks need a license attribute with a file name which contains a valid license key. The file must exist in the persistent folder defined for JEM node.

Here is a sample:

<taskdef name="task" classname="org.pepstock.jem.ant.[...]Task"/> <target name="step1">

<task license=”2013-03-03_Pepstock_JEM_AntUtil.license”>… … … … </task>

</target>

These utilities are custom ANT task (and then ANT JCL must be used) and they can manage:

• GDG

• Nodes

• Roles

• Common resources

• Statistics

7.1 GDG UtilityA generation data group (GDG) is a collection of historically related sequential data sets that are arranged in chronological order. That is, each data set is historically related to the others in the group. They have sequentially ordered absolute and relative names that represent their age. Older data sets have smaller absolute numbers.

The relative name is a signed integer used to refer to the latest (0), the next to the latest (-1), and so forth, generation.

The group of datasets (usually called root) is a properties file that contains the list of generations and files related to it.To guarantee the consistency of file, JEM asks to GRS to lock both the root file.

The utility is able:

• to create a new GDG, initializing the root.properties and optionally creating a first empty generation 0

• to rebuild the root.properties if some inconsistent information are present there, aligning the content of folder with properties file

ANT advanced utilities (Enterprise Edition Only) 25/51


• to cleanup the folder of oldest files

The task parses a list of commands (separated by ; ) found inside a data description called COMMAND (be careful because is case sensitive).

7.1.1 Defining GDG

Before using a GDG it's mandatory to have the folders and root otherwise a exception will occur.The command syntax is following:

gdg-ddname is the data description name where a GDG name is indicated (with disposition NEW) and will be created as a folder. With argument NOEMPTY , a empty generation is created as well.A complete sample is following:

<project name="CREATEGDG" default="step1" basedir="."> <description> CREATEGDG </description> <property name="jem.job.name" value="CREATEGDG"></property> <property name="jem.environment" value="ENV1"></property> <taskdef name="gdg" classname="org.pepstock.jem.ant.GDGTask"/> <target name="step1"> <gdg license=”2013-03-03_Pepstock_JEM_AntUtil.license”> <datadescription name="GDG1" disposition="NEW"> <dataset name="gdg/test1"></dataset> </datadescription> <datadescription name="GDG2" disposition="NEW"> <dataset name="gdg/test2"></dataset> </datadescription> <datadescription name="COMMAND" disposition="SHR"> <dataset> DEFINE GDG GDG1; DEFINE GDG GDG2 NOEMPTY; </dataset> </datadescription> </gdg>



</target> </project>

7.1.2 Rebuilding GDGs

It could be happen that root.properties file could contain wrong information comparing with the file system.The command syntax is following:

The REBUILD command will align the data deciding what master data source to use. The master data sources could:

• ROOT when the information inside of root.propeties leads the alignment, creating nonexistent files or removing files not in root (is the default)

• GENERATIONS when the information inside of folder of GDG leads the alignment, removing wrong keys of root.propeties and adding new ones

gdg-ddname is the data description name where a GDG name is indicated (with disposition OLD).A complete sample is following:

<project name="REBUILDGDG" default="step1" basedir="."> <description> REBUILDGDG </description> <property name="jem.job.name" value="REBUILDGDG"></property> <property name="jem.environment" value="ENV1"></property>

<taskdef name="gdg" classname="org.pepstock.jem.ant.GDGTask"/> <target name="step1"> <gdg license=”2013-03-03_Pepstock_JEM_AntUtil.license”> <datadescription name="GDG1" disposition="OLD"> <dataset name="gdg/test1"></dataset>> </datadescription>



<datadescription name="GDG2" disposition="OLD"> <dataset name="gdg/test2"></dataset> </datadescription> <datadescription name="GDG3" disposition="OLD"> <dataset name="gdg/test3"></dataset> </datadescription> <datadescription name="COMMAND" disposition="SHR"> <dataset> REBUILD GDG GDG1; REBUILD GDG GDG2 MASTER(ROOT); REBUILD GDG GDG3 MASTER(GENERATIONS); </dataset> </datadescription> </gdg> </target></project>

7.1.3 Cleaning GDGs

It deletes the oldest files of a GDG.The command syntax is following:

The number-versions is the amount of generations that you want to maintain in your folder. For example, using the number '2' it removes the oldest generations, leaving only 2 files. gdg-ddname is the data description name where a GDG name is indicated (with disposition OLD).A complete sample is following:

<project name="CLEANUPGDG" default="step1" basedir="."> <description> CLEANUPGDG </description> <property name="jem.job.name" value="CLEANUPGDG"></property> <property name="jem.environment" value="ENV1"></property> <taskdef name="gdg" classname="org.pepstock.jem.ant.GDGTask"/> <target name="step1"> <gdg license=”2013-03-03_Pepstock_JEM_AntUtil.license”> <datadescription name="GDG1" disposition="OLD"> <dataset name="gdg/test1"></dataset> </datadescription> <datadescription name="COMMAND" disposition="SHR">



<dataset> CLEANUP GDG GDG1 VERSIONS 2; </dataset> </datadescription> </gdg> </target></project>

7.2 Nodes UtilityWith nodes utility you can start and drain nodes by batch, using wild cards to perform on a group of nodes.

The commands syntax is following:


Here is a JCL sample:

<project name="NODES" default="step1" basedir="."><description>

NODES action</description>

<property name="jem.job.name" value="NODES"/><property name="jem.job.environment" value="ENV-1"/>

<taskdef name="nodes" classname="org.pepstock.jem.ant.NodesTask" />

<target name="step1"><nodes license="Pepstock_JEM_GRS.license">

<dataDescription name="COMMAND" disposition="SHR"><dataSet>



DRAIN *; START *;DRAIN 10.*;

</dataSet></dataDescription>

</nodes></target>

</project>

7.3 Roles UtilityThis utility is able to manage all security entities, so roles, permissions and users by a batch and it's very helpful when massive definitions must be done.

In details, the utility is able:

• to create and delete roles

• to grant and revoke permissions to roles

• to add and delete users to roles


7.3.1 CREATE and REMOVE roles

It's possible to create and remove roles by their names (comma-separated).The commands syntax is following:

If you try to add an existing role or to remove an nonexistent one, an exception occurs.



7.3.2 GRANT and REVOKE permissions to roles

It's possible to grant and revoke permissions, by their name or wild card (comma-separated), to roles, by their name (comma-separated).


If you try to grant or revoke an invalid permission or if roles don't exist, an exception occurs.

7.3.3 ADD and DELETE users to roles

It's possible to add and delete users, by their name or wild card (comma-separated), to roles, by their name (comma-separated).


If you try to add or delete user to nonexistent role, an exception occurs.

Here is the complete JCL sample:

<project name="ROLES" default="step1" basedir=".">

<property name="jem.job.name" value="ROLES"/><property name="jem.job.environment" value="ENV-1"/>



<taskdef name="roles" classname="org.pepstock.jem.ant.RolesTask" />

<target name="step1"><roles license="Pepstock_JEM_GRS.license">

<dataDescription name="COMMAND" disposition="SHR"><dataSet>

CREATE Role;GRANT jobs:* TO Role;REVOKE jobs:* TO Role;ADD usr1, usr2 TO Role;DELETE usr2 FROM Role;REMOVE Role;


</roles></target>

</project>

7.4 Common-resource UtilityThis utility is able to manage common resources which are used by data sources in JCL for java program.

The common resources are defined and reported by a XML language as following:

<resources><resource name="FTPlocalhost" type="ftp">

<property name="url" visible="true" override="false">ftp://localhost:2121

</property><property name="username" visible="true" override="false">

admin</property><property name="password" visible="false" override="false"

hash="8c6976e5b5">51898c5cb75e3

</property></resource>

</resources>

The <resources> element is the optionally root of common-resources definitions. It can contains 1 or more <resource> element. If it is missing, that means one only resource is defined or reported and the root element is simply <resource>.

The <resource> element must have 2 mandatory attributes:

• name: name or common-resource, used inside of JCL to reference the data source

• type: type of resource, based on it a JNDI reference and resource factory must be implemented

The <resource> element can have 1 or more <property> elements needed to create the data


ftp://localhost:2121/


source at run-time. The <property> element can have 4 attributes:

• name: mandatory name of property.

• visible: optional boolean attribute which defined if it's readable or it must be encrypted or decrypted. To do it, it's necessary to access to web application and take the value encryption and hash to check if the value is correct. Default is true.

• override: optional boolean attribute which defines if the attribute could be override at run-time inside of JCL (for instance the FTP binary property). Default is true.

• hash: optional attribute which is used when you try to set or get a property with visible parameter set to false. To do it, it's necessary to access to web application and take the value encryption and hash to check if the value is correct.

In details, the utility is able:

• to get one or more common resources definition, saving to a dataset

• to remove a common resource by name

• to add and update a common resource


7.4.1 Getting common-resources

It's possible to get a common resource definition by its name (an exception occurs if the common resources doesn't exist).

The command syntax is following:

By default (if FILE command is missing), the common resources definition is reported in the OUTPUT data description. If FILE is present, the definition is written inside of data description related by name. If NOENCRYPTION attribute is present, the properties of common resources defined as visible=false , are written without any encryption.


<project name="RES" default="step1" basedir=".">



<property name="jem.job.name" value="RES"/><property name="jem.job.environment" value="ENV-1"/>

<taskdef name="resources" classname="org.pepstock.jem.ant.CommonResourcesTask"/>

<target name="step1"><resources license="Pepstock_JEM_GRS.license">

<dataDescription name="OUTPUT" sysout="true" disposition="MOD"/><dataDescription name="my" sysout="true" disposition="MOD"/><dataDescription name="COMMAND" disposition="SHR">

<dataSet>GET FTPlocalhost;GET FTPlocalhost FILE OUTPUT;GET FTPlocalhost FILE OUTPUT NOENCRYPTION;GET FTPlocalhost FILE my;GET FTPlocalhost FILE my NOENCRYPTION;


</resources></target>

</project>

It's possible to get a all or a subset of resources definitions by wild card of their types and names.


By default (if FILE command is missing), the common resources definitions are reported in the OUTPUT data description. If FILE is present, the definitions are written inside of data description related by name. If NOENCRYPTION attribute is present, the properties of common resources defined as visible=false , are written without any encryption.






<dataDescription name="OUTPUT" sysout="true" disposition="MOD"/>



<dataDescription name="my" sysout="true" disposition="MOD"/><dataDescription name="COMMAND" disposition="SHR">

<dataSet>GETLIST *;GETLIST * FILE OUTPUT;GETLIST * FILE OUTPUT NOENCRYPTION;GETLIST * FILE my;GETLIST * FILE my NOENCRYPTION;

GETLIST *.F*;GETLIST *.F* FILE OUTPUT;GETLIST *.F* FILE OUTPUT NOENCRYPTION;GETLIST *.F* FILE my;GETLIST *.F* FILE my NOENCRYPTION;

GETLIST ftp.*;GETLIST ftp.* FILE OUTPUT;GETLIST ftp.* FILE OUTPUT NOENCRYPTION;GETLIST ftp.* FILE my;GETLIST ftp.* FILE my NOENCRYPTION;



</project>

7.4.2 Remove common-resource

It's possible to remove a common resource definition by its name (an exception occurs if the common resources doesn't exist).







<dataDescription name="COMMAND" disposition="SHR">



<dataSet>REMOVE FTPlocalhost;



</project>

7.4.3 Setting common-resources

It's possible to add common resource definitions (if the common resources exist will be updated).


By default (if SOURCE command is missing), the common resources definitions must be present in the INPUT data description. If SOURCE is present, the definitions must be located inside of data description related by name. If NOENCRYPTION attribute is present, the properties of common resources defined as visible=false , are read without any decryption.






<dataDescription name="myNoEnc" disposition="SHR"><dataset>

<![CDATA[<resources> <resource name="FTPlocalhost" type="ftp"> <property name="url" visible="true"override="false"> ftp://localhost:2121 </property> <property name="username" visible="true" override="false"> admin </property> <property name="password" visible="false" override="false"> admin </property>




</resource></resources>]]>

</dataset></dataDescription>

<dataDescription name="my" disposition="SHR"><dataset>

<![CDATA[<resources> <resource name="FTPlocalhost" type="ftp"> <property name="url" visible="true"override="false"> ftp://localhost:2121 </property> <property name="username" visible="true" override="false"> admin </property> <property name="password" visible="false" override="false" hash="8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918"> 51898c5cb75e3d81a9736567637d9b6ef6942e492987e7d5801c9893b0e40c8a </property> </resource></resources>]]>

</dataset></dataDescription><dataDescription name="INPUT" disposition="SHR">

<dataset><![CDATA[<resources> <resource name="FTPlocalhost" type="ftp"> <property name="url" visible="true"override="false"> ftp://localhost:2121 </property> <property name="username" visible="true" override="false"> admin </property> <property name="password" visible="false" override="false" hash="8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918"> 51898c5cb75e3d81a9736567637d9b6ef6942e492987e7d5801c9893b0e40c8a </property> </resource></resources>]]>

</dataset></dataDescription><dataDescription name="COMMAND" disposition="SHR">

<dataSet>SET ;SET SOURCE my;SET SOURCE myNoEnc NOENCRYPTION;


</target></project>





7.5 Statistics UtilityThis utility is able to manage to extract statistics, which are stored by JEM in file system changing file every day, to be stored in data warehouse infrastructure for data analytical and performance and capacity management purposes.

JEM doesn't have any goal to maintain and do this kind of work, so an external infrastructure must be in place.

The target of this utility is to read XML(not-well formed because there's any any root element) files, creates java object and calls a transformer and loader class passing the object. Transformer and loader contains all logic to manage the data.

Java object, read by utility, is org.pepstock.jem.node.stats.Sample and , using all public methods of it, you could access to following data:

Field Type Scope Description

Key String General Unique key of sample, built on timestamp. A sample is taken EVERY minute. Format is “yyyy-MM-dd HH:mm”

Date String General Date when sample has been created. Format is “yyyy-MM-dd”.

Time String General Time when sample has been created. Format is “HH:mm”.

Environment String General JEM environment where the node is in

Member-Key String Node Unique key for JEM node. This is a UUID representation.

Member-Label String Node Name of JEM node, composed by ip-address and listening port.

Pid (process id) Long Node Process ID (based on system) of JEM node

Number-Of-JCL-Check Long Node Number of JCL taken by JEM node for JCL checking, during last minute

Number-Of-JOB-Submitted Long Node Number of JOB submitted and executed by JEM node, during last minute

Total-Number-Of-JCL-Check

Long Node Total amount of JCL taken by JEM node for JCL checking, since JEM node start




Total-Number-Of-JOB-Submitted

Long Node Total amount of JOB submitted and executed by JEM node, since JEM node start

CPU User Long Machine Total amount of milliseconds of CPU used for USER purposes since machine starting

CPU System Long Machine Total amount of milliseconds of CPU used for SYSTEM purposes since machine starting

Idle Long Machine Total amount of milliseconds spent by machine in IDLE since machine starting

CPU Total Long Machine Total amount of milliseconds of CPU used since machine starting

CPU % User Double Machine Percentage of CPU used for USER purposes since last minute

CPU % System Double Machine Percentage of CPU used for SYSTEM purposes since last minute

% Idle Double Machine Percentage of time spent in IDLE since last minute

Memory Available Long Machine Total amount of memory (in byte) available

Memory Used Long Machine Total amount of memory (in byte) used

Memory Free Long Machine Total amount of memory (in byte) free

CPU User Long Node Total amount of milliseconds of CPU used for USER purposes by JEM node since node starting

CPU System Long Node Total amount of milliseconds of CPU used for SYSTEM purposes by JEM node since node starting

CPU Total Long Node Total amount of milliseconds of CPU used since machine starting

CPU % Total Double Node Percentage of CPU used by JEM node since last minute

Memory Used Long Node Total amount of memory (in byte) used

Name String HazelcastMap Name of Hazelcast map

Owned Entry Count Long HazelcastMap Number of map entries owned by JEM node




Backup Entry Count Long HazelcastMap Number of map entries of backup hold by JEM node

Owned Entry Memory Cost Long HazelcastMap Amount of memory cost (number of bytes) of owned entries in JEM node

Backup Entry Memory Cost Long HazelcastMap Amount of memory cost (number of bytes) of backup entries in JEM node

Locked Entry Count Long HazelcastMap Number of currently locked locally owned keys

Lock Wait Count Long HazelcastMap Number of cluster-wide threads waiting to acquire locks for the locally owned keys

Dirty Entry Count Long HazelcastMap Number of entries that JEM node owns and are dirty (updated but not persisted yet)

Hits Long HazelcastMap Number of hits (reads) of the locally owned entries

Number Of Gets Long HazelcastMap Number of get operations of JEM node

Number Of Puts Long HazelcastMap Number of put operations of JEM node

Number Of Removes Long HazelcastMap Number of remove operations of JEM node

Number Of Events Long HazelcastMap Number of events received in JEM node

Number Of Other Operations

Long HazelcastMap Total number of other operations in JEM node

Total Get Latency Long HazelcastMap Total latency (in milliseconds) of get operations in last minute

Total Put Latency Long HazelcastMap Total latency (in milliseconds) of put operations in last minute

Total Remove Latency Long HazelcastMap Total latency (in milliseconds) of remove operations in last minute

When the utility is able to read a sample object, it calls a class which must implement org.pepstock.jem.node.stats.TransformAndLoader . This interface is called when a statistics file is open and when it is close, when a sample is created correctly and when an error occurs reading a record to create a sample.

/** */public void fileStarted(File file) throws Exception;



/** */public void loadSuccess(Sample sample) throws Exception;

/** */public void loadFailed(String record, int line, Exception exception) throws Exception;

/** */public void fileEnded(File file) throws Exception;

The utility accepts 3 arguments, using the arg element:

• -days n : because JEM writes a file per day, with this argument it's possible to read the files created in the date calculated subtracting the value to current date.

Example: -days -1 is yesterday

• -date yyyyMMdd : because JEM writes a file per day, with this argument it's possible to read the files created in the date passed as argument. If -days argument is used, you can't use -date .

Example: -date 20030101 is January 1st, 2003.

• -class [full-class-name] : to manage the stored sample, it's necessary to implement transformer and loader interface and put the name of class in this argument. It's a mandatory arguments, so an exception occurs if is missing

Example: -class org.pepstock.jem.node.stats.DefaultTransformAndLoader is default and out-of-the-box loader class, that displays all sample and removes the files readed when is closed, to clean up the folder where JEM nodes store statistics.


<project name="STATS" default="step1" basedir=".">

<property name="jem.job.name" value="STATS"/><property name="jem.job.environment" value="ENV-1"/>

<taskdef name="stats" classname="org.pepstock.jem.ant.StatsCollectTask" />

<target name="step1"><stats license="Pepstock_JEM_GRS.license">

<arg value="-days"/><arg value="-1"/><arg value="-class"/><arg value="org.pepstock.jem.node.stats.DefaultTransformAndLoader"/>

</stats></target>

</project>



8 Spring Batch as JCLMany applications within the enterprise domain require bulk processing to perform business operations in mission critical environments. These business operations include automated, complex processing of large volumes of information that is most efficiently processed without user interaction. These operations typically include time based events (for instance month-end calculations, notices or correspondence), periodic application of complex business rules processed repetitively across very large data sets (or instance insurance benefit determination or rate adjustments), or the integration of information that is received from internal and external systems that typically requires formatting, validation and processing in a transactional manner into the system of record. Batch processing is used to process billions of transactions every day for enterprises.Spring Batch is a lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. Spring Batch builds upon the productivity, POJO-based development approach, and general ease of use capabilities people have come to know from the Spring Framework, while making it easy for developers to access and leverage more advanced enterprise services when necessary.Spring Batch provides reusable functions that are essential in processing large volumes of records, including logging/tracing, transaction management, job processing statistics, job restart, skip, and resource management. It also provides more advanced technical services and features that will enable extremely high-volume and high performance batch jobs through optimization and partitioning techniques. Simple as well as complex, high-volume batch jobs can leverage the framework in a highly scalable manner to process significant volumes of information.SpringBatch is part of Spring .

8.1 Properties definitionTo use Spring Batch inside JEM, Spring Batch XML JCL must contain the mandatory properties that JEM needs, using a specific bean by class org.pepstock.jem.springbatch.JemBean and setting the id attribute to jem.bean .

Properties are described as following:

• Job Nameis optional string property, called for JemBean jobName .

If missing, name attribute of job element will be used. If both are missing, an exception occurs.

• Environment

is optional string property, called for JemBean environment .

If missing, the JEM node environment definition is used.

• Domain

is optional string property, called for JemBean domain .

Spring Batch as JCL 42/51

http://static.springsource.org/spring-batch/

http://http//www.springsource.org/







• Affinity

is optional string property, called for JemBean affinity .


• Useris optional string property, called for JemBean user .

If missing, default value ( null ) will be used. An exception occurs if the user, who submitted the job, is not authorized to change the user job execution.

• Locking Scope

is optional string property, called for JemBean lockingScope .

If missing, default value ( job ) will be used. If the value is not equals to one the possible values (job, step ), an exception occurs.

• Hold

is optional boolean property, called for JemBean hold .

If missing, default value ( false ) will be used.

• Priority

is optional integer property, called for JemBean priority .

If missing, default value ( 10 ) will be used.

• Memory

is optional integer property, called for JemBean memory .

If missing, default value ( 256 ) will be used. Be aware the unit is MegaBytes.

• Classpath

is optional string property, called for JemBean classPath .

Defining a classpath, it will be able to extend the classpath of JVM where SpringBatch is running. The value is a string and the files are separated by ; . If the file doesn't represent an absolute path, the JEM will use relative position from jem.classpath folder. You could use variables that JEM substitutes in string value. If missing, default value ( null ) will be used.


is optional string property, called for ANT emailsNotification . If missing, default value ( null ) will be used.

A example usage of org.pepstock.jem.springbatch.JemBean is following:

<beans:bean id="jem.bean" class="org.pepstock.jem.springbatch.JemBean"> <beans:property name="jobName" value="JOB1"/>



<beans:property name="environment" value="ENV1"/> <beans:property name="domain" value="domain"/> <beans:property name="affinity" value="classA"/> <beans:property name="user" value="newUser"/> <beans:property name="hold" value="true"/> <beans:property name="lockingScope" value="job"/> <beans:property name="priority" value="99"/> <beans:property name="memory" value="1024"/> <beans:property name="classPath" value="${jem.classpath}/jdbc.jar"/> <beans:property name="emailsNotification" value="[email protected];[email protected]"/></beans:bean>

To work correctly inside JEM (it works anyway but without some JEM job management features), it's mandatory to inherit your job element to jem.baseJob job one and the steps to jem.baseStep , both defined as abstract.

This is mandatory because it's mandatory to add org.pepstock.jem.springbatch.StepListener to listen job and steps start and end, so JEM is aware about the status of executed job. If the listener is not defined, the job is executed anyway but without all details about steps execution. The id jem.listener is suggested, to avoid overriding with other step listener.

<job id="jem.baseJob" abstract="true"> <listeners> <listener ref="jem.listener"></listener> </listeners></job> <step id="jem.baseStep" abstract="true"> <tasklet> <listeners> <listener ref="jem.listener"></listener> </listeners> </tasklet></step>

The job name is extracted from JCL reading the id attribute of <job> element and must be equals to property of org.pepstock.jem.springbatch.JemBean , as following:

<job id="JOB1" parent="jem.baseJob"> <step id="step0" parent="jem.baseStep"> <tasklet ref="test"></tasklet> </step></job>

8.2 Data Descriptions and Datasets in SpringBatchData descriptions and datasets are implemented by specific beans. Because of all in Spring is a bean, data descriptions could be defined using org.pepstock.jem.springbatch.tasks.DataDescription bean as following:



<beans:bean id="id-data-description" class="org.pepstock.jem.springbatch.tasks.DataDescription">

<beans:property name="name" value="name-data-description"/><beans:property name="sysout" value="true/false"/>

<beans:property name="disposition" value="NEW/OLD/MOD/SHR"/><beans:property name="datasets">

<beans:list>… … …

</beans:list></beans:property>

</beans:bean>

The id attribute of main bean is used only by Spring Batch but necessary to relate data descriptions to tasks and or beans.

Data description needs a mandatory name property. This name is used inside the business logic to access to datasets by JNDI so this name must be unique in the task definition. The sysout property is optional one that if there is means that the data description is a sysout.

The disposition property is optional and must be a valid value, one of the following string enumeration: NEW, MOD, OLD or SHR.

Datasets are children of a data description definition and identifies a file or data necessary for business logic. Datasets could be defined as following:

<beans:bean class="org.pepstock.jem.springbatch.tasks.DataSet"><beans:property name="name" value="file-name"/><beans:property name="datasource" value="datasource-name"/>

</beans:bean>/** * Inline dataset */ <beans:bean class="org.pepstock.jem.springbatch.tasks.DataSet">

<beans:property name="text" value=" Here I am This is a DD"/></beans:bean>

Datasets bean must have the name attribute (by name property) and which could represent (by value property):

• file name (absolute or relative path), composed by SpringBatch properties if necessary. If it's a relative path, JEM adds the content of jem.data variable, used to identify the global file system with all data.

• GDG file name, following the same rules of a normal file name (see previous item).

• Temporary prefix file name

If the name attribute is missing and text is present, a temporary file is created, putting the content of value property attribute.

Datasets could have the datasource attribute which could represent a relation with a data source, previously defined. It works only the FTP resources and the name of dataset represents the file to manage by FTP.



The reference dataset is not implemented in Spring Batch because with usage of bean it's enough to pass the id of the bean to all steps which need it.

8.3 Data sources in SpringBatchData sources are implemented by specific tags. They could be defined as following:

<beans:bean id="my-db" class="org.pepstock.jem.springbatch.tasks.DataSource"> <beans:property name="name" value="my-db"></beans:property> <beans:property name="resource" value="resource-db"></beans:property></beans:bean> <beans:bean id="my-ftp" class="org.pepstock.jem.springbatch.tasks.DataSource"> <beans:property name="name" value="my-ftp"></beans:property> <beans:property name="resource" value="my-resource-ftp"></beans:property> <beans:property name="properties"> <beans:list> <beans:bean class="org.pepstock.jem.springbatch.tasks.Property"> <beans:property name="name" value="binary"></beans:property> <beans:property name="value" value="true"></beans:property> </beans:bean> </beans:list> </beans:property></beans:bean>

The id attribute of bean is used only by Spring Batch but necessary to relate data sources to tasks and or beans.

Data source needs a mandatory name attribute. This name is used inside the business logic to access to data source (for every kind of data source, as jdbc or ftp) by JNDI so this name must be unique in the task definition. Data source needs a mandatory resource attribute. This name is used to locate the common-resource, defined previously in JEM. If resource is not defined, an exception occurs.

Data source element can contain one or more property element to override at runtime the value of some properties defined to create a data source. The property element needs a mandatory name attribute. An exception could occur if you try to override a property created in JEM with override="true" typically used for password, user-id or URL.

8.4 Locks in SpringBatchLocks are implemented by specific tags. They could be defined as following:

<beans:bean id="id-lock" class="org.pepstock.jem.springbatch.tasks.Lock"><beans:property name="name" value="unique-name-for-env" />

</beans:bean>

The id attribute of bean is used only by Spring Batch but necessary to relate locks to tasks and or beans.



Lock needs a mandatory name attribute. This name is used to create a lock inside the JEM environment in exclusive. This name must be unique in the task definition.

8.5 TaskletsThe tasklet is a simple interface that has one method, execute , which will be a called repeatedly by the TaskletStep until it either returns RepeatStatus.FINISHED or throws an exception to signal a failure. Each call to the Tasklet is wrapped in a transaction. Tasklet implementers might call a stored procedure, a script, or a simple SQL update statement.

JEM prepared a own tasklet implementation, org.pepstock.jem.springbatch.tasks.JemTasklet which implements the execute method in final way and it calls another method that the developer must implement. The method looks like as following:

/** * Is abstract method to implement with business logic, where it's possible * to access to resources by JNDI. * * @param stepContribution step contribution * @param chuckContext chunk context * @return status of execution * @throws Exception if errors occur */public abstract RepeatStatus run(StepContribution stepContribution,

ChunkContext chuckContext) throws Exception;

A sample how to define a onw tasklet with data descriptions is following:

<beans:bean id="my-tasklet" class="org.pepstock.test.springbatch.MyTaskLet"> <beans:property name="dataDescriptionList"> <beans:list> <beans:ref local="my-single"></beans:ref> </beans:list> </beans:property></beans:bean>

Here is java code of tasklet sample:

public RepeatStatus run(StepContribution stepContribution,ChunkContext chuckContext) throws Exception {

Hashtable<String, String> env = new Hashtable<String, String>();env.put(Context.INITIAL_CONTEXT_FACTORY,


try {InitialContext context = new InitialContext(env);/** single file input stream **/Object object = (Object) context.lookup("my-single");FileInputStream inputStream = (FileInputStream) object;



/** reads input stream **/Scanner sc = new Scanner(inputStream);sc.useDelimiter("\n");while (sc.hasNext()) {

String record = sc.next().toString();System.out.println(record);

}sc.close();

} catch (Exception ex){ex.printStackTrace();

}return RepeatStatus.FINISHED;

}

8.6 ChunksSpring Batch uses a 'Chunk Oriented' processing style within its most common implementation. Chunk oriented processing refers to reading the data one at a time, and creating 'chunks' that will be written out, within a transaction boundary. One item is read in from an ItemReader, handed to an ItemProcessor, and aggregated. Once the number of items read equals the commit interval, the entire chunk is written out via the ItemWriter, and then the transaction is committed.

JEM prepared a ItemWriter and a ItemReader, to be used inside of Spring Batch JCL, which items are able to read and write datasets defined inside of data description.They are org.pepstock.jem.springbatch.items.DataDescriptionItemReader and

org.pepstock.jem.springbatch.items.DataDescriptionItemWriter .

They use a delegate to read and write, implemented in org.pepstock.jem.springbatch.items.SimpleFileItemReader

and org.pepstock.jem.springbatch.items.SimpleFileItemWriter .Data sources and locks concepts are not available with chuck approach.A sample how to use chunk with data description is following:

<beans:bean id="delegateItemReader"

class="org.pepstock.jem.springbatch.items.SimpleFileItemReader"></beans:bean> <beans:bean id="delegateItemWriter"

class="org.pepstock.jem.springbatch.items.SimpleFileItemWriter"></beans:bean> <beans:bean id="itemReader"

class="org.pepstock.jem.springbatch.items.DataDescriptionItemReader"> <beans:property name="dataDescription"> <beans:ref local="input"></beans:ref> </beans:property> <beans:property name="delegate" ref="delegateItemReader"></beans:property></beans:bean>





<beans:bean id="itemWriter"

class="org.pepstock.jem.springbatch.items.DataDescriptionItemWriter"> <beans:property name="dataDescription"> <beans:ref local="output"></beans:ref> </beans:property> <beans:property name="delegate" ref="delegateItemWriter"></beans:property></beans:bean>

A sample how to use configure chuck in the step element is following:

<job id="JOB1" parent="jem.baseJob"> <step id="step0" parent="jem.baseStep"> <tasklet> <chunk reader="itemReader" writer="itemWriter"></chunk> </tasklet> </step></job>

8.7 UtilitiesInside the JEM, you could find some common tasklets which are helpful for your JCLs. JEM provides:

• NullTask : do nothing! Just exiting in RC=0!

• WaitTask : waits for seconds (passed as argument) and ends in RC=0

• CopyTask : copies one or more files into another one

8.7.1 NULL tasklet

NullTasklet is a SpringBatch tasklet that doesn't do anything, without any statement in its main method. It could be useful because it performs all locks for datasets defined in the tasklet.

Here is a sample:

<beans:bean id="null" class="org.pepstock.jem.springbatch.tasks.utilities.NullTasklet"></beans:bean> <job id="NULL" parent="jem.baseJob"> <step id="nothing" parent="jem.baseStep"> <tasklet ref="null"></tasklet> </step></job>



8.7.2 WAIT tasklet

WaitTasklet is a SpringBatch tasklet that wait for some seconds before to end in return code 0. The amount of seconds to wait must be passed as argument to main program. If the parameter is missing, WaitTasklet waits for forever and just canceling it could end. It could be useful because it performs all locks for datasets defined in the tasklet.

Here is a sample:

<beans:bean id="wait" class="org.pepstock.jem.springbatch.tasks.utilities.WaitTasklet"> <beans:property name="seconds" value="60"></beans:property></beans:bean> <job id="WAIT" parent="jem.baseJob"> <step id="waiting" parent="jem.baseStep"> <tasklet ref="wait"></tasklet> </step></job>

8.7.3 COPY tasklet

CopyTasklet is a SpringBatch tasklet that copies one or more datasets into another one. The input datasets must be defined in a data description called INPUT and the output dataset must be defined in a data description called OUTPUT.

Here is a sample:

<beans:bean id="icegener" class="org.pepstock.jem.springbatch.tasks.utilities.CopyTasklet"> <beans:property name="dataDescriptionList"> <beans:list> <beans:ref local="INPUT"></beans:ref> <beans:ref local="OUTPUT"></beans:ref> </beans:list> </beans:property></beans:bean><beans:bean id="INPUT" class="org.pepstock.jem.springbatch.tasks.DataDescription"> <beans:property name="name" value="INPUT"></beans:property> <beans:property name="disposition" value="SHR"></beans:property> <beans:property name="datasets"> <beans:list>



<beans:bean class="org.pepstock.jem.springbatch.tasks.DataSet"> <beans:property name="name" value="gdg/jemtest(0)"/> </beans:bean> <beans:bean class="org.pepstock.jem.springbatch.tasks.DataSet"> <beans:property name="text" value=" These records are added to OUTPUT file: Record1 test abcdefghjklilmnopqrstuvzxw Record2 test abcdefghjklilmnopqrstuvzxw Record3 test abcdefghjklilmnopqrstuvzxw"/> </beans:bean> </beans:list> </beans:property></beans:bean> <beans:bean id="OUTPUT" class="org.pepstock.jem.springbatch.tasks.DataDescription"> <beans:property name="name" value="OUTPUT"></beans:property> <beans:property name="disposition" value="NEW"></beans:property> <beans:property name="datasets"> <beans:list> <beans:bean class="org.pepstock.jem.springbatch.tasks.DataSet"> <beans:property name="name" value="gdg/jemtest(+1)"/> </beans:bean> </beans:list> </beans:property></beans:bean> <job id="ICEGENER" parent="jem.baseJob"> <step id="copy" parent="jem.baseStep" next="step2"> <tasklet ref="icegener"></tasklet> </step></job>

Date post:	18-Mar-2018
Category:	Documents
Upload:	doannhu
View:	227 times
Download:	1 times

JCL REFERENCE - Pepstock.org - JCL Reference.pdf · JEM is able to manage different JCL by a JEM...

Documents