WebSphere Java Batch - IBM · IBM WebSphere Java Batch is a functional solution that is built on...

1

WebSphere Java Batch

WP101783 at ibm.com/support/techdocs

Version Date: September 11, 2012

2

2

IBM WebSphere Java Batch

WP101783 at ibm.com/support/techdocs© 2012, IBM Corporation

• Business Pressures on Traditional Batch

• IBM WebSphere Java Batch Overview

• IBM WebSphere Java Batch Feature Focus

• IBM WebSphere Java Batch for z/OS Focus

• IBM WebSphere Java Batch Deployment Scenarios

•Wrap-Up Summary

Agenda

This presentation is divided into six sections as shown on this chart.

The first section -- "Business Pressures" -- provides some explanation of the business pressures facing batch processing. This sets the context for the information on WebSphere Java Batch that follows.

The next section -- "WebSphere Java Batch Overview" -- provides a framework of understanding for what "WebSphere Java Batch" is and how it operates.

The third section -- "Feature Focus" -- takes a closer look at some key functions provided with WebSphere Java Batch. This provides a better sense for the capabilities of the function.

The fourth section -- "z/OS Focus" -- provides information on how WebSphere Java Batch for the z/OS platform provides z/OS-specific benefits to batch processing.

The fifth section -- "Deployment Scenarios" -- paints a picture of several ways in which WebSphere Java Batch might be deployed and used.

The final section -- "Summary" -- provides a high-level wrap-up of the information provided in the earlier parts of the presentation.

3

Business Pressures on Traditional Batch

Or, "Hey, who closed my batch window?"

4

4



Concept of "Dedicated Batch" Window Going Away

24 x 7 x 365 AccessUsers of your online systems expect availability at all hours

Users from other parts of the world means availability is expected around the clock

Mobile UsersUsers are no longer tied to a desk and a computer. Today users have access to mobile computing devices that are with the user wherever they may be.

Day or night, home or office.

Online

Batch

Online

Batch

In the past ... Today ...

Windows of time which used to be dedicated to batch processing are shrinking. The demands of online processing require more and more ...

The need to process batch work has not gone away.

The need to perform the work concurrent with OLTP has emerged.

There used to be a time when "batch" and "online" processing were quite separate from one another. It used to be the case that online processing was stopped altogether so batch processing would have access to the system resources to complete its work.

But those days are well behind us. Online processing is becoming more and more a 24 x 7 operation. This is because client access is increasingly global, with people from different time zones seeking access at all times during the day. There may be times during the day when online processing is greatly reduced, but in most cases it is never stopped altogether. That means batch processing much work at the same time as online. To the degree it has dedicated time, that window of time is shrinking.

The advent of mobile devices means client access is now even more frequent than before. The transaction work that flows back to information processing systems because of mobile device activity is increasing. And mobile device activity may occur at any time of night or day.

It's truly a 24 x 7 world for online processing. But batch processing has not gone away. There are still requirements to do batch work. So it's evident batch and online must be done at the same time ... and in a manner that implies both work cooperatively.

5

5



The Value of Shared ServicesIt's not just that the window is shrinking ... it's also the cost pressures on maintaining the batch and OLTP environments:

Efficiencies through consolidation around common assets

Batch Infrastructure

OLTP Infrastructure

OLTP Applications

OLTP

Development Tools

Batch Applications

Batch

Development Tools

Homegrown Middleware

Infrastructure

Batch Support Staff OLTP Support Staff

Batch + OLTPCommon Infrastructure

Batch + OLTPCommon Tooling

Common Support Staff

Homegrown Middleware

Infrastructure

Shared Java

Assets

OLTP

Applications

Batch

Applications

In addition to the batch window shrinking, there are pressures to contain and reduce costs related to information processing as well. Having separate infrastructures for online and batch ... separate computers, separate tools, separate support staff ... is an idea that is increasingly being drawn into question.

Further, custom-built mechanisms for running batch -- particularly Java batch -- represent middleware solutions that require a considerable amount of effort to maintain and support.

The trend is definitely in the direction of convergence. Online and batch processing are both information processing, and as such can and should be handled in a cooperate, converged manner. This allows for efficiencies in infrastructure, staff, development tooling, and potentially Java assets shared between OLTP and batch. Homegrown middleware solutions are not favored and will eventually be phased out.

6

6



Java for Batch Processing?Yes ... for many very good reasons:

Java is a registered trademark of Oracle

Availability of SkillsJava is a programming language with wide adoption in the industry. Skills for Java programming are common and affordable.

Tooling SupportDevelopment tooling for Java has advanced to the point where some tools (IBM Rational Application Developer) are very powerful andsophisticated.

This also provides an opportunity to consolidate to a common tooling environment for both OLTP and batch development.

z/OS Specialty EnginesPressures on cost containment often dictate greater use of z/OS specialty engines. Java offloads to zAAP. Java batch does as well.

Processing in OLTP RuntimeRunning Java batch in the same execution runtime as Java OLTP provides an opportunity to mix and manage the two processing types together under the same management model.

Which brings us to Java as a programming environment for batch processing. Some may ask, "Why use Java when other languages -- COBOL, for instance -- work well?"

First, the question is not Java or COBOL, but not both. There is a very practical realization in the world that batch assets written in COBOL are still very useful. To the extent Java enters the discussion, it is as a complement to COBOL, not a total replacement.

Second, there are business pressures that make Java an attractive batch solution:

• Skills -- Java skills are simply more plentiful than COBOL skills. That's a fact. Many organizations have very good Java development staffs. It makes sense to leverage those Java staffs for batch work as well.

• Tooling -- today's development tooling products are powerful and quite sophisticated. Acquiring them also represents an investment in technology and an investment in skills. It makes good sense to leverage that investment across multiple information processing disciplines.

• Specialty Engines -- COBOL runs on general processors, Java runs on specialty engines. Specialty engines (zAAPs) provide a way to lower overall System z costs ... both in acquisition costs as well as ongoing licensing costs. Specialty engines are an attractive solution and leveraging them for batch work is desirable.

• Cooperative Processing -- online processing runtime infrastructures are very often designed around Java. For example, IBM's WebSphere Application Server is a powerful Java processing runtime. There is an investment in maintaining that online infrastructure. So to the extent that same investment can be leveraged for batch work as well, the better.

The bottom line is that Java for batch processing is very much an area of growing interest, and in fact is in use in many very large processing operations.

7

7



The Objective -- OLTP and Batch Mixed and Managed:OLTP and Batch do not need to be "either / or" ... it can be "both":

With IBM WebSphere Batch this is possible. OLTP and Batch processing within a common execution runtime

(WebSphere Application Server) allows the WAS platform to mix and manage the two workload types.

11:00pm Midnight 1:00a 2:00am 3:00am

OLTP ProcessingBatch Processing Batch Processing

OLTP Processing Batch

Batch OLTP Batch OLTP Batch

Batch Processing OLTP Processing

OLTP Batch OLTP Processing Batch OLTP

Compute

Processing

Resources

The picture on this chart represents an objective of cooperative processing of online and batch within same execution runtime -- the managed intermixing of online work with batch work. This is possible when the execution runtime is a multi-threaded design (which IBM WebSphere Application Server is), and the execution runtime is designed to manage the many processes simultaneously (which IBM WebSphere Application Server does).

IBM WebSphere Java Batch is a functional solution that is built on top the proven runtime platform of IBM WebSphere Application Server (WAS for short). WAS is today very much capable of handling concurrent online processes simultaneous. To process batch work all WAS needs is an understanding that batch is present, and a container design for handling batch work.

With that we're ready to go into what IBM WebSphere Java Batch is.

8

OverviewA high-level look at the IBM WebSphere

Java Batch model

9

9



IBM Compute Grid V8 and IBM WAS V8.5The IBM WebSphere Java Batch function is provided in two ways today:

IBM WebSphere

Compute Grid

Version 8

IBM WebSphere

Application Server

Version 7 or 8

Operating Systems Supported:AIX, IBM i, Linux, Windows, HP-UX,

Solaris, Linux for System z, z/OS

Add the function ("Augment")

IBM WebSphere

Application ServerVersion 8.5

Operating Systems Supported:AIX, IBM i, Linux, Windows, HP-UX,

Solaris, Linux for System z, z/OS

Compute Grid V8 function

incorporated into WAS V8.5Java Batch Function

Java

Execution

Runtime

Function is identical between the two environments

Compute Grid V8 available for those who have not yet migrated their execution runtimes to WAS V8.5

We're going to start by getting on the table a key bit of information about how IBM WebSphere Java Batch is delivered ... even before we explain what exactly it is.

IBM WebSphere Java Batch is delivered as part of two different products at this time:

• IBM WebSphere Compute Grid V8

• IBM WebSphere Application Server Version 8.5

Key Point: the Java batch function provided in each is identical.

Here's a quick background on the history of the function to set some context as to why it is presently offered in two different products:

• Originally the Java Batch function was provided as part of a product called "WebSphere Extended Deployment," or XD. XD consisted of Java Batch, another function called "Virtual Enterprise," and a third function called "eXtreme Scale."

• The three functions were then split out into separate products so customers could better choose which function they wished to employ. That was the start of WebSphere Compute Grid, which was the Java batch function. That exists today in the form of WebSphere Compute Grid V8.

• Compute Grid has always relied upon an existing WebSphere Application Server runtime to operate. Compute Grid is really a set of Java batch function that is added to a WAS runtime foundation. That act of adding the function is called "augmenting," which is a fancy word for incorporating the Compute Grid function into the configuration of the WAS runtime.

• When the WAS V8.5 product was in the planning stages, the decision was made to roll the Compute Grid (and Virtual Enterprise) function into the base WAS foundation.

So that's where we are today -- Compute Grid V8, which requires an existing WAS V7 or V8 runtime onto which it is "augmented," or WAS V8.5 which has the Compute Grid function rolled into the WAS base.

10

10



Batch Container Added to the WAS RuntimeAt a very high-level, you may think the IBM WebSphere Java Batch function as a "batch container" operating alongside the other containers of WAS itself:

Container-managed Services

Web Container

Application Web Modules


EJB Container

Application EJB Modules


Batch Container

Batch Applications

WebSphere Application Server Runtime Environment

Batch job dispatching and

management system

Job resiliency services

(skip record, step retry)

Data record read and

write support services

Parallel job management

and execution services

Checkpoint and job

restart servicesCOBOL module call

services

The WebSphere Application Server product provides what's known as a "Java EE" (or "Enterprise Edition") runtime. Java EE is an industry standard specification for an application server that provides a long list of standard application specifications. Part of the design of this "Java EE" runtime is the idea of a "container." Containers are simply runtime functions that provide managed services to the programs that run in the containers. Container-managed services mean the application programs themselves may focus on their core business logic and not have to implement common functions. WAS by itself has two containers -- Web and EJB. The addition of the WebSphere Java Batch function adds a third -- a batch container.

Like other containers, the batch container provides function services to the batch applications that run in the container. The chart shows a summary listing of some of those services. The purpose of this presentation is to provide more information about those services. For the purposes of this chart, just scan through the list of services and think about a Java batch program having access to those services. Then think about what it might take if you had to write those services yourself. But of course you don't have to write them yourself -- they're provided with the batch container.

11

11



Overview of the Management and Execution Model This picture illustrates some of the key components of the WebSphere Java Batch model as provided in Compute Grid V8 and WAS V8.5:

Job

Dispatching

FunctionJob Properties

Declaration File

Job Execution

Endpoint

Batch Applications

Job Execution

Endpoint

Batch

Applications

Development Libraries and

Tooling Support

Job Management

Console1

2

3

4

5

1. Job Management Console (JMC) provides a view into the batch

environment and allows you to submit and manage jobs

2. Job declaration file (xJCL) provides information about the job to be

run, such as the steps, the data input and output streams and the

batch class files to invoke

3. The Job Dispatching function interprets the xJCL, dispatches the

job to the endpoint where the batch application resides, and

provides ability to stop and restart jobs

4. The Execution Endpoint is a WAS server in which the deployed

batch applications run

5. The development libraries and tooling assist in the creation of the

batch applications

A comprehensive Java batch execution platform

Built on the proven Java runtime environment of WebSphere Application Server

Now we'll start to go a bit deeper into the technical specifics of WebSphere Java Batch.

What this chart is saying -- at the very highest level -- is that WebSphere Java Batch provides a separation of several key elements of batch processing:

• Job submission and dispatching -- the act of having a Java batch job begin the process of execution. This is done through a defined interface.

• Job description -- to avoid hard-coding job properties into the application code, it's always best to externalize that information. An external job properties file is used to tell the job submission function what the job is and how to run it.

• Job execution -- the execution of the job takes place in an "endpoint," which is really a WAS application server where the Java batch application is deployed. The Job Dispatching function signals to the endpoint to begin execution of the batch code named in the job declaration file.

• Job development and deployment -- Java batch applications implement the batch logic, and they are deployed into the endpoint servers like any WAS application is deployed. Those Java batch application class files are invoked when the dispatching function requests it, based on the submission of a job declaration file.

The numbers on the chart correspond to the notes on the chart. If you're familiar with batch processing on z/OS with things like JCL and JES, then this should look familiar. The concepts are the same, because the concepts are still very much applicable even though it's Java rather than compiled COBOL.

Key Point: this function separation of the "job" from the "batch application," and the separation of "dispatching" from "execution," provides considerable flexibility as you'll see.

12

12



Batch Job and Batch Job StepsA batch job consists of one or more steps executed in order specified in xJCL:

xJCL

Properties of the overall job

Job Step 1• Java class

• Input and output declarations

• Other properties of the step

Job Step 2• Java class



Job Step n• Java class



Job The xJCL is submitted through the Job Management ConsoleInterfaces provided: HTTP browser, command Line, Web Services, RMI

The Job Dispatching function interprets xJCL and determines which endpoint has batch application class files deployed

Dispatching Function invokes job and passes to the endpoint an object containing all the properties in xJCL

Steps are executed in order, with conditional step processing if declared

Dispatching Function maintains awareness of job state

When job ends, job output file accessible through Job Management Console

A Java batch "job" is a logical collection of one or more "steps" that are executed in the order specified in the job declaration file (xJCL). The Java code that does the batch processing is named in the xJCL as part of the step declaration.

Along with the Java class file that implements the Java batch step, the xJCL may also specify job step properties, such as where to get data and where to write data to.

The xJCL is passed into the Job Dispatching Function as part of job submission. The dispatching function reads the xJCL to determine what the job involves, then it signals to the batch endpoints where the Java batch application is deployed to begin execution.

Note: this does not involve launching a JVM. The JVM is already active as part of the WAS runtime. This is a signal for the JVM to begin execution of a deployed class file.

As part of the dispatching to the endpoint, the dispatching function passes over an object that contains all the properties that were specified in the xJCL. Those properties are available to Java batch job as it executes. For example, the properties that specify the JDBC data source to use to access DB2 to get data. The Java batch code may abstract all of that and rely on the actual data being part of the properties object passed to the batch container at time of job execution.

The steps are processed in the order specified in the xJCL. And yes, conditional step processing is supported, so if the first step fails it's quite possible to have the second step not execute. During job execution the dispatching function is well aware of the state of the job. The batch endpoints are maintaining state information in a relational table that is accessible by the dispatching function.

When the job ends, the job output log is accessible through the Job Management Console, where it may be viewed or downloaded.

13

13



Job Execution "State"The following picture illustrates a simplified view of the job states ... it helps illustrate a key point: executing jobs can be acted upon; failed jobs restarted.

Submitted

Executing

Ended

Restartable

Stop or Cancel

Problem

Restart

The Job Management Console provides you ability to act upon an executing job

The Batch Container is maintaining checkpoint status and will restart at the last checkpoint interval

This is possible because of the Java batch runtime services that are part of

the batch container modelIf you were to write this yourself then just what's shown here would require a significant amount of custom batch middleware code. IBM

WebSphere Java Batch provides that as part of the product.

In a perfect world, a job would go through three states -- submitted, executing, and successful end. But we don't have a perfect world -- things sometimes go wrong. Or, you may wish to act upon a batch job during execution.

So to that end WebSphere Java Batch provides the ability -- through the Job Management function -- to see the state of a job and to act upon it if you choose.

For example, imagine a batch job that is submitted and dispatched to an endpoint. That job enters step 1 and something goes wrong. WebSphere Java Batch will put that job into a "restartable" state. You may then inspect the job log, determine what went wrong, correct that problem, and restart the job. WebSphere Java Batch will then start the job from where it left off.

Note: or, more precisely, at the last good checkpoint interval. We'll talk about checkpoint processing in a bit.

Aside from cases where something goes wrong, you may wish to stop, suspend or cancel a job during execution. You may do that as well through the Job Management function. The container will cease execution at the next checkpoint and put the job into a restartable state.

This is an execution model considerably more sophisticated than a simple process of launching a JVM and having it run some Java "main" type program. The function we've described here --which is only a small set of the total WebSphere Java Batch function -- would take a great deal of time and effort for you to code yourself. And if you did code it yourself you'd have a custom middleware function you'd then have to maintain and support. But thankfully you don't have to create and maintain custom middleware because IBM provides it as a supported middleware function in WebSphere Java Batch.

14

14



Batch Data Stream Framework (BDSF)This is a key function service provided by the batch container - it abstracts data read and write operations so your code may focus on the business logic:

Batch Data Stream Framework

Supplied "patterns" for data access:

• JDBC read or write operations

• JPA read or write operations

• File read or write operations

• z/OS Data Set read or write operations

Your Java class that implements the supplied framework and provides the

specific data access logicExample: SQL query for JDBC

Your job step Java class, which implements the business logic

required for the batch processingData object

passed based on

your mapping in

BDSF class

Batch Data Stream retrieves result set from data persistence store (DB, file, etc.)

Batch Data Stream maps data fields to data object

For each record in result set, BDSF invokes your job step, passing a data object mapped to your specifications

Your job step code stays focused on business logic, not Java stream handling and data object formatting

The "Batch Data Stream Framework" (BDSF) is a significant piece of the provided services for batch programming. Its purpose is to provide a container function for data access so your Java batch logic does not have to deal with all the lower-level stream handling. The BDSF provides a set of popular "patterns" for data access -- JDBC, JPA, UNIX file and z/OS data set read and write operations.

You implement a BDS (batch data stream) by extending the Java class files provided with WebSphere Java Batch. This is where you provide such customization such as the SQL query strings to use to fetch the result set from the JDBC database. You name your customized instance of a BDS in the xJCL for a job step. A job step may have both an input and output function, so that job step would have two BDS declarations -- one for input, and one for output.

The batch data stream will perform the read operation and then iterate through the result set. For each iteration it will map the data record to a data object you define, then invoke your batch step logic and pass in the data record object. Imagine a SQL query that returns 1000 customer records. The BDS will then take the first record, map the customer data into a data object you define, then call your batch step, which will process the record (and likely call an output BDS). The input BDS will then get control back, map the second record to a data object, and call your batch step. The process repeats until the result set is completed.

This takes the data handling away from your batch processing logic. Your batch processing logic does what it needs to do (calculate tax, determine interest earned, calculate penalties, etc.), but it does not have to handle getting the records and mapping the data. That's all handled by the BDS that you implement from the supplied class file with the necessary customization to fit your environment.

We bring up the BDSF because (a) it's a significant functional service of WebSphere Java Batch, and (b) many of the other services are defined and operate at the BDSF level. We'll see that coming up.

15

15



Integration with Enterprise Scheduler FunctionsThe Job Dispatching Function has a Message Driven Bean (MDB) interface. IBM supplies a program that integrates schedulers with WebSphere Java Batch:

Enterprise SchedulerExample: IBM Tivoli

Workload Scheduler, CA

Workload Automation CA 7,

or BMC Control-M

WSGRID ProgramShell script, BAT file or JCL job

Input Queue

Output Queue

Message Driven Bean Interface

WSGRID is seen by Scheduler as any other batch job it starts and monitors

WSGRID interacts with Job Dispatching, submitting the job and processing Java batch job output back to STDOUT or JES Spool if z/OS

WSGRID program stays up for life of job in WebSphere Java Batch

To the Scheduler, WGRID is the Java Batch job ... but behind WSGRID is all the WebSphere Java Batch function we'll discuss

WebSphere MQ or the integrated Default Messaging of WAS

Enterprise scheduling functions, such as IBM's Tivoli Workload Scheduler, may integrate with the WebSphere Java Batch function by using a supplied program called "WSGRID." The scheduler submits the JCL (or invokes the shell script or BAT file; all forms of submission are supported) that starts the WSGRID program. WSGRID then turns and submits the Java batch job to WebSphere Batch by using the Message Driven Bean (MDB) interface of the Job Dispatching function.

Key Point: WSGRID stays active for the duration of the Java batch job execution in WebSphere Java Batch. The enterprise scheduler knows nothing of the Java batch execution. It sees WSGRID. WSGRID receives back from WebSphere Java batch the output from executing job. When the Java batch job completes, that information feeds back to WSGRID, which then completes.

In summary, WSGRID uses message queuing to interact with WebSphere Java Batch. A message flows from WSGRID to WebSphere Java Batch to submit the job; many messages flow back to WSGRID so the Java batch output can be spooled back to STDOUT of the WSGRID job. On z/OS that means the Java batch output goes to JES spool, which means normal archiving mechanisms may archive the Java batch output.

The enterprise scheduler sees WSGRID, which is like a mirror of the Java batch job executing in WebSphere. When WSGRID ends, it means the Java batch job has ended.

16

Feature FocusA closer look at some of the features and functions of the IBM WebSphere

Java Batch model

17

17



Transactional Checkpoint ProcessingThe batch container provides the ability to checkpoint at intervals based on either record count or time. The container keeps track of last checkpoint.

Batch Container

Java Batch

Application

xJCL says:

Checkpoint = 5

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Commit Processing

Last good checkpoint persisted

Checkpoint interval (record or time) specified in the xJCL

This is a function of the batch container, not your application code

As checkpoint intervals are reached, container commits and records the checkpoint attained

In the event of a failure, job may be restarted at the last good checkpoint

Set the checkpoint interval based on your knowledge of balance between recoverability and efficiency

Another service of the batch container is checkpoint processing. This allows long transactional batch jobs to have a point to fall back to in the event of an error (or you stopping the job), and it gives the container a point to restart from when the job is restarted.

It's important to understand your batch step code does not do any of this. This is handled by the container based on definitions in the xJCL. In the xJCL you specify the type of checkpoint policy (record-based, or time-based), and the value to use (for example, checkpoint every 100 records).

The chart shows an example where the checkpoint interval is 5. That means every 5 records the container will process commit and store the checkpoint value for the job step. If the job does fail and is restarted, then the container looks in the database table and restarts from that point forward.

Setting the checkpoint interval is a balance between granular recoverability (a lower checkpoint interval) and efficiency (a higher checkpoint interval).

18

18



Skip-Record ProcessingProvides a container-managed way of tolerating data read or write errors so the job itself may continue on. Information about data errors may be logged.

Batch Data Stream Framework

Java Batch

Application

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

Data Record

xJCL tells BDSF:

• How many data read

or write exceptions to

consume

• What exceptions to

consider for skip-

record processing

• Alternatively, what

exceptions to exclude

from skip-record

processing

Objective: allow job to continue if a data read or

write exception occurs in BDSFWhy fail a million-record job just because of one or two read or write

exceptions? Better to complete the job and allow auditors to go back

and investigate the few exceptions.

Skip-Record processing allows BDSF to keep exception and not surface it to your applicationThis takes burden off your application code to explicitly handle data

read or write exceptions that may occur

A "skip-record listener" may be called so your code may lot information about skipped recordMore on "batch listeners" coming up

xJCL properties allow you to specify how many records may be skipped and what exceptions to

include or exclude from consideration

When skip limit is reached, further exceptions are

surfaced to application. That may result in job failing and going into a restartable stateNormal restart-at-checkpoint would occur

Skip-record processing is a function added to Compute Grid V8 and WAS V8.5. It provides a means of defining container behavior when a read or write operation by the Batch Data Stream Framework (BDSF) experiences an exception.

Without this function, exceptions thrown by the BDSF will bubble up to the batch application. If the exception is not explicitly handled, the job will go into a restartable state. The job can then be restarted, but that implies manual intervention and the loss of time.

The objective of this function is to allow transaction batch jobs to survive the occasional error. With this function enabled, an exception that occurs on a BDSF read or write operation is "consumed" by the BSDF ... that is, not surfaced up to the application. The job survives the exception.

Properties specified in the xJCL indicate how many records may be skipped. Beyond that limit no skipping takes place ... exceptions after that limit is reached are surfaced up to the application. Other properties in the xJCL allow you to specify what exceptions to include for skip-record consideration or, alternatively, which records to exclude from consideration.

It's important to point out that there is a mechanism to log information about records being skipped. It's called a "skip-record listener," which is a mechanism where custom code you provide is called each time a record is skipped. Your code may then log specific information about the exception and the record skipped. Your auditors may then review the skipped records after the job completes and take appropriate action to complete the skipped records.

19

19



Retry-Step ProcessingProvides a means of retrying a job step in the event of an exception thrown. If successful on retry then the job continues and your processing completes.

xJCL tells Container:

• How many step retries may be

attempted

• What exceptions to consider for

retry-step processing

• Alternatively, what exceptions to

exclude from retry-step processing

• Whether to process a delay before

attempting a retry of the step

Objective: retry step in attempt to allow overall job

to continue and complete when an unanticipated

exception is thrown

This is at level higher than skip-record ... this is if

an unhandled exception is thrown when the job

step function is called

Batch container falls back to last good checkpoint

and restarts from there

A "retry-step listener" may be called so you can

perform custom action upon retry-step processingMore on "batch listeners" coming up

xJCL properties allow you to specify how many retry attempts will be performed and what

exceptions to include or exclude from

consideration

When retry limit is reached, job will go into restartable stateNormal restart-at-checkpoint would occur

On exception, retry

up to n times

Retry-step processing is a little like "skip-record" in that the objective is to provide the batch job some resiliency to survive exceptions that pop up during execution. But in this case the focus is exceptions that occur at a higher level -- that is, exceptions that come up during execution of the batch step itself (as opposed to exceptions on read or write operations, which occur down in the Batch Data Stream Framework code).

Here again, properties in the xJCL allow you to specify how many times a batch step may be retried, and which exceptions to include or exclude from consideration. You may also code how long to delay before retrying a batch step.

Imagine a batch step is executing and during one of the invocations of processJobStep() an

exception is thrown. If you have a greater-than-zero retry-step property coded in the xJCL then the container will, upon exception, roll back to the most recent checkpoint and then retry the step starting from the checkpoint.

As was the case with "skip-record," there is a "listener" that gets invoked with retry-step as well. This allows some code you write to be called at two points -- onError() and onRetry(). Here

you may log information about the exception and take whatever other actions you feel are appropriate.

20

20



Batch "Listeners"These are callout points where your customer "listener" code will be called when key events occur. The callouts are managed by the batch container:

Job Listener• Callouts occur:

Start of the job; Start of each step; End of each step; End of job

• Register your code to container with property in xJCL

• Use this to perform any special setup or cleanup actions at those points in the lifecycle of a batch job

Retry-Step Listener• Callouts occur:

When the exception is thrown; When the retry is attempted

• Register your listener with code in application createJobStep()

method

• Use this to take action at these points, such as logging information about the exception and the point in the processing where it occurred

Skip-Record Listener• Callouts occur:

On skipped read or skipped write operation

• Register your listener with code in application createJobStep()

method

• Use this to take action at these points, such as logging information about the exception and the record skipped

Job Listener

Retry-Step Listener

Skip-Record Listener

Listeners provide ability to have your code called at key points during batch job execution

Several times now we've mentioned "listeners" that get invoked at certain points of batch processing. Listeners are implemented with code you write which you "register" to the batch environment so it knows what to call when those call-out events take place.

There are three listeners we'll focus on -- Job listeners, Retry-Step listeners, and Skip-Record listeners.

• Job listeners -- this is invoked at the start of the job, and the start and end of each batch step, and at the end of the job. You register your listener class file by coding a property in the xJCL. At those points during batch job execution the container will call the methods for that lifecycle point. Your code then does what you determined was appropriate for that point in the job lifecycle ... setup work, cleanup work, logging ... whatever.

• Retry-Step listeners -- this is invoked whenever a job step is retried using the new Retry-Step processing function. The listener is called when an exception is thrown that triggers Retry-Step processing, and again just as the step is being retried. You register your listener with a simple line of code in the batch application itself.

• Skip-Record listeners -- this is invoked whenever a record is skipped using the new Skip-Record processing function. The listener is called when an exception is thrown that triggers Skip-Record processing. Your code may then log information about the record that has been skipped and the exception that took place. You register your listener with a simple line of code in the batch application itself.

Listeners provide you a way to insert custom processing logic into the batch job lifecycle at key points during execution.

21

21



Parallel Job ManagerThe Parallel Job Manager (PJM) provides a way to "parameterize" logic so parallel sub-jobs may act on a slice of the overall batch job data:

One job processing 1M customer records

1 - 100K

100K - 200K

900K - 1M

Ten sub-jobs acting on a 1/10th slice of data each

Sub-job

Sub-job

Sub-job

or

Time = 0 Time = 1 Time = 10

Objective is reduction in overall job completion timeWhich shortens overall batch window if other

jobs are dependent on this job for completion

xJCL specifies whether job is to be run in parallel, and if so how:

• One JVM, multiple threads

• Multiple JVMs

Your "parameterizer" code is called at start so data range may be segmented into sub-job slices

Job is submitted, then PJM dispatches "sub-jobs" to act on each data range"Parameterizer" code constructs data range query strings to be used by each sub-job

PJM manages "top-job" and all subordinate "sub-jobs" to completion

The Parallel Job Manager (PJM) function provides a way to run batch jobs as parallel sub-jobs. If the data lends itself to parallel sub-jobs ... meaning, the data can be divided in such a way the sub-jobs will not encounter access contention or locking issues ... then running the jobs as parallel sub-jobs can shorten the total execution time of the overall job.

The first part of this function is a property in the xJCL that determines if the job is to be run in parallel, and if so, then how many sub-jobs should be run, and if they should be run on separate threads in a single JVM or across multiple JVMs.

The second part of this function is a call the container does to code registered in the xJCL as the "parameterizer." This is code you write that determines how the data is to be split up for each sub-job. For example, if the data involves customer numbers, then one way to split the data is by customer number ranges.

The parameterizer then returns control back to the container, which then creates the specified sub-jobs and dispatches them. Each sub-job will operate on the data range determined by your parameterizer code.

The PJM watches over each of the sub-jobs and collects up the results to form the return code of the overall job. If a sub-job goes into a restartable state, the overall job is restartable. You then restart the top-job, and the PJM manages the restart of the sub-job that was in a restartable state.

22

Java Batch on z/OSA review of what IBM WebSphere Java

Batch brings specific to z/OS

23

23



The Value Statements of WebSphere Batch on z/OSIf we start from a high level, we see the following platform benefits that accrue up to Java batch running on the platform:


WebSphere Application Server

z/OS Operating System

z/OS Parallel Sysplex

System z Hardware

Java Batch Applications

• Batch runtime services

• Batch development tooling

• Proven Java runtime environment

• WAS deployment and management model

• WAS Qualities of Service

• Decades of maturity, stability and reliability

• Consolidated operation and management model

• Rich set of system facilities: WLM, SMF, RMF, SAF

• z/OS instance clustering with central data sharing

• Elimination of single points of failure for availability

• Near linear scalability up to 32 nodes

• Engineered from beginning for reliability and stability

• Engineered for high levels of I/O

• Extremely long mean time between failure

• Speciality engines for specific work offload

• Dynamic capacity expansion

• Logical partitioning using PR/SM hypervisor

As we turn to look at the specific ways in which WebSphere Java Batch takes advantage of the z/OS platform, it's useful to step back and look at this from a broad perspective.

The chart shows the System z and z/OS environment as a layered architecture, starting from the hardware at the bottom working up through to the Java batch application on top. The bullets to the right correspond by color to the layer of the architecture in the illustration. Start at the bottom and work towards the top:

• Hardware -- the System z hardware has a long history of reliability. It is designed to have redundant and resilient components to provide for availability. It is also designed to handle high levels of data input/output, which is often the case with transactional batch processing.

• Parallel Sysplex -- this is a z/OS operating system clustering mechanism that provides a

shared data facility for the members of the cluster. The shared data mechanism, provided by the Coupling Facility, has a near-linear scalability up to the maximum 32 cluster members supported.

• z/OS Operating System -- z/OS, like the hardware on which it runs, has a long history ofreliability. Its design permits many different processes to run simultaneously Further, it has a very rich set of management facilities built up around it. It is an extremely robust and reliable operating system platform.

• WebSphere Application Server -- this provides the Java runtime foundation for WebSphere Java batch. WAS provides the underlying services key to Java batch -- data access support such as JDBC, transactional support, and clustering.

• WebSphere Java Batch -- this is the focus of this presentation ... this is what provides the runtime environment for Java batch programs.

You Java batch applications run on top this layered stack of operational benefits. That's the picture in total.

24

24



Scaling Up the Java Batch Solution on z/OSThere are several ways in which a WebSphere Java batch solution can be scaled up to provider greater batch throughput and shorter execution windows:

Controller

Region

Servant Region

Servant Region

Controller

Region

Servant Region

Servant Region

Sysplex-Enabled Data SubsystemsDB2, CICS, IMS, MQ

Sysplex-Enabled Data SubsystemsDB2, CICS, IMS, MQ

Parallel Sysplex Data Sharing

WebSphere eXtreme ScaleCaching Grid

PJM

1

3

4

5

1. VerticalWAS z/OS servant regions provide a type of "vertical cluster," giving you

additional batch compute resources

2. Capacity on DemandCPU processors may be dynamically

added to a z/OS LPAR, increasing the

capacity for processing work

3. HorizontalWAS z/OS clustering on top Parallel

Sysplex provides near-linear

scalability up to 32 nodes with a

central data sharing model

4. Parallel ProcessingThe Parallel Job Manager may be used

to partition data into sub-jobs, which

may then be run on multiple threads, different servants, or different servers

on other LPARs.

5. Data CachingWebSphere eXtreme Scale provides a

data caching grid from which Java

batch may fetch and store data

2

WebSphere Java Batch applications running on System z and z/OS have several means by which you may scale up the processing. The chart above illustrates several of the approaches:

1. With WAS z/OS the ability to create "vertical clusters" of servant regions. This provides additional JVMs in which to run Java batch, which implies additional worker threads on which to dispatch batch processes.

2. The System z hardware and z/OS operating system architecture allows CPU resources to be dynamically allocated to a logical partition, providing additional "capacity on demand" if needed.

3. Horizontal clustering for availability and scalability is provided by the WAS z/OS foundation on which Java Batch is built. WAS z/OS clustering may then take advantage of Sysplex-enabled middleware functions such as DB2 or CICS. The shared data model of Parallel Sysplex means the data accessed by Java batch is the same on any logical partition.

4. The Parallel Job Manager function discussed earlier may take advantage of vertical and/or vertical clusters.

5. Java batch applications that have intense data requirements might benefit further from taking advantage of IBM's WebSphere eXtreme Scale, which is a distributed in-memory caching grid architecture.

There are many paths to scalability provided by System z and z/OS.

25

25



WLM ClassificationThe submitted job can be tagged with a WLM "transaction class," which may then be used to map the job to a WLM Service Class or Reporting Class:

Job

Dispatching

Function

Job Execution

Endpoint

Batch Applications

Configurable rules map job submission to a "Transaction

Class" (TC) name

xJCL

TC name sent to endpoint where batch job will run

WLM "CB" subsystem rules map TC name to Service

Class and Reporting Class

z/OS WLM

Batch job runs under that Service Class and data is

gathered under the Reporting Class

Classifying to a Service Classallows WAS z/OS to place work into separate servant regions based on Service ClassA somewhat sophisticated practice not widely used

Classifying to a Reporting Classallows WLM to gather system information for all work running under that ClassA much more common practice that is very useful for understanding usage patterns and for capacity planning

One of the key points of integration with the z/OS operating system by WebSphere Application Server is with WLM. WAS z/OS is able to classify work to WLM Service Classes and Reporting Classes. Classifying to specific Service Classes allows WLM to better manage work of differing goals; classifying to specific Reporting Classes allows RMF to report on work more granularly.

The WebSphere Java Batch function on z/OS has the same ability. Configurable rules in the Job Dispatching Function allow submitted batch jobs to be classified to individual Service Classes or Reporting Classes.

With multiple Service Classes it becomes possible to have batch jobs of one service class goal be dispatched to one servant, and batch jobs of a different service class goal be dispatched to another. WLM then may manage system resources to the z/OS address space (the servant) to help meet the assigned goals.

Being able to classify batch jobs to different WLM Reporting Classes allows you to collect system resource usage information by batch job, even though multiple batch jobs may be running in the same server.

26

26



SMF 120.9 Activity RecordingWAS z/OS supports the use of activity recording using the SMF 120.9 record. WebSphere Java Batch extends the record with batch activity information:

WebSphere Java BatchCompute Grid z/OS V8

WAS z/OS V8.5

SMF Buffers and Data Sets

Job activity records allow you to understand how your system is being used and to provide chargeback data

Activity recording available on all platforms, but only z/OS uses SMF, which is an extremely efficient logging mechanism

Provides historical records for usage analysis and batch capacity planning

Information captured:• Job submitter

• Date and time of submission

• Final job state

• Total CPU used for job

• General processor used for job

• zAAP usage derived: Total - GP = zAAP

For even more granular information about batch job resource usage you can turn on SMF recording of batch activity. This is an extension of the WAS z/OS SMF 120.9 record to include batch activity information.

27

27



Use of JZOS ServicesJZOS is a set of functions that make using Java on z/OS much easier and useful. The JZOS class libraries may be used in batch application development:

Job Execution Endpoint

Batch

Applications

JZOS Libraries

z/OS

JZOS is technology acquired by IBM from Dovetail Technologies* and incorporated into z/OS**

Examples of some z/OS services available:DfSort - interface for invoking DFSORT

MvsConsole - class with static methods to interface with the MVS console.

MvsJobSubmitter - class for submitting batch jobs to JES2 or JES3 from a Java program

PdsDirectory - class for opening a PDS directory and iterating over its members.

WtoMessage - simple data object/bean for holding a WTO message and its parameters.

ZUtil - static interface to various z/OS native library calls other than I/O.

* www.dovetail.com

** http://www-03.ibm.com/systems/z/os/zos/tools/java/products/jzos/overview.html

WebSphere Java Batch and JZOS are not mutually exclusive ... the JZOS class libraries may provide exactly what you need for your batch application to access

z/OS functions and services

JZOS is a set of technology developed by Dovetail Technologies that makes running Java on z/OS must more useful. JZOS provides a set of Java interfaces that give your Java program access to z/OS services such as what's listed on the chart.

Note: JZOS also has a JVM-launcher function to run Java "main" programs in a "batch" mode.That function is not compatible with WebSphere Java Batch because the JVMs in which batch runs with WebSphere Java Batch are always-up JVMs. No need to launch a JVM. What we're talking about here are Java class libraries with accompanying native libraries that provide access to z/OS services.

To use the JZOS Java class libraries you import them into your application and STEPLIB the batch server to the native libraries.

28

28



COBOL ContainerThe COBOL Container provides a way to call and execute COBOL modules in the WAS z/OS server address space ... a very efficient way to call COBOL

1. Batch application runs in the WAS

z/OS servant region address space

2. The COBOL container is created as a

separate LE enclave in the address

space

3. COBOL DLLs are accessed using

STEPLIB or LIBPATH

4. COBOL Container code provides the

"glue" between the Java environment

and the native COBOL

5. Java batch code uses supplied class

methods to create the container and

use it

6. Call stubs provide an easy way to call

the COBOL DLL and marshal data

back and forth

7. The call stubs are generated by a

supplied utility that uses COBOL

source to understand data bindings

8. JDBC Type 2 connections created in

the Java batch program may be

shared into the COBOL module in the

COBOL Container

COBOL Container Call Code

WebSphere Java Batch Container

COBOL Container LE Enclave

COBOL Module

Call

Stubs

WAS z/OS Servant Region Address SpaceSeparate LE Enclave from COBOL Container

Call Stub

Generator

IBM Rational Application Developer

Compiled COBOL LibraryPDSE or USS Directory

1

2

3

4

5

6

7

Lines of code needed to invoke COBOL many times less than other means of calling COBOL from Java

8

The COBOL Container function provides a way to run compiled COBOL DLLs inside the WAS z/OS servant address space where the Java batch application runs. This makes calling the COBOL module much more efficient than other ways to call COBOL from WAS.

The COBOL Container is code written by IBM that provides the Java-to-native bridge. It creates a separate Language Environment (LE) enclave for the COBOL code. That insures the COBOL enclave does not "dirty" the WAS enclave, and vice-versa.

In addition, a code-generation tool called the "Call Stub Generator" makes calling the COBOL DLL and passing and getting data much easier. The Call Stub Generator operates in IBM Rational Application Developer 7.5 or later. It reads the COBOL source to understand the data input and output requirements, and it generates the Java call stub and Java data bindings automatically. Your batch application developer then uses the generated call stubs rather than having to do all that coding themselves.

This also provides the ability to create a JDBC Type 2 connection in the Java batch application and share that connection with COBOL running in the COBOL container.

Key Point: the COBOL Container provides a highly-efficient way to use COBOL assets as part of the Java batch processing in the WebSphere Java Batch environment.

29

Deployment ScenariosA review of some potential ways to deploy

the WebSphere Java Batch function

30

30



Co-Location on z/OSWith the WebSphere Java Batch function on z/OS several advantages surface:

Job Execution

Endpoint

Batch

Applications

Data

SubSystemsDB2, CICS, IMS, MQ

z/OS

Use of cross-memory connectors for high-speed and low-latency access to data

• JDBC Type 2 connector for access to DB2

• CICS Transaction Gateway (CTG) local EXCI

• WebSphere Optimized Local Adapters (WOLA)

Much more secure -- cross memory data exchanges can not be 'sniffed' or intercepted

Parallel Sysplex data sharing provides highly available clustered environment without reliance on a single instance of a data subsystem

Use of COBOL Container technology for re-use of COBOL assets in very efficient calling pattern

Use of WebSphere MQ Bindings Mode for integration with Enterprise Scheduler for very fast job submission and job output return

Reduction of per-access latency is critical when dealing with large volumes of records where job completion time is important

One deployment scenario is to have WebSphere Java Batch deployed on z/OS in the same LPAR as the data.

WebSphere Java Batch on z/OS provides several advantages based on the WAS z/OS exploitation of z/OS. It also provides an opportunity for reduced latency per transactional record which can help reduce the overall batch job execution time.

31

31



Linux for System z and Hipersocket Access to z/OS DataHipersockets is a technology that maps a TCP/IP network onto the memory backplane of a System z divided into multiple logical partitions (LPAR):


Batch

Applications

Data


z/OS LPAR Linux for System z

Memory Backplane of System z CEC

TCP/IP Network

To programs and processes that use Hipersockets it looks like a routed TCP/IP network

Advantages of Hipersockets• Secure -- does not go over adapters or external wires

• Efficient -- memory transfer speeds implies lower overall latency

Advantages of Linux for System z• Consolidation -- host many Linux images in a virtualized environment

Virtualizing on the zVM hipervisor provides a means of quickly scaling up in Linux instances to meet requirements

Locating the WebSphere Java Batch function on Linux for System z still allows very good access to z/OS data across Hipersockets, which is a TCP network mapped to the physical memory of the System z machine. Network traffic does not go out over network adapters or wires, so there is an inherent security about it. Because it is mapped to memory it is very efficient.

Linux for System z is attractive because it provides a means of consolidating many Linux images from distributed servers onto zVM. The consolidation is often attractive from a financial perspective, and management of the Linux images may be done under the umbrella of a single platform.

32

32



zEnterprise and zBXThe zEnterprise system is designed around principle of right-fit placement:


Batch

Applications

Data


z/OSzBX Virtual

Servers

10Gb Intra-Ensemble Data Network

System z LPAR serves as the anchor for a zEnterprise "node"

A zBX blade extension rack hosts IBM p or IBM x blades capable of hosting AIX, Linux or Windows virtual servers

A 10Gb network connects it all

WebSphere Java Batch endpoints may be placed where the work they do makes best sense:

• Batch processes requiring a highly available and highly secure environment may operate on z/OS

• Batch processes that use relatively more CPU may be offloaded to zBX blade servers

• WebSphere Java Batch Dispatching function would be able to "see" all the different endpoints

and dispatch based on where batch applications were deployed

IBM zEnterprise and the BladeCenter Extension (zBX) provide a means of locating WebSphere Java Batch onto a multi-OS platform with a dedicated data network. The 10Gb network that connects the virtual servers of a zEnterprise has routers or intermediate hops. Batch endpoints executing on servers in the zBX may access data on z/OS across this network.

33

Wrap-Up and Summary

34

34




Key Features:� Java Batch programming model

� Java Batch container built on WAS QoS

� Development and deployment tooling

� Batch execution environment

� Concurrent OLTP and batch workloads

� Enterprise scheduler integration

� Parallel processing of batch jobs

� Container based checkpoint and restart

� Mixed batch workloads

� COBOL support on z/OS

WebSphere Application Server v8.5 integrates capabilities from WebSphere Compute

Grid and delivers a complete enterprise level Java batch processing solution

Compute Gridcapabilitiesintegrated

into WAS 8.5

The first of three summary charts.

35

35



WebSphere Java Batch - Value Proposition

� Reliable batch infrastructure – Built on the proven Qualities of Service

delivered by WebSphere Application Server.

� Incremental modernization – Move at your pace to reduce risk.

� Resource efficiencies – Focus resources on business logic and leave the

infrastructure to WebSphere

� Enterprise integration – Integrate with existing enterprise schedulers to help

deliver a robust end-to-end solution.

� Enables new execution patterns – Dynamic OLTP and Batch runtime

environment built on WebSphere; highly parallel batch jobs; and many others.

� Supports a SOA strategy of reuse – Enable the cost effective sharing of

business logic across both the OLTP and Batch paradigms.

� Eliminate batch windows – Transition from traditional batch windows to

running batch 24x7 concurrent with OLTP.

Move batch into the WebSphere environment and integrate with OLTP to gain the

benefits of concurrent processing, shared business logic, and cost efficiencies

The second of three summary charts.

36

36



WebSphere Java Batch - Key Use Cases

� Batch Modernization – Migrate from a native batch runtime, typically developed in programming languages like C, C++, PL/I, and COBOL, to Java.

� Highly Parallel Batch Jobs – Execute a single large batch job that is broken

into chunks and executed concurrently across a grid of resources.

� Dynamic OLTP & Batch Runtime – Dynamically provision resources for

execution to meet operational goals.

� Batch as a Service – Expose business capabilities as a service and leverage

usage accounting features for tracking and chargeback.

� Replace Homegrown Batch Frameworks – Eliminate costly proprietary

batch infrastructures and focus development resources on business logic.

� Shared business logic across OLTP and Batch – Leverage the proven

WebSphere platform to share logic across both batch and OLTP.

Evolve to a single infrastructure for both OLTP and Batch that enables you to leverage existing applications and focus resources on business logic

The final summary chart.

Date post:	23-Mar-2020
Category:	Documents
Upload:	others
View:	12 times
Download:	0 times

WebSphere Java Batch - IBM · IBM WebSphere Java Batch is a functional solution that is built on...

Documents