Dynamic Data Exchanges with the Java Flow Processor

Presenter: Scott Bowers

Date: April 25, 2007

2

History of Java Flow Processor (JFP)

Developed in 2004 for NJDEP to facilitate dynamic data publishing and dynamic data submittal services

Used at New Jersey DEP to support Facility Registry System (FRS) submittal and query services

(production), RCRA Handler and Permitting submittal and query services

(production) Used at Mississippi DEQ to support

FRS submittal and query services (testing only) Use at Kentucky DEP to support

RCRA Handler, Permitting, and Corrective Action (testing only) This utility is freely available to any Exchange Network

partner!

3

What is the Flow Processor?

The Flow Processor: Is a generic Java application that knows nothing about

specific Data Flows. Is aware that data is retrieved from a database and needs

to be formatted as XML. Is dynamic, allowing SQL Selects and output XML to be

changed without modifying the core Java code. Is extensible, allowing additional Java extensions to be

created and assigned to data flows. Can export or return data (submit to CDX or save to

permanent storage) in a variety of formats (XML, ZIP, or TXT).

Can be invoked with or without a Node

4

What is a Flow? The Flow Processor operates on an entity called a

Flow A Flow is identified by a unique name The only mandatory component of a Flow is the

model The model consists of a primary SQL query, and Zero or more detailed queries

The remaining components of a Flow are optional Zero or more PreProcessors (Java class) An XML Translation (XSLT) Zero or more ExportManagers (Java classes)

5

Simple Generic XML Submittal to CDX

Get Flow Config Data

XML(SQL)

JFP

ExportGenerate

XMLFrom SQL

Transform

XSLT

ExportMgr

Source

ExecuteSQL

TempXML

XSLT

FinalXML CDX

*Execution is from Left to Right JFP Step

6

Data Retrieval Processing: Contact Example

Primary Queryselect contact_id “id”

from ContactsWhere dept=‘Sales’

id100200300

Detailed Query 1select contact_id, address, city, state, zip from AddressWhere contact_id=#id#

Detailed Query 2select contact_id, home, cell from Phone_NumbersWhere contact_id=#id#

Detailed Query 3select contact_id,region from Sales_RegionsWhere contact_id=#id#

<address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip></address_info><address_info> … </address_info><address_info> … </address_info>

<phone_info> … </phone_info><phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell></phone_info><phone_info> … </phone_info>

<region_info> ... </region_info><region_info> ... </region_info><region_info> <contact_id>300</contact_id> <region>Southwest</region></region_info>

7

Transformation Step: Contact Example

<address_info> <contact_id>100</contact_id> <address>100 Maple Ave.</address> <city>Detroit</city> <state>MI</state> <zip>11111</zip></address_info><address_info> … </address_info><address_info> … </address_info>

<phone_info> … </phone_info><phone_info> <contact_id>200</contact_id> <home>717-555-1211</home> <cell>717-555-2212</cell></phone_info><phone_info> … </phone_info><region_info> ... </region_info><region_info> ... </region_info><region_info> <contact_id>300</contact_id> <region>Southwest</region></region_info>

<get_contact_data>

</get_contact_data>

<SalesForceDataService version="1"> <SalesContact id="100" region="Midwest"> <ContactInformation> <Address>100 Maple Ave.</Address> <City>Detroit</City> <State>MI</State> <ZipCode>11111</ZipCode> <PhoneNumbers> <Number type="Cell">717-555-3456</Number> <Number type="Home">717-555-1278</Number> </PhoneNumbers> </ContactInformationn> </SalesContact> <SalesContact id="200" region=“Midwest"> ... </SalesContact> <SalesContact id="300" region=“Midwest"> ... </SalesContact></SalesForceDataService>

XSLT

Temporary XML from Data Retrieval

Final XML OutputPost Transformation

8

Extending the Java Flow Processor The Flow Processor can be extended by developing custom

PreProcessors and ExportManagers classes. At runtime, the Flow Processor Engine uses Java Reflection to

instantiate a PreProcessor or ExportManager class. A Java class that acts as a PreProcessor must implement the

Java Interface FlowPreProcessManager. A Java class that acts as a ExportManager must implement the

Java Interface FlowExportManager. Several Java extensions are included in the JFP:

DefaultPreProcessManager – Issues zero or more INSERT or UPDATE statements defined in the Flow Processor Settings database tables.

CDXExportManager – Submits an XML payload to a URL. FileExportManager – Writes the resulting output to a flat file.

9

Flow Processor Complete Processing StepsFlo

w P

roce

ssor

Eng

ine

Time

Use the configuration file to locate the Data Access Definition files

Configuration

Data Access Definitions

Use the Data Access Definition files to locate and access the Flow Settings

Flow SettingsUse the Flow Settings to determine the XSL Stylesheets, FlowPreProcess Managers and FlowExportManagers

XSL Stylesheets

Flow PreProcess Manager(s)

Flow Export Manager(s)

Execute each FlowPreProcess Manager class in the order specified by the Flow Settings

Use the Data Access Definitions and Flow Settings to retrieve the data of the Flow

Source Database

Final Output File

For each data set retrieved, build the generic XML data format, and apply the XSL Stylesheets to convert the data into the final format

Execute each FlowExportManager class in the order specified by the Flow Settings

10

Deployable Components The Java Flow Processor engine, default Pre-

Processors, and Export Managers are packaged into a single Java Archive file (cgi-flow-process.jar)

It requires several third-party Open Source Java libraries, most notably: iBATIS – Provides the data access layer from the

source database and SQL mapping utilities Apache – Several Java libraries provide support for

XML parsing, XML Transformation, Connection Pooling, and Web Services (AXIS)

Log4J – Provides logging service

11

Invoking the Flow Processor Invoked from a DOS command line or Unix shell script

using the FlowProcessorCmdLine class. Accepts just one configuration file to configure the Flow

Processor Called within existing Java applications using the

FlowProcessor class directly. Two public methods are available: performSubmit – This requires only a Flow name and

returns a message of the resulting operation performQuery – This requires a Flow name and a Java

Map of potential query arguments. This returns the resulting byte stream from the JFP engine.

12

Node Functional Specification 1.1

<message name='Query'> <part name='securityToken' type='xsd:string'/> <part name='request' type='xsd:string'/> <part name='rowId' type='xsd:integer/> <part name='maxRows' type='xsd:integer/> <part name='parameters' type='typens:ArrayOfstring'/> </message> <message name='QueryResponse'> <part name='return' type=’xsd:string'/> </message>

Dynamic Node Query Services with JFP

FlowProcessor class

public byte[] performQuery( String flow_code, int start_row, int max_rows, Map arguments)

Node query()FlowProcessor performQuery()

JFP Engine

Byte StreamXML

13

Dynamic Node Query Processing Steps Convert Node Functional Specifications 1.1 WSDL into Java

code (e.g., Node11 class) public String query(String securityToken, String request, String

startRow, String maxRows, String[] parameters) Modify Node11’s query() method to retrieve an instance of the

FlowProcessor class Either use the incoming request parameter as the Flow code or

translate the incoming request into the appropriate Flow code Convert the incoming parameters array into a Java Map Call the FlowProcessor’s performQuery() method with correct

Flow code and Map of incoming parameters Convert the returning byte[] into a String and return this as the

result of the Query method

14

Using Apache Formatting Object Processing (FOP), stylesheets can transform XML into PDF

SQL Reports can be built and stylesheets can transform output into HTML reports.

Other Output Possibilities with JFP

Source

Gather Configuration Information

JFP Configuration

Data

HTML

Desired Output XSLT

SQL ConfigurationAs XML

1 2a

Temporary XML

2

1b

1a

Generate Temporary XML

Payload

3a

Transform XML into Final Payload

Desired Output XSLT

3b

2b

Final XML Payload

Text File

3

3c

3c

3c

4 Export Payload

CDX

File System

Other System

PDF

3c

4a

4a

4a4a

4b

4b

4b

Questions?