© 2012 IBM Corporation Introduction to WebSphere MQ File Transfer Edition.

© 2012 IBM Corporation

Introduction to WebSphere MQ File Transfer Edition

© 2011 IBM Corporation2

WebSphere MQ: Connectivity to, from and within an Enterprise

A Universal Message Bus for access to data wherever it exists to support your business. MQ moves:

– Data– Messages– Files– Events– Services

Provides a comprehensive range of Messaging capabilities to support your Business requirements for data integration

– Reliability and availability QoS– Messaging integration patterns– Managed File Transfer– SOA foundation

Provides appropriate data accessand data privacy controls to help meet audit and regulatory requirements

MQ 7.5 is GAed on 6/15/2012, single, integrated offering for all messaging functions

2

Petrol Forecourt

BranchOutlet

Regional Office

Retail Store

Refinery

MobilePhone

Sensore.g. RFID

Enterprise

PervasiveDevice

CSS: F S


What problems does Managed File Transfer Solution solve

Reliability– Do you experience lost file, partially written file and duplicate due to inconsistent network or system

being not available? – What is the business impact when files aren’t properly transferred?– How much time and energy do you spend trying to figure out what happened when a file went

astray?– How much time and effort does it take to recover/restore things after a failure?

Performance– Do you need to transfer large file or many files at the same time? Are you able to meet the SLA? – Every time when a file transfer fails, does it require to be restarted from the beginning?

Security– Do you have any security concerns, such as plain file in transit which contains sensitive data? – How about access control, such as who can transfer what file(s) from source to destination?

Visibility and Traceability– Do you have any visibility of the file transfer status and progress, whether the file is partially sent,

completes successfully or not, in a timely fashion? – Do you want to track the entire journey of a file across the enterprise? – Are you able to show in an audit where your last 10 transferred files and documents came from and

went?

Automated and Integrated– Do you need various ways to invoke file transfer such as ad-hoc, scheduled, event triggered, or

through programming API?– Do you need to integrate file transfer with the rest of your IT infrastructure and business processes?


Partially written files (often only detected after substantial process and time is wasted)

Cost of re-transmitting big files (no check-point restart)

Lack of Character Set conversion (file data can be unusable after transfer

FTP – the lowest common denominator for file transfer

Limited Flexibility

Limited Security

Little visibility or traceability

Limited Reliability

Username and password are often in plain text

Lack of Privacy, authentication and encryption often not available

No notifications when a transfer fails

No central management and progress monitoring

Cannot track the entire journey of files

All resources have to be available at the same time

FTP is 30 years old, and showing its age. It is no meets business needs

ftp Scripts are typically scattered across machines, and require platform specific skills

Often only one ftp transfer can run at a time


WebSphere MQ File Transfer EditionAdds managed file transfer capabilities to WebSphere MQ

Simple Centralized file transfer configuration, management and monitoring via graphic tool

Automated Enables scheduled and event-driven file transfers

Reliable Solid reliability of WebSphere MQ + automated retry, error recovery and checkpoint restart

Any file size Multi threading, efficiently handles anything from bytes to terabytes, and data bombardment

Secure Provides role based access control. Protects file data using SSL + MQ AMS.

Auditable Full logging and auditing of file transfers + archive

Integrated Programming API, Conversion between file and messages

Cost Effective Exploits investment in WebSphere MQ. Wide range of support ( inc. z/OS and IBM i)

A B C X Y Z

……

WebSphere MQ File Transfer Edition


WebSphere MQ - A consolidated transport for both files and messages

Traditional approaches to file transfer result in parallel infrastructures

– One for files – typically built on FTP– One for application messaging – based on

WebSphere MQ, or similar

High degree of duplication in creating and maintaining the two infrastructures

MQ File Transfer Edition reuses the MQ network for managed file transfer and yields:

– Operational savings and simplification– Reduced administration effort– Reduced skills requirements and maintenance

File Transfers Application Messaging

Consolidated Transportfor Messages & Files

Maximizes the ROI


How does WebSphere MQ FTE work?

MQ FTE AgentMQ network

Agent QM MQ FTE Agent

MQ network

1. Send file data 2. Receive file data

MQ Server


How does WebSphere MQ FTE work?


Components of a typical WMQ File Transfer Edition Network

1. WebSphere MQ Provides the transport for all communication between agents, commands and the

coordination hub

2. Agents The endpoints for file transfers Long running, multi-threaded MQ applications, Associated with one particular queue manager (V6 or V7) Efficient transfer protocol for files

3. Commands Send instructions to agents and display information about agent configuration Many implementations of commands: MQ Explorer, Command Line, Scripting,

API

4. Coordination Hub Stores configuration information about the WMQ FTE network Collects events for the file transfers Keeps a historical account of transfers that have taken place


Ways to configure and control managed file transfers

Graphical Eclipse-based MQ Explorer

Command Line Interface

Job Control Language (JCL)

XML Scripts using Apache Ant


Centralized configuration using MQ Explorer

Eclipse GUI integrated into WebSphere MQ Explorer

• Shows progress of current transfers and outcome of recent transfers

• 3rd party and bespoke applications can also subscribe to these events


Managed file transfers using MQ Explorer

Graphical user interface that allows:– Operators to create ad-hoc file transfers to

satisfy a particular business need

File transfers can:– Specify code page conversion– Group together multiple files into a single

transfer– Recursively move directory structures


Scheduled transfers

Transfers can be scheduled to take place at a specific time

Scheduled transfers can be used to initiate a managed file transfer during a processing window

Scheduled transfers can repeat:– Every day, week, year etc.– For a specified number of repetitions– Until an end date/time


Directory Monitoring

Monitor file system for presence of “trigger files” andstart a file transfer operation

Directory monitoring can be used to integrate with an application which produce files as its output – without needing changes to the existing application

“Trigger files” can be:– Specified using wildcards– Found by recursively searching directory trees

Attributes of file transfer can be inferred from trigger file, for example:– Transfer all files from same directory as trigger file– Select destination agent name from parent directory of trigger file– Name destination file with timestamp of trigger file– Etc.

Supports extension via user exit routines


Via the Command Line...

Equivalent function to the GUI is also scriptable from the command line

Administrative commands:– Define and delete agents– Configure agents– List and show details about agents– Start and stop agents– Create and delete scheduled transfers– Create and delete directory monitors

Operational commands:– Start transfers– Cancel transfers– List in-progress transfers


… and also Job Control Language (JCL)


Scripting

Script together complex sequences of transfer operations which execute conditionally

– E.g. If step1 successful do step2 else email administrator

Use scripting to call out to other programs to perform pre/post processing of file data– E.g. Run a program prior to the transfer to generate the file, or after the transfer

to process the file– The output from any programs run is

captured in the transfer audit information

Based on the Apache Ant language which is:

– Open… (so you can get a book on it…)– ..and extensible (so you can use Ant

tasks developed elsewhere)


Example of a file transfer script

1. Application writesfile to file system

ExistingApplication

WMQFTE

Agent

WMQFTE

Agent

ExistingApplication

*tap*

2. Agent monitors filesystem, spots arrival of file and based on rules, transfers the file

3. FTE transports fileto destination

4. At destination MQ FTE writes file to file system

5. FTE can also start another application to process the file


Other Examples

Transfer one or multiple files from source to destination

Transfer file(s) from source to multiple destinations

Unzip, Combine and Zip files to send from source to destination

Concatenate files from multiple sources into a single file

Split a single file into multiple files and send to different destinations

Validate the file data before file transfer

Send email notification if file transfer fails


20

Integration with event driven applications

MQ FTE Helps:– Deliver files as message payloads and vice versa– Monitor queues and transfer message payloads to files– Bridge between batch based applications and event driven applications

WebSphere MQ&

File Transfer Edition


MQ queue monitoring

The WMQ FTE agent can monitor queues for the arrival of messages, then perform an action, such as transferring the payload from the message to a file

Conditions that can be monitored for:– Queue not empty– Complete group of messages

ExistingApplication

WMQFTE

Agent

ExistingApplication

WMQFTE

Agent

FTE agent can monitor for files arriving…

It can also monitor for messages arriving on a queue…


Example Usage of converting data between files and messages

One file to one message

WMQFTE

One file to a group of messages

One message to one file

A group of messages (or all messages on the queue) to one file

WMQFTE

WMQFTE

WMQFTE

The file can be split based on:

– Size– Binary delimiter– Regular expression

One message becomes one file

Optionally, a delimiter can be inserted between each message used to compose the file

One file becomes one message


API for Invoking File Transfer

MQ FTE Agent

DatabaseLogger

Central Management Tool

MQ network

MQ Server MQ FTE Agent

MQ network

File Transfers can be invokedby ESB as a service

ESB

3. Send file data

2. Start file transfer

4. Receive file data

1. Send request


File Broker – integration with ESB

MQ FTE Agent

DatabaseLogger


MQ network

MQ Server

MQ FTE Agent


File Transfers can be initiated via- Scheduler- File Directory Monitor- Ad Hoc- Script

ESB

1. Send file data

2. Process file dataif required such asdata transformation


File to Message(s) orMessage(s) to File conversion


Web Gateway - Integration with Web

Pain-points:– Difficult to mix human imitated file transfers with existing infrastructure for machine-to-

machine managed file transfer– Managed file transfers to zero-install, small-footprint devices

File Transfer Edition Helps:– A RESTful API for exchange files with an WMQ FTE network– Example web 2.0 applications to use as a starting point

Agent

WebSphere MQ

Agent Agent

Agent Agent

HTTPS

HTTP

HTTPS

Web Gateway


Protocol Bridging Agents Support for transferring files located on FTP and SFTP servers

– The source or destination for a transfer can be an FTP or an SFTP server

Enables incremental modernization of FTP-based home-grown solutions– Provides auditability of transfers across FTP/SFTP to central audit log– Ensures reliability of transfers across FTP/SFTP with checkpoint restart

Fully integrated into graphical, command line and XML scripting interfaces– Just looks like another FTE agent…

Audit information

Agent

WebSphere MQ

Agent Agent

ProtocolBridgeAgent

FTP/SFTP

FTP/SFTP

Server

FTP/SFTPClient

FTP/SFTPClient

FTP/SFTPClient

Files exchanged between FTE and FTP/SFTP


Security - File Transfer Access Control

WebSphere MQ already provides access control that can be used to prevent unauthorized users from accessing MQ objects (such as queues)

File Transfer Edition extends this to include authorities that relate to file transfer operations (e.g. should this user be able to transfer files from the source system, and to the destination system)


Securing file data transfer with SSL and WMQ AMS

WMQ FTE supports transport level encryption using SSL

Data is encrypted before it is sent over a channel and decrypted when it is received

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

svrconn channel

sndr/rcvrchannels

WebSphereMQ

QueueManager

WebSphereMQ

QueueManager

svrconn channel

sndr/rcvrchannels

When combined with WMQ Advanced Message Security

– Allows file data to be encrypted at the source system and only decrypted when it reaches the destination system

– Data is secure even when at rest on a queue

Agent Agent

AgentAgent


File Transfer Solution Design Considerations

Who and How to invoke file transfer - scheduled, event driven, ad-hoc, or via API?

How to transfer the file reliably and securely - HTTP, FTP, MQ or in combination?

How to access the file data at source and destination – File directory, MQ, HTTP, etc.?

Is any file data processing required such as transformation?

Is there any pre and post file transfer processing steps required at source and destination, such as notification?

How to handle file transfer failure?

Is there any file transfer tasks which need to be executed in sequence?

Is there any integration points with other applications/systems which are not based on files?

What’s monitoring and auditing requirement if any?

Is security a concern?

Other considerations

How big is the file, and how often are the file transfers required?

How many systems (source and destination) are involved and what system? Same location or not?

What’s the topology for file transfers, hub/spoke or point to point?


Resources Information Center:

– http://publib.boulder.ibm.com/infocenter/wmqfte/v7r0/index.jsp

Redbooks / Redguides / Redpapers: – Getting Started with WebSphere MQ File Transfer Edition V7

• http://www.redbooks.ibm.com/abstracts/sg247760.html– IBM WebSphere MQ File Transfer Edition Solution Overview

• http://www.redbooks.ibm.com/abstracts/redp4532.html– Managed File Transfer for SOA using IBM WebSphere MQ File Transfer Edition

• http://www.redbooks.ibm.com/abstracts/redp4533.html– B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60

and WebSphere MQ File Transfer Edition• http://www.redbooks.ibm.com/abstracts/redp4603.html

– IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a Multi-Enterprise Solution

• http://www.redbooks.ibm.com/abstracts/sg247927.htm– Multi-Enterprise File Transfer with WebSphere Connectivity

• http://www.redbooks.ibm.com/abstracts/sg247886.htm

Trial Download:– http://www.ibm.com/software/integration/wmq/filetransfer/

Early Design Program– Interested in participating in the development of future versions of FTE?

• Ask your local IBM representative to nominate you for the FTE EDP program


Any Questions?

© 2012 IBM Corporation

Back-up material


MQ Advanced Message Security Is a MQ plug-in, which pre-req MQ

Adds additional security services to WebSphere MQ

Enables secure message transfers at application level

Replaces costly, home-grown solutions that lack management controls

Extra Security capabilitiesPolicy-based authentication for each applicationProtects message contents end-to-end including when residing in queues

Centralized control of fine-grained MQ security policiesAudit logs of data and queue accessNo changes needed to applications or queues


WMQ vs WMQ AMS

WebSphere MQ– Authentication (local OS for server apps or peer authenticated SSL for client apps)– Authorisation (OAM on distributed, RACF on z/OS)– Integrity (SSL for channels)– Privacy (SSL for channels)

WebSphere MQ AMS– As above, additionally:– Integrity (Digital signature of message content)– Privacy (Message content encryption)

34


Securing file transfer with MQ AMS

MQ FTE Agent

DatabaseLogger


MQ network

MQ ServerMQ FTE Agent

MQ network

File data is encrypted and signed before transfer

1. Send file data encrypted and signed 2. Receive file data encrypted and signed

MQ AMS Client MQ AMS Server MQ AMS Client

File data is decrypted and signature verified after transfer


36

WebSphere Message Broker Nodes for MQ FTE

FTEInput node – Build flows that accepts file transfers from the WMQ FTE network

FTEOutput node– Build flows that are designed to send a file across a WMQ FTE network

When WMQ FTE nodes are used in a flow an FTE agent is automatically started in the Message

Broker Execution Group

Message Flow

Execution Group

Message Broker

FTEInput FTEOutput

WMQFTE

AgentWMQFTE

Agent

WMQFTE

Agent

WMQFTE

Agent

Part ofWMB 7.0.0.1

Part ofWMB 7.0.0.1


Integration with Message Broker (1)

MQ FTE Agent

DatabaseLogger


MQ network

MQ Server

MQ FTE Agent


Web Services, MQ, DB, etc.

File Transfers can be initiated via- Scheduler- File Directory Monitor- Ad Hoc- Script/API

data transformation, Enrichment, convert file data to message(s), etc.

Integrate withnon file oriented systems

Message Flow

WMB

1. Send file data

2. Process file dataif required




Integration with Message Broker (2)

MQ FTE Agent

DatabaseLogger


MQ network

MQ Server

MQ FTE Agent

MQ FTE Agent

MQ network

File Transfers can be invokedBy Message Broker as a service

Message Broker

WMB

2. Send file data

1. Start file transfer



Trading Partner

Integration with IBM Sterling Connect:Direct

FTEAudit

Agent

WebSphere MQ

Agent Agent

C:DBridgeAgent

C:DNode

C:DNode

C:DNode

C:DNode

C:DAudit

Reference

Inside the FTE audit trail…

The audit information for each FTE transferreferences related C:D audit information

The Connect:Direct Bridge capability supports managed file transfers that span FTE and C:D with a joined up audit trail

Date post:	24-Dec-2015
Category:	Documents
Upload:	shona-hodges
View:	226 times
Download:	0 times

© 2012 IBM Corporation Introduction to WebSphere MQ File Transfer Edition.

Documents