Enterprise Transactions in BankingA case study with JEE and Oracle

Raphael Süess

Credit Suisse AG

8588

Joe Pullen

Espalier Technologies

2

AGENDA

> Transact ion Processing in Credit Suisse

> Mot ivat ion for a JEE based Transact ion Plat form

> Case Study: Enterprise Transact ions in Payments

– Transact ions Overview

– Java Persistence (JPA)

– Five Enterprise Tact ics

3

Transaction Processing in Credit Suisse

Belts and braces are meant to hold one's trousers up. Going belt and braces is a double insurance against having them fall down.

Deliver timely and efficient services that guarantee to never compromise customer data.

4

Transaction Processing in Credit Suisse

Mainframe> Primary platform for enterprise critical

transactions– Handles millions of online transaction per day– Provides all important quality attributes (Trx

failover, guaranteed business continuity in disaster case)

– Customer data on DB2 or IMS– Sysplex, z/ OS, IMS, CICS, PL1, DB2

JAP (Java Application Platform)> Standardized, global platform for JEE

applications– Standard mechanism for security, audit ing,

deployment, etc. Platform globally rolled- out.– Mostly used to implement presentation layer

and parts of the business logic.

– Currently not enabled for enterprise transactions.

– Solaris, WebLogic, Java, Oracle

5

Motivation for a JEE based Transaction Platform> Provide an alternative platform for mission critical banking

applications.

– Bring online transaction applications (OLTP) onto JAP (Java Appl. Platform)

– Assure company wide TRX platform reuse (guaranteed failover and DR)

– Defined operational processes to properly operate and monitor the platform.

– Support Credit Suisse strategy for skill shift PL/ 1 Java

FeasibilityStudy

Pilot Application+ Infrastructure

PlatformIntegration General Availability

2007 2008 2009 2010

Roadmap for enterprise transaction processing on JAP

6

Transactions Overview> Terminology

– Transaction: A logical unit of work, bracketed by the transaction statements begin and commit/ abort.

– Distributed transaction: A transaction involving more than one resource manager (e.g., a database and queuing).

– ACID: Atomicity, Consistency, Isolat ion, Durability.

– Atomicity: Either all or no changes are committed persistent ly.

– 2PC: Two- phase commit protocol used to guarantee atomicity for distributed transactions.

> Two- phase commit

– First phase: Coordinator asks involved resource managers to prepare to commit.

– Second phase: If all resource managers voted to commit, then tell them to commit, else tell them to rollback.

7

Case Study: Starting Position

• Complex hybrid system architecture (mainframe and JEE)

• High communication overhead (costs)

• Distributed processing logic

• Business logic does not run in a transactional context.

8

Case Study: Pilot Application

• Single payments type to be fully processed on JEE.

• Distributed transaction processing with JMS (MQ) and JDBC (Oracle).

• Non transactional CORBA calls (read only).

• All cross platform updates calls via transactional persistent queues.

9

Case Study: Pilot JTP Setup

• Persistent shared MQ queues (clustered MQ Queue managers) on mainframe.

• WebLogic clusters with JTA migratable targetsusing Mult i- Data Source (MDS) configuration.

• Oracle RAC Database on shared SAN.

• Oracle DataGuard for disaster recovery.

10

Case Study: Enterprise Transactions with JEE

> Distributed Transactions

– Manager (JTA, JTS): WebLogic as transaction coordinator.

– Resource Managers: JDBC (Oracle, MDS), JMS (WebSphereMQ), JCA.

– Management: JMX, JTA interfaces, pending transactions

– Issues: Transaction scopes, t imeouts, error handling, latency

> Integration

– EJB3: Dependency and resource injection, annotations

> Persistence

– JDBC 2.0: Well established standard database access

– JPA 1.0: Domain object mapping, persistence units

> Messaging

– JMS 1.1: Message driven beans and XA resource injection

11

Case Study: Transactions Overview• Potentially long

running transactions

• Broken into eff icient smaller technical transactions

• Work manager policies (thread-constraints)

• Load balancing using JMS (persistent queues)

• Resources pools (Queue Managers, MDS, CORBA)

• Queue depth

12

Case Study: Transactions Overview

• Global transact ions managed by WebLogic• Local transact ions managed by Oracle and

MQ

• JMS using XA connection factories• JDBC using XA datasources

13

Case Study: Java Persistence (JPA) Usage> Provider

– Portability: Minimize use of provider specif ic features> Development processes

– Forward: Code generation from UML2 for JPA mapping– Reverse: Reverse engineering from exist ing DB2 database objects

> Performance and scalability issues– Testing: Signif icant "real world" load testing to identify

bott lenecks – Caching: Prepared statement cache, L2 caching– Performance: Consistently execute 20+ SQL operations (50% updates),

several COBRA calls and put 5- 10 MQ messages in < 1 sec

– Database: Large sequence caches, reverse indexes, part it ioning for purging, consistent model

– Concurrency: Optimist ic and pessimist ic locking, indexes onforeign keys

14

Five Enterprise Tactics

> Subsystems– Problem: Complex mesh of independent implementation classes– Solution: Encapsulate enterprise concerns

> Diagnostics– Problem: Complex environment for root cause analysis – Solution: Consistent, system wide data driven diagnostics

> Handling poison messages– Problem: Rolled back messages continually redelivered

– Solution: Use of message properties and delayed redelivery > Coarse grained locks

– Problem: Excessive contention with increased concurrency

– Solution: Use lock table to manage collection of related objects > EJB3 Timers

– Problem: Heavy weight scheduling uses lots of resources – Solution: Cascading EJB3 timer

15

Tactic 1: Subsystems

• A subsystem is an atomic coarse- grained facade over a mesh of f ine-grained implementation objects (JPA, JMS, CORBA)

• Encapsulates:• Transactions• Data access• Message dispatch• Exception handling• Management• Database model

• Separate deployment• JEE standards

• EJB,JMS,JPA,JMX

16

Tactic 2: Diagnostics

• EJB interceptors implementation• Dynamic control of components via JMX (script or console)• MQ queue and message analysis (queue depth, latency, connections)

• Data driven logging for operational awareness and trending

• Productive problem analysis

• Fast root cause analysis

• Profiling and simulation for production testing

• Separate database objects

17

Tactic 3: Poison Messages

• If (re)- delivery count exceeds maximum allowed then handle differently and commit transaction to remove it from queue

• JMS_IBM_PutDate and JMS_IBM_PutTime header propert ies can be used to calculate the latency of the messages in the queue

18

Tactic 4: Coarse Grained Locking

• Use single pessimist ic row lock to lock a larger set of rows (objects in JPA)

• Simplif ies locking action• Not all members of the group

have to be loaded by JPA in order to lock them

• Used to select collections of payments to be scheduled by schedule t ime and payment type

• Take care of foreign key indexes and unnecessary updates on the lock table

19

Tactic 5: EJB3 Timers

• Timers created with TimerService from injected SessionContext resource• @Timeout annotation method executed at fixed intervals until t imer is

canceled• Timer specific serializable object can be associated with t imer

• Use for f ine grained tasks

• Continue in parallel until work is completed

• Execute in container context

• Callback object• Init ially triggered

from external coarse grained scheduler

20

Appendix:

> Code Samples

– Transactional MDB

– JMS Sender

– JMX Subsystem Management

– JMX MDB Management

– Logging using Synchronization

21

Code Sample: Transactional MDB@MessageDriven(mappedName = "jms.pay.sic.inbound",

name = "ExternalMessagingBean", activationConfig = { @ActivationConfigProperty(propertyName = "destinationType",

propertyValue = "javax.jms.Queue") })@DeclareRoles( { "MessagingConsumer" })@RunAs("MessagingConsumer")@Interceptors( { PayContextInterceptor.class, SimulationInterceptor.class, LoggingInterceptor.class, TransactionLoggingInterceptor.class })public class ExternalMessagingBean implements MessageListener, PayLoggable{

@Resource(name = "maximumRetries")protected Short maximumRetries = 10;

@EJBprotected DiagnosticsManager diaManager;

@Resourceprotected MessageDrivenContext context;

@TransactionAttribute(value = javax.ejb.TransactionAttributeType.REQUIRED)public void onMessage(Message msg){

• Transact ion required

22

Code Sample: Transactional MDB

<weblogic-enterprise-bean><ejb-name>ExternalMessagingBean</ejb-name><message-driven-descriptor>

<pool><initial-beans-in-free-pool>20</initial-beans-in-free-pool><max-beans-in-free-pool>20</max-beans-in-free-pool>

</pool><destination-jndi-name>jms/pay/sic/inbound</destination-jndi-name><connection-factory-jndi-name>ibm.mq.cf.SINGLE_XA_CF1</connection-factory-jndi-name>

</message-driven-descriptor><transaction-descriptor>

<trans-timeout-seconds>60</trans-timeout-seconds></transaction-descriptor><dispatch-policy>captureWorkManager</dispatch-policy>

</weblogic-enterprise-bean>

• Init ial and max bean pools

• XA connection factory• Transaction t imeout• Custom dispatch policy

<work-manager><name>captureWorkManager</name><min-threads-constraint>

<name>MinThreadsConstraint</name><count>20</count>

</min-threads-constraint><max-threads-constraint>

<name>MaxThreadsConstraint</name><count>20</count>

</max-threads-constraint></work-manager>

• Work manager defined to constrain number of threads used for MDBs

• Can be deployed WebLogic server wide or as part of the application deployment

23

Code Sample: JMS Sender• XA connection factory and queue resources injection for sending

• JMS Session pooling• Cleanup JMS resources

@Resource(name = "jms/mdbQCF", mappedName = "ibm.mq.cf.SINGLE_XA_CF1", type = ConnectionFactory.class)

private ConnectionFactory mdbQCF;

@Resource(name = "jms/mdbQueue", mappedName = "jms/pay/sic/outbound", type = Queue.class)

private Queue mdbQueue;

connection = mdbQCF.createConnection();session = connection.createSession(true, Session.AUTO_ACKNOWLEDGE);producer = session.createProducer(mdbQueue);

TextMessage message = session.createTextMessage();message.setText(msgData);

producer.send(message);

• Possible to manage rollback using @ApplicationException(rollback= true) or setRollbackOnly() on injected MessageDrivenContext

• Foreign JMS provider

24

Code Sample: JMX Subsystem Management@ManagedAttribute(name="ProcessingMessageCount",

description="Processing Message Count")public Long getProcessingMessageCount(){

return MDBHelper.getHelper(server).getProcessedMessageCount(MDB_NAME);}

@ManagedAttribute(name="ProcessingMessagesCommitted", description="Processing Messages Committed")

public Long getProcessingMessagesCommitted(){

return MDBHelper.getHelper(server).getTransactionsCommitted(MDB_NAME);}

@StartMethodpublic void start(){

log.info("Starting SubSystem " + name + ": " + description);MDBHelper.getHelper(server).resume(MDB_NAME);

}

@StopMethodpublic void stop(){

log.info("Stopping SubSystem " + name + ": " + description);MDBHelper.getHelper(server).suspend(MDB_NAME);

}

• JMX subsystems for monitoring and management of processing components

• JMX annotations for creation of dynamic managed beans

• Defined lifecycle method annotations

25

Code Sample: JMX MDB Management

AttributeValueExp at1 = Query.attr("Type");ValueExp v1 = Query.value("MessageDrivenEJBRuntime");QueryExp exp1 = Query.eq(at1, v1);

Set<ObjectInstance> instances = server.queryMBeans(new ObjectName("com.bea:*"), exp1);

for ( ObjectInstance instance: instances ){

String EJBName = (String)server.getAttribute(instance.getObjectName(), "EJBName");

if (EJBName.equals(name)){

// Example Suspend MDB dispatchingserver.invoke(instance.getObjectName(), "suspend",

new Object[] {}, new String[] {});

// Example getting messages processed by MDBserver.getAttribute(instance.getObjectName(), "ProcessedMessageCount");

}}

• MDB managed using standard WebLogic EJB runt ime JMX beans

26

Code Sample: Logging using Synchronization

• Create listener class in @AroundInvoke interceptor method and register listener with registerSynchronization of transaction

• Allows logging of container related exceptions such as JTA t imeouts and database constraints

protected static final class SynchronizationListener implements Synchronization{

@Overridepublic void afterCompletion(int status){

// Committed, Marked Rollback, RolledbackremoveListener(tx);

}

@Overridepublic void beforeCompletion(){

// Logging and other }

}

status prior to the start of the two-phase transaction commit process

status after the transaction is committed or rolled back

27

Code Sample: Logging using Synchronization

<May 25, 2009 4:42:06 PM CEST> <Notice> <Stdout> <BEA-000000> <2009-05-25 16:42:06.402 CEST ; FINEST ; com.csg.cs.pay.process.diag.tx.TransactionLoggingInterceptor$SynchronizationListener ; beforeCompletion ; ; weblogic ; ID:c3e2d840d8e3f0c14040404040404040c43cde09b707300a ; none ; ; JTX1_pay_next_processing ; JTX ; [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' ; [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' ; ; beforeCompletion: 0334237775dbae1be4da started since 51002 ms and TTL 9000 ms with local props = {weblogic.jdbc.jta.pay_next_jdbc=[autoCommit=true,enabled=true,isXA=true,isJTS=false,vendorID=0,connUsed=true,doInit=false,'null',destroyed=false,poolname=pool1,appname=null,moduleName=null,connectTime=234,dirtyIsolationLevel=false,initialIsolationLevel =2,infected=false,lastSuccessfulConnectionUse =1243262526386,secondsToTrustAnIdlePoolConnection=10,currentUser=null,currentThread=null,lastUser=null,currentError=null,currentErrorTimestamp=null,JDBC4Runtime=true,supportStatementPoolable=true,needRestoreClientInfo=false,defaultClientInfo={}] } ;><May 25, 2009 4:42:06 PM CEST> <Notice> <Stdout> <BEA-000000> <2009-05-25 16:42:06.464 CEST ; FINEST ; com.csg.cs.pay.process.diag.tx.TransactionLoggingInterceptor$SynchronizationListener ; afterCompletion ; ; weblogic ; ID:c3e2d840d8e3f0c14040404040404040c43cde09b707300a ; none ; ; JTX1_pay_next_processing ; JTX ; [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' ; [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)' ; ; afterCompletion: 0334237775dbae1be4da has status Committed started since 51080 ms and TTL 120000 ms with local props = {weblogic.jdbc.jta.pay_next_jdbc=[ No XAConnection is attached to this TxInfo ] } ;>

• Use to log t ime for f inal completion of transaction (2PC protocol)• Extended access to WebLogic specif ic transaction attributes and

instrumented transaction monitor attributes

Raphael Süess www.credit- suisse.comCredit Suisse AG raphael.sueess@credit-suisse.com

Joe Pullen www.espalier.comEspalier Technologies joe.pullen@espalier.com