Logging Last Resource Optimization for Distributed Transactions in

Oracle WebLogic Server

T. Barnes, A. Messinger, P. Parkinson, A. Ganesh, G. Shegalov, S. Narayan, S. Kareenhalli

OLTP: Online Transaction Processing

Transaction is an ACID contract● Atomic – all or nothing

● Consistent – from the application perspective

● Isolated – masked concurrency through locking or snapshots

● Durable – once committed changes survive subsequent failures

Checking = 2000Savings = 8000

Checking = 1000Savings = 9000

time

beginc -= 1000s += 1000commit

OLTP: Single Resource● A and D are typically implemented using Write-Ahead Logging

● Transaction recovery is “simple”: REDO phase, UNDO phase.BEGIN TRANSACTION

/* LSN = 1: log for undo and redo in MM buffer*/

UPDATE Accounts SET balance = balance – 1000 WHERE Number = 1

/* LSN = 2: log for undo and redo in MM buffer*/

UPDATE Accounts SET balance = balance + 1000 WHERE Number = 2

/* LSN = 3: log commit and force (5-6 orders slower)*/

COMMIT TRANSACTION

Accounts LSN=1

1 1000

2 8000

Accounts LSN=2

1 1000

2 9000

Accounts LSN=0

1 2000

2 8000

OLTP: Distributed / Two-Phase Commit

Transaction Coordinator

Resource 1 Resource 1

prepare --> force-log prepare force-log prepare

<-- OK <-- OK

commit --> force-log commit force-log commit

<-- ACK <-- ACK

Like a wedding ceremony

Coordinator: Will you ...? (prepare)

Resource: I will (OK)

Coordinator: I pronounce you … (commit)

2PC is A CI D

● 2PC is not about Concurrency Control.

● 2PC transaction is therefore

○ Globally Atomic

○ Locally Isolated

○ Locally Consistent

○ Globally Durable

OLTP: Queued Transactionsclient` app server database

begin transaction

req_q.enqueue(req1)

commit transaction

begin transaction

creq = req_q.dequeue()

resp = creq.execute()

res_q.enqueue(resp)

commit transaction

begin transaction

resp = res_q.dequeue()

process(resp)

commit transaction

WebLogic Server: Java EE++

● App containers: Web (Servlets, WS), EJB, app clients

● Services: Messaging (JMS),

Transactions (JTA), Database (JDBC), …

Example: Queued Transactions (JEE)

@MessageDriven(

mappedName="jms/inboundQueue“

activationConfig = {@ActivationConfigProperty(

propertyName = “connectionFactoryJndiName",

propertyValue = “jms/inboundConnectionFactory"

)}))

@TransactionAttribute(TransactionAttributeType.REQUIRED)public class OrderMDB implements javax.jms.MessageListener {

@Resource javax.jms.Queue outboundQueue; @Resource javax.jms.ConnectionFactory outboundCf; @Resource javax.sql.DataSource orderDataSource; public void onMessage(Message message) { java.sql.Connection jdbc = orderDataSource.getConnection();

javax.jms.Connection jms = outboundCf.createConnection();

// update SQL via JDBC, notify via JMS connections …

}

}

“School” Presumed Abort 2PC TM Resources

Timeline

prepare

yes

force-log prepared

commit

force-log commit

all-prepared: force-log commit

all-commit: log end

ack

2n+1 writes, 4n messages

PA2PC (1): TM (Coordinator)

PA2PC (2): Resource (Participant)

“Real Life” XA 2PC

TM Resources

Timeline xa_prepare

ack_preparedforce-log prepared

xa_commitforce-log commit

all-prepared: force-log commit

all-commit: log end

ack_committed

xa_startack_started

ack_endedxa_end

2n+1 writes, 8n messages

Standard 2PC Optimizations

● 1PC: if only one resource enlisted, prepare skipped

● Read-Only: if voted read-only, commit skipped

● XA ceremony of xa_(start|end) is always present

Nested 2PC: Coordinator Role Transfer

TC Res2 Res3

prepare commit

commitcommit

● Last Resource is committed in one phase

● 2n messages/ 2n-1 forced writes

● Known topology: linked Databases

[Gray’78]

p

c

c

WebLogic Design Constraints and Goals

● No control over foreign XAResource, TM and topology

● Broadband: minimize blocking RPC, not messages

● Unneeded XA on Res3: save xa_start, xa_end

Typical WLS Deployment

● JMS and TM share the same FileStore

● Collocated JMS connection cost is negligible

● JDBC Datasource is remote: blocking RPC

● DB internal resources (locks, latches, etc.) are more

expensive and JEE is not a single client

● Outbound JMS notifies about a JDBC update

● Ideally: JDBC updates visible before JMS updates

JDBC as Logging Last Resource

● User enables a non-XA JDBC Datasource as LLR

○ LLR table WL_LLR_<server> in the DS schema

○ No XA overhead for the LLR

● TM log is local log UNION LLR table log

○ WLS does not boot if any LLR table is unavailable

● Restriction: 1 LLR datasource / Transaction

● No coordinator transfers as in Nested 2PC

XA 2PC Commit with LL Resource

1. Prepare concurrently all non-LLR XAResources

2. Insert XID into the LLR table

3. Commit the LLR-Resource

4. If 3 is successful, commit non-LLR XAResources

5. Lazy garbage-collection of 2PC records of completed

transactions is piggybacked on future LLR transactions

LLR Failure Recovery

● Failure before LLR.commit() => global abort

● Failure during LLR.commit() => similar to media failure

○ Wait until LLR Datasource / table is available for read

○ Presence of the LLR commit log decides the global outcome

○ If unavailable for AbandonTimeoutSeconds log abandoned

● JVM/OS crash: TM scan local log UNION LLR

○ Usual transaction outcome resolution

● 2PC recovery guarantees are not compromised

LLR Savings

Back-of-the-envelope for the single-threaded case with Jeff Dean’s numbers [Google key notes]: ● xa_start (RPC),

● xa_end (RPC),

● xa_prepare (RPC + force-log)

● Insert into LLR table + commit done via single RPC

------------------------------------------------4xRTT + 1xDiskSeek = 4x500,000ns + 10,000,000ns = 12 milliseconds

LLR in DS Wizard: Non-XA Driver

LLR in DS Wizard: Safe unlike Emulate

Research Workload EAStress2004 [SPECjAppServer’04]

EAStress2004 Benchmark HW/Setup

Driver 1 (3x Dealer, 3x Manufacturing)

Driver 2 (3x Dealer, 3x Manufacturing)

External Supplier Emulator

2x Quad Core 3.7Ghz x86_64, 2MB L2, 8GB RAM

2x Quad Core 2.7Ghz x86_64, 2MB L2, 16GB RAM

2x Quad Core 2.7Ghz x86_64, 2MB L2, 16GB RAM

System Under Test

Oracle WebLogic Server 11gR1 (Middle Tier)

Oracle RDBMS 11gR1 EE(Database Tier)

2x Quad Core 2.83Ghz x86_64, 6MB L2, 16 GB RAM

4x Quad Core 2.7Ghz x86_64, 2MB L2, 64GB RAM

Performance Evaluation (Utilization)

MDB scenarioMT = WLS on JrockItDB = Oracle Database

EAStess2004 v1.08, IR = 700 (not reviewed by SPEC)

Performance Evaluation (Response Time)EAStess2004 v1.08, IR = 700 (not reviewed by SPEC)

Purchase Manage Browse Manufacturing

XA 4.20 2.40 5.40 3.75

LLR 1.50 1.20 1.90 3.00

improvement 2.8x 2x 2.8x 1.25x

Future Improvements (probably in 12c)

● LLR does not detect Read-Only

● Transaction GUID instead of LLR table for Oracle

Thank You. Questions?

oracle.com/weblogicoracle.com/benchmarks

WebLogic FileStore

● XA-capable KV store on local file system

● Mime design: allocate under write-head

○ fast writes

○ slow recovery

○ works well up to a couple of GiB

● Transactional use: for JMS messages and JTA logs

● Non-transactional use: Diagnostics and Config