High-speed ObjectWeb Loggerfor J2EE™ Application Servers

michael.giroux@objectweb.org

Agenda

History & Requirements Implementation Status

History

Suggested during ApacheCon 2003 Project initiated one week after ApacheCon 2003 Mailing List created December 2003

– howl@objectweb.org

Journal Prototype Jan 26, 2004– Hiram Chirino– employs Doug Lea’s Concurrent Programming in Java™

Mike Spille pyrasun.xa_log Mar 09, 2004– www.theserverside.com search for “XA Exposed”

CVS commit core functionality Mar 30, 2004

Requirements for HOWL Journal

Support JOTM Transaction Logging– ObjectWeb’s Java Open Transaction Manager– jotm.objectweb.org

API tuned for transaction log– Binary data – byte[]– Application specified sync– Replay logged records for recovery operations

Performance and operational objectives– 10,000 XA TX/Sec– No more than 25% CPU– Minimal impact on server restart time – Detect incomplete journal data

BSD License

2PC Transaction Lifecycle

From http://jroller.com/page/pyrasun/Weblog?catname=/XAWith Permission from Mike Spille

Agenda

History & Requirements Implementation Status

Implementation

Log is private to application– Sharing the log is controlled by the application

Multiple physical files (2 or more)– Easier than circular reuse– Restart effected by number of files

Multiple records buffered into logical blocks– java.nio.ByteBuffer– Application data records– HOWL control records– Threads may block until IO is forced to disk

Block Format

Enough information to verify complete write– Header

Signature “HOWL” Block Sequence Number (used in record keys) Block size (bytes) Bytes used Hash Code (optional) Time Of Day

– Footer Signature “LWOH” Block Sequence Number Time of Day (same as header)

Batch Multiple Writes Into Single Force

IDE Disk Drive on Windows XP system capable of approximately 30 writes per second.

– 10,000 tx / 30 writes = 334 TX/Write– Write cache turned off

Approximately 200 bytes per XA COMMIT– XID plus additional TM information

Approximately 67,000 bytes per force IO buffers 4Kb (configurable)

– Reasonable for non-peak periods of operation

Approximately 16 buffers per force– Buffers written as they fill

File Switch optimized

File not reused until pending transactions complete Before any log can be reused, active COMMIT

records must be moved to current log file Could do this immediately at log switch

– Records copied unnecessarily

HOWL defers the move to allow resource managers to finish committing

– Normally it is not necessary to move any records.

But how does HOWL know?

The MARK

How to know when it is OK to reuse log space? HOWL Logger intended for logging transient data

– XA COMMIT becomes obsolete at end of 2PC– Log files can be reused when 2PC completes

HOWL maintains an active MARK– position of the oldest active record

MARK moved by application or automatically– Logger.mark(long key)– LogEventListener notified of log overflow– AutoMark mode

Log MARK example

LogEventListener.logOverflowNotification

1. Logger detects that log #2 is 50% full and calls the registered LogEventListener

2. LogEventListener copies active records from log #1 to log #2 and updates MARK

3. Log #1 is now available for use when log #2 fills.

write position

MARK1 2

MARK

write position

1 2

Setting LogEventListener

import org.objectweb.howl.log.LogEventListener;

import org.objectweb.howl.log.Logger;

public class ApacheConTest extends TestCase

implements LogEventListener {

Logger log = new Logger();

void logOverflowNotification(long overflowFence) {

// . . .

}

public void testExample1() throws Exception {

log.open();

log.setLogEventListener(this);

}

}

LogEventListener

Logger notifies application of log overflow– Current log file 50% full– Next file contains active MARK

void logOverflowNotification( long overflowFence ) {long newLowKey = Long.MAX_VALUE;for (int i=0; i<activeTx.length; ++i) {

if (activeTx[i].key < overflowFence) // move records that are below the

fence log.put(activeTx[i].data, false);

else if (activeTx[i].key < newLowKey) // remember lowest key newLowKey = activeTx[i].key;

}// set new mark with forcelog.mark(newLowKey, true);

}

Synchronization

org.objectweb.howl.log.LogBufferManager Circular queues eliminate object creation Buffer Pool

– LogBuffer[] bufferPool– bufferManagerLock protects buffer pool

Write queue– LogBuffer[] writeQueue– bufferManagerLock protects “put” pointer– forceManagerLock protects “get” pointer– Guarantees order of writes

No IO issued while holding bufferManagerLock Writing does not block puts.

Separate locks for ‘put’ and ‘write’

bufferManagerLock Array of buffers to be filled. Shut while data is copied to

current buffer. Current buffer moved to

write queue when full Write queue “put” pointer

incremented

forceManagerLock Circular queue of buffers to

be written. Enforces order of buffer

writes. “get” pointer incremented as

buffers removed from queue Buffer returned to pool after it is written to disk

Implementation Recap

Multiple Log Files Records collected into blocks Blocks protect data integrity Log MARK protects active data LogEventListener optimizes file switch Overlap “put” and “write” via multiple locks

Using the HOWL Logger

Construct new instance– import org.objectweb.howl.log.Logger;– Logger log = new Logger();

Open the Logger– log.open();– May throw exception

Put records into the log and manage log MARK– key = log.put(byte[] data, boolean force)– key = log.put(byte[][] data, boolean force)– log.mark(key)

Close the Logger– log.close();

Simple Example

// Simple program to open a default log

// and write one record

import org.objectweb.howl.log.Logger;

public class ApacheConTest extends TestCase {

public void testExample1() throws Exception {

byte[] data = "Record Data".getBytes();

long key = 0L; // record key within log

Logger log = new Logger();

log.open();

key = log.put(data,true); // put with force

log.close();

}

}

Logger

Public methodsvoid open();void close();

void setLogEventListener(LogEventListener listener);

long put(byte[] data, boolean force);long put(byte[][] data, boolean force);

void mark(long key);void mark(long key, boolean force);void setAutoMark(boolean mode);

void replay(ReplayListener listener);void replay(ReplayListener listener, long mark);

String getStats();

Configuration object– JavaBean™– Default values for all propertiesConfiguration();

Configuration(Properties prop);

Configuration(File propFile);

Construct Configuration objectimport org.objectweb.howl.log.Configuration;

configuration cfg = new Configuration();

cfg.setLogFileDir(“/usr/geronimo/logs”);

Construct new Logger instanceimport org.objectweb.howl.log.Logger;

Logger log = new Logger(cfg);

log.open();

Customized Configurations

Recovery using replay

Logger.replay(ReplayListener rl, long mark); ReplayListener Interface

– LogRecord getLogRecord()– onError(LogException)– onRecord(LogRecord)

LogRecord LogRecordType

– USER– END_OF_LOG– XACOMMIT– XADONE– others

LogRecord

Constructor specifies initial size of data bufferLogRecord lr = new LogRecord(int size);

Data buffer is reallocated for larger records Public members

long key; // record key as returned by put()

long tod; // time of day the record was written

short type; // record type (see LogRecordType)

Public methodsbyte[][] getFields(); // returns record data

Replay Example

import org.objectweb.howl.log.ReplayListener;import org.objectweb.howl.log.Logger; public class ApacheConTest extends TestCase implements ReplayListener { Logger log = new Logger(); public LogRecord getLogRecord() { return new LogRecord(128); } public void onRecord(LogRecord lr ) { /* do something with each record */ byte[][] fields = lr.getFields(); } public void onError(LogException e) { . . . }

public void testExample1() throws Exception { log.open(); log.replay(this); . . . }}

XA Specific Classes for JOTM

XALogger– Extends Logger with methods specific to XA

XACommittingTx putCommit(byte[][] data) putDone(byte[][] data, XACommittingTx tx) XACommittingTx[] activeTx

XACommittingTx– Copy of log key returned by put() methods– Reference to the COMMIT data record– Allows XALogger to move COMMIT records

XALogRecord– Extends LogRecord with method to get XACommittingTx

associated with record.

XALogger

XALogger class extends the basic Logger with features specific to XA.

– putCommit Return XACommittingTx object

– putDone Accepts XACommittingTx object

– replayActiveTx()

Implements LogEventListener– Moves COMMIT records in response to

LogEventListener.logOverflowNotification() Log Key in XACommittingTX object updated after move

– MARK updated when all records moved

XALogger Active Transaction List

Entries added to list by putCommit Entries removed randomly by putDone logOverflowNotification must scan entire list Predictable number of entries Preallocate XACommittingTx objects

– Objects assigned an index in activeTx array

XACommittingTx.index locates entry in list– putDone locates entry directly by index

No object creation == no garbage collection

XALogger Example

import org.objectweb.howl.log.xa.XALogger;public class ApacheConTest extends TestCase

implements ReplayListener{ XALogger log = new XALogger(); public void onRecord(LogRecord lr) { XALogRecord xlr = (XALogRecord) lr; // rebuild TM tables using xlr // and save XACommittingTx for subsequent putDone() pending_tx[x].xacommittingtx = xlr.getTx(); } public void testExample1() throws Exception { log.open(null); log.replayActiveTx(this); }}

Metrics Reporting

JMX MBeans publish metrics while running– JDK 1.5 jconsole– MC4J version 1.2 (Beta 7)

http://mc4j.sourceforge.net/

Logger.getStats() returns session metrics– XML document– Includes statistics for all Logger components– Should be called after Logger.close() for complete detals

How well did we do?

Did we achieve 10,000 TX/SEC?– YES – given proper hardware– Your mileage will vary!

Does code use less than 25% CPU?– YES – based on informal measurements– Varies based on JVM

Pure Java– YES

Do we support JOTM recovery?– YES – according to this impartial observer

Throughput Measurements

ATA/133MB/sec, 2MB,

7,200 RPM

SATA/ 150Mb/sec, 8MB,

10,000 RPM

Threads Buffer Size TX/Sec Latency (ms) TX/Sec Latency (ms)

1 2 Kb 8 125 10 100

25 2 Kb 524 44.2 5,336 2.3

50 4 Kb 1,009 45.6 12,408 1.8

200 4 Kb 2,850 64.3 22,650 4.4

1200 8 Kb 14,913 75.2 38,430 25.6

Windows XP SP2, 512 Mb memory, JDK 1.4.2_05

Buffer Size Impact on Throughput

ATA/133MB/sec, 2MB,

7,200 RPM

SATA/ 150Mb/sec, 8MB,

10,000 RPM

Threads Buffer Size TX/Sec Latency (ms) TX/Sec Latency (ms)

25 1 Kb 324 72 7,800 1.5

25 2 Kb 561 39 5,330 1.7

25 3 Kb 202 124 250 99

Agenda

History & Requirements Implementation Status

Status

What’s done– Interfaces and Classes currently available

What needs to be done– Future enhancements

Projects Using HOWL

Interfaces (org.objectweb.howl.log)

LogEventListener– logOverflowNotification

LogRecordType– USER– CTRL (records generated by HOWL)

ReplayListener– onRecord

Basic Logger (org.objectweb.howl.log)

Configuration– Javabean exposes configuration parameters

Logger– Implements basic logging functionality

LogRecord– Used by replay to return journal records to the application

XA Logger (org.objectweb.howl.log.xa)

XACommittingTX– Represents entries in active transaction table

XALogger– LogEventListener moves COMMIT records– Extends Logger

putCommit putDone replayActiveTx

XALogRecord– Extends LogRecord

getTx isCommit

Future enhancements

Read single record(s) from journal– Requested by ActiveMQ project (DONE)

User Documentation Utilities to create, initialize, dump logs Event Log support

Event Log Support

HOWL intended for multiple users– Geronimo, JOTM, ActiveMQ

Many Logging interfaces– Commons-logging, ObjectWeb Monolog, J2SE Logger,

LOG4J

HOWL events need to be part of application log HOWL EventLog interface should allow application

to supply a log

Projects Using HOWL

ObjectWeb JOTM– Java Open Transaction Manager– jotm.objectweb.org

Apache Geronimo Transaction Manager– David Jencks integrated HOWL with Geronimo TM.– Refactored to accommodate MAVEN builds– Public class constructors allow for use with GBeans.

Codehaus ActiveMQ– Temporary message store

Questions?

Development Tools

Eclipse 3.0.1– http://eclipse.org

Apache ANT– http://ant.apache.org

JUnit– http://junit.org/index.htm

Findbugs– http://findbugs.sourceforge.net/

EMMA Code Coverage– http://emma.sourceforge.net/

Recommendations

Fastest drive available– 10,000 RPM SATA drive works very nicely

Separate physical drive to minimize head movement Pre-allocated files

– File grow requires use of “rws” allocation causing one additional IO for each write

– File grow may fail causing application failure