Migration of large applications from

WebSphere to JBoss

Markus Dahmakquinet tech@spree

markus.dahm@akquinet.de

Who I am

Markus Dahm: §Master thesis in computer science in 1996 on

nested-parallel data models§Research assistant at Freie Universität Berlin§Project founder of Apache BCEL§Project founder of Apache BCEL

Since 2002:§Senior Architect and developer for akquinet§Technical Architect in J(2)EE projects§Project lead§Reviews and Coaching

Agenda§WebSphere environment and application architecture§The plan …§Migration–Application “modernization”–Application “modernization”

§Problems occurred/problems solved§Lessons learned

Application mission

Consolidation and integration

4

Technical environment (1/2)

Several different applications with some common components

§EJB-2 session beans, Swing-based fat clients, custom O/R mapping

framework

§Strategic decision by customer for IBM hardware and software

→ WebSphere application server→ WebSphere application server

→ DB2 RDBMS

→ AIX

§ In-house development on vanilla PCs

→ Single (slow) application server for all

developers

5

Technical environment (2/2)

The setting for (almost) all projects was

§WebSphere application server 5 on AIX

– Embedded MQSeries support

§DB2 V8 RDBMS on AIX§DB2 V8 RDBMS on AIX

§EJB 2.1 via XDoclet

§Third party systems:

– SAP R/3

– MapInfo GIS

6

Development cycle

§Deploy times > 10 minutes

– Workaround with MockEJB (dead project by now…)

– unsatisfied developers

– low productivity– low productivity

§Deployment errors hard to debug, e.g. weird “AWT headless

exception”

§Cumbersome development with XDoclet

§Early wish to switch to JBoss, got rejected

7

Agenda§WebSphere environment and application architecture§The plan …§Migration–Application “modernization”–Application “modernization”

§Problems occurred/problems solved§Lessons learned

At last …

Reason for change of mind was

§Wanted to switch from WAS 5.1 to 6.x

– EJB-2.1 → EJB-3 targeted

– Java 5 after all– Java 5 after all

§Client run-time was not applicable for the customer’s environment

– After 3 month

– Even with official IBM support

→ Approval for JBoss 4.2.3 (.ORG at first)

9

Migration strategy

Smallest project with minimal risk first (1)

§With typical features

§No hard dead lines at that time

§Already renovated project structure

Gather experience (2)

§Guide lines, architecture

§Example implementations

§ Internal article

Finally (3): Parallel Migration of all other projects

10

Beyond migration: Modernization

Modernization Goals:

§Migration to Java 5 (Annotations and Generics)

§Architectural redesign

§Moving from EJB-2.1 to EJB-3 (POJOs)§Moving from EJB-2.1 to EJB-3 (POJOs)

§Replacing the JDBC-based OR-Mapping with standard JPA/Hibernate

– Optimized data model and queries

– First level cache for free

– Second level cache?

§Simplified development and deployment

11

Agenda§WebSphere environment and application architecture§The plan …§Migration–Application “modernization”–Application “modernization”

§Problems occurred/problems solved§Lessons learned

First candidate: Water meter reading (Ablesesteuerung, ALS)

Domain

§Reading and evaluating water

meters

§Accounting with SAP

§Controlled via web application§Controlled via web application

Application size

§120 TLOC

Project state

§Modernization planned

(EJB3, JPA, Maven)

13

1:1 Migration from WebSphere to JBoss

Proved to be quite simple in fact ...

§Create a new XA-Data-Source

§Configure JMS Topics and Queues

§Migrate the EJB-2 session beans§Migrate the EJB-2 session beans

– Extend XDoclet configuration comments

– Change XDoclet configuration in ANT build

§Simplified deployment for client PCs

– No WebSphere runtime environment required

Investment: 11 man-days

14

ALS-Modernization

§ “Mavenization”, application architecture redesigned

§Simplified OR-Mapping with JPA

– Less code J

– Transformation of RDBMS-specific code 6– Transformation of RDBMS-specific code 6

– Problems with true LIEs (LazyInitializationException) L

§Dependency injection with EJB-3 J

§Good Unit and integration tests prove to be a “conditio sine qua non” !

(a condition you can’t do without)

Investment altogether: ~70 man-days

15

Benefits for ALS project?

Project leader:

§Estimated 20-30% acceleration through modernization

Controller:

§ Iteration after modernization up to 15% faster than estimated before§ Iteration after modernization up to 15% faster than estimated before

Developers:

§Developer quickly adopted new technologies

§Faster, better development

16

Next step

Parallel migration of other (much bigger) projects

§ “Update” to EJB-3, but no further modernization

§WV

605 TLOC → 27 man-days estimated

§AE-K

459 TLOC → 25 man-days estimated

§UZI

68 TLOC → 4 man-days estimated

17

Some challenges (1/2)

Isolation of applications

§Single classclassclassclass----loaderloaderloaderloader----repositoryrepositoryrepositoryrepository per application is not enough

– JNDI requires CallByValue=trueCallByValue=trueCallByValue=trueCallByValue=true

WebSphere (and AIX servers) require to use IBM JDK 1.5WebSphere (and AIX servers) require to use IBM JDK 1.5

§Clients use JRE 1.6

– Be careful not to use Java 6 API

§Some adaptations for printing graphics

§Other minor subtleties, but nothing unsolvable

18

Some challenges (2/2)

Problems in the applications

§Detected cycles in EJB dependencies (ejb-ref)

§Some application required disabling JBoss’ CallByValue-optimization

for remote EJBs

§JBoss is stricter on validation of deployment descriptors

§Parallel existence of EJB-2 and EJB-3 during migration

19

Spin-off project: JBoss configuration

Project to configure JBoss server for different settings

§Optional upgrade of JMS sub system

§Delete unnecessary services of default configuration

§Every project’s server gets its own set of ports§Every project’s server gets its own set of ports

§Different data bases and schemas for

– Local development

– Test system

– Production system

§Use .ORG or EAP

20

Well done!?1 month testing with users

→ Projects go productive

21

Goals accomplished?

§Migration to JBoss of all applications ü

§Full migration to Java 5 ü (by now)

§Migration from EJB-2.1 to EJB-3 ü

§Architectural redesign L 6§Architectural redesign L 6

§Using JPA/Hibernate K (partially)

§Simplified development and deployment J

22

Agenda§WebSphere environment and application architecture§The plan …§Migration of the applications–Application “modernization”–Application “modernization”

§Problems occurred/problems solved§Lessons learned

... Problems in production

§One application caused a

server side memory leak

requiring reboot after 4h

– Slow-down

§Massive disturbance of §Massive disturbance of

operating activities

§No way back!

L24

PriorityPriority 1: Re1: Re--enableenable operationsoperations

PriorityPriority 2: Error 2: Error analysisanalysis

25

A dog may have fleas AND lice…

... collective effect of multiple bugs cause confusing symptoms.

§ Introduction of „Keep-Alive-Job“ because of JMS problems

– caused huge memory consumption in JMS-Topics and Queues

– Disarmed configuration

→ application survived longer→ application survived longer

§Further hints of problems with JBoss-Remoting (many technical

objects in memory)

– Discovery of incompatible libraries on client and server

→ Version update and alignment

– jboss-client-all.jar had been forgotten

→ application survived even longer

26

Free space for true error analysis

4 day analysis by two developers

Result:

§Firewall “swallowed” TCP

connection after 1h of inactivity

without alerting sender or receiver.without alerting sender or receiver.

§Standard configuration of JBoss

Remoting did not recognize this

Solution: adapted configuration

→ No more problems

→ Phew!

27

Accolade!

Article with detailed description in our blog

http://blog.akquinet.de

First comment from Ron Sigal (Core Developer JBoss Remoting)

Bravo! Great job of analysis.

I was aware of your solution, and, to be honest, I didn’t see why it

worked. Your blog has sent me back to think about it, and now I

understand....

:-)

28

Punch line

§Probably WebSphere had the same

problem, but reacted differently

§Silent filling of JMS buffers (dead letter

queues on file system) until system halt

§Silent solution at that time:§Silent solution at that time:

Regular purging of queues via shell

script

→ A clear pro of Open Source, similar

debugging would have been impossible

without source code

29

Agenda§WebSphere environment and application architecture§The plan …§Migration of the applications–Application “modernization”–Application “modernization”

§Problems occurred/problems solved§Lessons learned

Time for campfire stories…

31

Recommendations

Pre-condition for migration (from whatever application server)

§Take time and opportunity to refactor your code before migration

– Halt development of features

– Migration of modules one by one– Migration of modules one by one

§Write good tests to prove your application still works J

§Stick to (JEE) standards whenever possible

– No lock-in

32

Pros

§Faster development and (hot) deployment

§Easier testing of components

§Open Source vs. WAS embedded licenses

§More control§More control

§No special client run-time environment required

§Easier configuration

– Server can easily be re-created

33

Cons

§Difficult to estimate costs

§Up to three times more time needed

– Estimated 27, needed 90 man days in one project

– Most of it due to modernization– Most of it due to modernization

§Cost-efficient migration is possible!

– If preconditions are met

– Yet be ready for surprises…

34

35