Distributed Middleware Factory

transcript

Ramón Arellano (Accenture) and Paul René Jørgensen (Telenor)

About us

Ramón Arellano

Work experience

6 years of experience as a technical consultant from

Accenture

E-mail

ramon.arellano@accenture.com

Paul René Jørgensen

Work experience

5 years of experience from Telenor

E-mail

paul-rene.jorgensen@telenor.com

Distributed Middleware Factory - definition

• Distributed

– In the geographical sense of the word, i.e. various locations spread around the world

• Middleware

– The communication layer that lets applications interact across hardware and network environments

• Factory

– A facility used to manufacture goods in an effective way, using standardized industrial processes and specialized equipment

Distributed Middleware Factory – definitioncontinued

Multiple teams in different locations,producing software for system integration,on the same platform,using standardized architecture and tools

So what are we going to talk about?

• Middleware platform for Telenor Fixed Line Services

• Metro - A distributed middleware factory

• In this talk we’ll share our experiences from its evolution and some secrets from the past 6 years

Epochs in Metro history

• Epoch 0 – Medieval Age

• Epoch 1 – Gunpowder Age

• Epoch 2 – Enlightenment

• Epoch 3 – Industrial Age

• Epoch 4 – Modern Age

Epoch 0 – Medieval Age

• Year 2000

• Challenges

– Where is the vision for the future?

– Use of the existing middleware platform is not mandatory

– Kings and kingdoms and their point to point integrations

Medieval Age – Gunpowder Age – Enlightenment Age – Industrial Age – Modern Age

The Metro Vision

All integration between systems for Telenor Fixed Line services to be implemented through Metro platform

Metro platform goals

• To offer test and production environments to all development project

• To enforce architecture and development standards

• To supply a common set of core components

Distributed development model

• The platform is managed and controlled by a small, centralized team

• The Telenor employer selects software supplier

• The software supplier performs the implementation on the platform

Centralized Platform Team, Oslo, Norway

Delivery Process,Stockholm,

Sweden

Customer Identification,Lillehammer,

Norway

Product Availability,

Halden, Norway

Line Information, Bangalore, India

Order Processing,

Oslo,Norway

60 other services, Oslo,

Norway

60 other services, Oslo,

Norway

60 additioinal services, Oslo,

Norway

Distributed vs Centralized

• Employer chooses software supplier

• Development is done where the supplier is

+ Easier to get new developers up and running

÷ Requires a stronger regime and documented standards

• Platform team chooses software supplier

• Development is done in-house

+ Easier communication between all teams involved

÷ Does not scale so easily

PlatformPlatform

EmployerEmployerEmployer

EmployerEmployerSupplier

EmployerEmployerEmployer

EmployerEmployerSupplier

Epoch 1 – Gunpowder Age

• Year 2001

• Challenges

– The volume and complexity of Telenor

– Common architecture

– How do we organize SI-testing?

Statistics

• In an average hour the platform is processing

– 61 200 web requests from the internet

– 115 200 EJB invocations

– 2 880 000 database queries

– 65 MB log messages (1,5GB in a day)

• Volume

– Over 50 systems integrated

– Average 200 deployments a year

– Over 250 different developers from all over the world have written code

– Metro platform team guides up to 30 teams at the same time

Name a technology, and we’ve got it

Technology Protocol

Oracle, Sybase, MySQL, DB2

JDBC, ODBC, DB2Connect

Siebel Siebel XML

IBM Mainframe (MVS) Terminal, Screen Scraping, MQ, CICS

Java J2EE, RMI, Corba

IIS / ASP Proprietary XML(JItegra Java COM bridge)

Web Services SOAP, Proprietary HTTP/XML

Tibco B2B EBXML

Layered architecture

Informal roles

• Experience has shown us the importance of having roles in a team

• The roles are more or less self imposed

• Examples

– The hardware geek

– The Maven guru

– The support girl

– The documentation dude

– The tool guy

Test Environments

• Implementation, building and unit testing on developer’s laptop

• System integration testing on 60 blade servers running the full J2EE runtime environment

– One for each development project!

• Acceptance Test environment administered by the platform team

Epoch 2 - Enlightenment

• Year 2002-2003

• Challenges

– We want to be more innovative

– How do we get implementation projects to help develop the platform?

– How do we organize our work and control the deliverables?

The 80/20 rule

• Innovative and flexible leaders introduced the 80/20 rule

• Developers were allowed to use 20% of their time to learn and play with new technologies

• Highly motivating, and has undoubtedly been fruitful for Metro

• Technologies such as multicasting (Log), FishEye (CVS), were introduced by experimenting

Boiling soup on stone

• The platform team is responsible for setting the standards for many projects and developers

• Dave Thomas's book, “The Pragmatic Programmer”, talks about “boiling soup on stone”

• Introducing new technologies and standards by curiosity and envy

• The whole package more valuable than each part

Metro Roadmap

• Our common workflow and methodology for all projects

• A set of phases with corresponding deliverables

• Waterfall model (may jump backwards and iterate)

• Each project gets their own supervisor responsible for QA

Roadmap control in Jira

• We’ve got a tool to enforce Metro Roadmap

• Atlassian’s inexpensive, but excellent tool Jira provides workflow functionality

• Controls the flow and stores all document deliverables

• It’s the platform team’s communication channel with development projects

Epoch 3 – Industrial Age

• Year 2004-2005

• Challenges

– Lack of documented standards

– New developers must get up and running fast

– We feel we could be more efficient in getting new applications out

Document everything

• Roadmap workflow

• Architecture principles

• Development environment setup

• Naming standards

• Step-by-step guidelines

• Testing and debugging

• Building and deployment

• FAQ

Documentation in Confluence

• Atlassian’s professional wiki, Confluence, for all documentation

• Wiki: easily updated, everybody gets the current document version

• Ported existing documents from MS Word

• Also used for blogging news and alerts through mailing lists and RSS feeds

We use Open Source

• Result of the 80/20 rule

– We’ve found free tools to improve our platform

• We love open source, and we give back

– Maven plugins

– Confluence plugins

– XRadar

– Jira plugins

Leading Star Example

• Leading Star Driven development

• Improves application consistency

• TelenorShop – a fictitious application

– It shows live examples of all our common components and frameworks in use

• Source code for all Metro applications is openly available internally

“Take a look at TelenorShop,and see how we’ve done it there”

Automation through 3rd party tools

• Automatic builds with unit testing using Maven

• Generating web-site with reports and JavaDoc using Maven

• Continuous integration using CruiseControl

• Mapping between data models using Dozer

• Support request through Jira

Automation through self-made tools

• Applying for deployment slots to acceptance test environment

• Generating overview of dependencies between all J2EE applications on server

• Searching logs in all environments

• Updating application configuration

• Backing up WebSphere Application server configuration

• A good factory scales easily

• New resources must be brought up to speed – fast

• No development without certification

• J2EE is a prerequisite

• Certification courses for

– Developers (2 day course)

– Project Leads (1 day course)

• 100 certified developers and project leads

Metro Certification

• Efficient development

• Consistent solutions

• Stable solutions

• Reusable services

• Better developer attitude

– “Ah, so that’s why we have to...”

• Externals pay a course fee ($)

Other certification benefits

Metro Certification content

Metro Supervisors Metro Support

Metro Certification Metro Guidelines

Metro Architecture

Metro Commons Comet

Metro Commons Frameworks

Metro Online Tools

IBM WAS Java SDK / J2EE Security / LDAPCVS and Maven

Repositories

DevelopmentEnvironment

System IntegrationEnvironment

Acceptance TestEnvironment

ProductionEnvironment

Business Logic

Epoch 4 – Modern Age

• Right now, right here

• Challenges

– Middleware is only noticed when something goes wrong

– How do we govern dependencies?

– Deployments are held back due to poor code quality

Commercialization

• We must learn how to better sell ourselves internally

– We want more applications on the platform and everyone to be aware of what Metro has to offer

• How

– Create a logo

– Offer introduction presentations

– Show off statistics of usage

– Present at JavaZone

Service Level Agreements

• SLAs between API owners (the kings)

– Wanted since the medieval age!

– No access to remote API without a signed SLA

• It will give us valuable information about inter-application dependencies

• We must work on enforcing this in the modern age!

– Requires a new tool (Any suggestions?)

Code Quality

• Step-up the code QA work

• Make it easier for the reviewers to focus on code that is being altered

– Reviewers are notified immediately on CVS commit

• We’ll introduce Crucible into the review process

– Defines a workflow, creates an arena for dialog

Architects must write code

“Real insight comes from active coding”

Practices of an Agile Programmer

Summary

• Key tools

• Key lessons

Key tools

• Confluence (wiki) for documentation

– No delay from writing to publication

• Jira

– No more forgotten issues in inboxes all over the place

– Issue management and process (roadmap) workflow

• Maven

– Stable and proven build system

• Eclipse based IDE

– Allow only one IDE

– Be an expert to support the developers

Key Lessons

• Easy to understand workflow for everybody – Roadmap

• Documented standards and guidelines – Wiki

• And a way to teach them (Certification)

• Decision from the top: All integration between systems to be implemented through the platform

Any questions?

paul-rene.jorgensen@telenor.com / ramon.arellano@accenture.com