BIO PRESENTATION

Better Software Conference June 26 – 29, 2006 Las Vegas, NV USA

T17

6/29/2006 1:30 PM

TESTING SOA MIDDLEWARE: AUTOMATING WHAT YOU CAN'T

SEE

Jon Howarth and Robert Ryan Wells Fargo

Jon Howarth Jon Howarth is the current manager of the Enterprise Business Services & Information Management (EBS&IM) Regression, Tools, Data Warehousing, and Data Services QA groups at Wells Fargo Bank. Jon Howarth has over 7 years of QA experience with a focus on automation testing. He has experience in developing automation frameworks for a variety of technologies including the web, middleware (web services, MQ), and the mainframe. He is always looking for new ways to innovate with tools for efficiency.

Robert Ryan

Robert Ryan leads special projects for Enterprise Business Services QA (EBS QA) at Wells Fargo Bank. He has 11 years of experience building and implementing IT systems that support financial transactions. He has been developing HTTP-based messaging systems since 1998 and has worked with Innoventry, Pacific Gas & Electric, and Wells Fargo to define and implement web services. His current focus is implementing test tools and processes for service-oriented architectures.

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 1

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 2

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 3

Presentation Goal

• Describe the complexities of testing a web services/SOA (Service Oriented Architecture) based middleware in an enterprise environment and our solution to this problem

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 4

Frame the Problem

• The application• SOA one of the complexities• Rapidly changing• Contract compliance

• The backend dependencies• Not the owner of the environment• Volatile data

• The tools• Application does not have a GUI• Immature tool set

• The QA process• Not traditional QA scope• Not black box, post condition verification• Non intuitive failure analysis

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 5

Frame the Problem

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 6

Background & Format

• Presenter background• Enterprise middleware QA group• Mission critical bank SOA software

• Presentation format• Frame the problem• The search for the solution• Tools• Skill set• Framework• Metrics

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 7

The Search for the Solution

• The requirements• Data management requirements• Tool(s) requirements• Vendor requirements

• The identification of tools• Research• Leverage vendors

• Proof of concept• Acceptance criteria• Tool integration• Tool acceptance

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 8

Solution - Architecture

• Manual functional testing• Test case management and workflow• Automated test execution• Data management

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 9

Solution - Tools

• Manual functional testing • Soap Scope, Mindreef• An industry standard web service testing tool

• Test case management and workflow• Test Director for Quality Center, Mercury Interactive• Integrated with Integra Enterprise

• Automated test execution• Integra Enterprise, Solstice Software• Supports key protocols for SOA and integration testing

• Data management• Data Repository, developed in-house• Tracks data conditioning requirements

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 10

Tool Architecture

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 11

Our Skill Sets

• Implementation team• Broader than testing• Diverse skill sets

• End users• Established QA disciplines

– Subject matter experts – Junior testers

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 12

Project Methodology

• Analysis of requirements• Used analysis best practices• Understand the needs of the QA group• Included end-users• Workflow and reporting

• Solution architecture• Map requirements to tools• Define interfaces between tools

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 13

Project Implementation

• Goals and deliverables• Defined milestones• Dedicated team members• Iterative approach• Commitment and support

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 14

Framework

• Data driven model• Traceability from test cases to test scripts• Flexibility of test script suites• Council

• Medium for improving workflow, tool, and process issues

• QA discipline representation

• Infrastructure in place to automate future web service applications

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 15

Lessons Learned

• Management support• Full buy-in• Facilitate change

• Variety of reactions from the end-users• We could have used an ambassador• Migration of legacy scripts is complex• Keep homebuilt tools simple

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 16

Project Metrics

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 17

Framework Metrics

• Project execution status• Release readiness status• State of data• Adherence to best practices

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 18

Summary

• A solution is possible• Take the time to architect the solution• The investment pays off• Vendor support

• Treat it like a project• Have a methodology• Dedicated resources• Management support

June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 19

Questions