BIO PRESENTATION
Better Software Conference June 26 – 29, 2006 Las Vegas, NV USA
T17
6/29/2006 1:30 PM
TESTING SOA MIDDLEWARE: AUTOMATING WHAT YOU CAN'T
SEE
Jon Howarth and Robert Ryan Wells Fargo
Jon Howarth Jon Howarth is the current manager of the Enterprise Business Services & Information Management (EBS&IM) Regression, Tools, Data Warehousing, and Data Services QA groups at Wells Fargo Bank. Jon Howarth has over 7 years of QA experience with a focus on automation testing. He has experience in developing automation frameworks for a variety of technologies including the web, middleware (web services, MQ), and the mainframe. He is always looking for new ways to innovate with tools for efficiency.
Robert Ryan
Robert Ryan leads special projects for Enterprise Business Services QA (EBS QA) at Wells Fargo Bank. He has 11 years of experience building and implementing IT systems that support financial transactions. He has been developing HTTP-based messaging systems since 1998 and has worked with Innoventry, Pacific Gas & Electric, and Wells Fargo to define and implement web services. His current focus is implementing test tools and processes for service-oriented architectures.
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 1
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 2
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 3
Presentation Goal
• Describe the complexities of testing a web services/SOA (Service Oriented Architecture) based middleware in an enterprise environment and our solution to this problem
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 4
Frame the Problem
• The application• SOA one of the complexities• Rapidly changing• Contract compliance
• The backend dependencies• Not the owner of the environment• Volatile data
• The tools• Application does not have a GUI• Immature tool set
• The QA process• Not traditional QA scope• Not black box, post condition verification• Non intuitive failure analysis
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 5
Frame the Problem
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 6
Background & Format
• Presenter background• Enterprise middleware QA group• Mission critical bank SOA software
• Presentation format• Frame the problem• The search for the solution• Tools• Skill set• Framework• Metrics
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 7
The Search for the Solution
• The requirements• Data management requirements• Tool(s) requirements• Vendor requirements
• The identification of tools• Research• Leverage vendors
• Proof of concept• Acceptance criteria• Tool integration• Tool acceptance
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 8
Solution - Architecture
• Manual functional testing• Test case management and workflow• Automated test execution• Data management
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 9
Solution - Tools
• Manual functional testing • Soap Scope, Mindreef• An industry standard web service testing tool
• Test case management and workflow• Test Director for Quality Center, Mercury Interactive• Integrated with Integra Enterprise
• Automated test execution• Integra Enterprise, Solstice Software• Supports key protocols for SOA and integration testing
• Data management• Data Repository, developed in-house• Tracks data conditioning requirements
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 10
Tool Architecture
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 11
Our Skill Sets
• Implementation team• Broader than testing• Diverse skill sets
• End users• Established QA disciplines
– Subject matter experts – Junior testers
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 12
Project Methodology
• Analysis of requirements• Used analysis best practices• Understand the needs of the QA group• Included end-users• Workflow and reporting
• Solution architecture• Map requirements to tools• Define interfaces between tools
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 13
Project Implementation
• Goals and deliverables• Defined milestones• Dedicated team members• Iterative approach• Commitment and support
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 14
Framework
• Data driven model• Traceability from test cases to test scripts• Flexibility of test script suites• Council
• Medium for improving workflow, tool, and process issues
• QA discipline representation
• Infrastructure in place to automate future web service applications
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 15
Lessons Learned
• Management support• Full buy-in• Facilitate change
• Variety of reactions from the end-users• We could have used an ambassador• Migration of legacy scripts is complex• Keep homebuilt tools simple
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 16
Project Metrics
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 17
Framework Metrics
• Project execution status• Release readiness status• State of data• Adherence to best practices
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 18
Summary
• A solution is possible• Take the time to architect the solution• The investment pays off• Vendor support
• Treat it like a project• Have a methodology• Dedicated resources• Management support
June 29, 2006 Automating what you can’t see: Testing middleware for the enterprise 19
Questions