© Copyright 2015 Vivit Worldwide© Copyright 2015 Vivit Worldwide

“Developing a Comprehensive Test Automation QA Strategy”

August 11, 2015

© Copyright 2015 Vivit Worldwide

Brought to you by

© Copyright 2015 Vivit Worldwide

Hosted by

Robert Linton

Testing, Quality and ALM SIG Leader

VP ALM at CorTechs, Inc.

© Copyright 2015 Vivit Worldwide

Today’s Speaker

Becky Karch

Director of Customer Success

TurnKey Solutions

© Copyright 2015 Vivit Worldwide

Housekeeping

• This “LIVE” session is being recorded

Recordings are available to all Vivit members

• Session Q&A:

Please type questions in the Questions Pane

© Copyright 2015 Vivit Worldwide

Webinar Control Panel

Questions

Toggle View Window between

Full screen/window mode.

Developing a Comprehensive Test Automation QA Strategy

TurnKey Solutions

August 2015

What We Will Cover

- Assessing your current state of test automation readiness

- Optimizing test automation, people, and processes

- Benefits of data-driven, scriptless test technology

- Maintaining regression tests

- Leveraging a mix of tools and technologies in your

testing

- Quantifying value from your test automation

Polling Questions

What is the biggest challenge you face with your software delivery cycle?

Testing process is slowing us down Knowing when to release Difficulty getting the teams organized Finding the right tools to streamline the process

Delivery

Feedback

Development

The Release Cycle

Faster release cycles shorten the gap between what the customer wants and how quickly you are able to deliver

Testing is typically the #1 time sink in the delivery process –this requires getting smart about what you test and how you execute your tests

Setting Quality GoalsApplication

#Tests Planned #Executed

%Executed of Tests Planned

#Passed

%Passed of Tests Executed

%Failed of Tests Executed

%Blocked of Tests Executed

Admin Rx HH Total 97 97 100.00% 97 100.00% 0.00% 0.00%

CAF Total 3 3 100.00% 3 100.00% 0.00% 0.00%

Carelink Total 223 122 54.71% 120 98.36% 1.64% 0.00%

CTM Total 22 10 45.45% 9 90.00% 0.00% 10.00%

EDTB Total 6 6 100.00% 6 100.00% 0.00% 0.00%

eMAR Total 565 565 100.00% 523 92.57% 1.59% 5.84%

Enterprise Prescription Writer Total 522 453 86.78% 443 97.79% 1.99% 0.22%

HCR Total 6 2 33.33% 2 100.00% 0.00% 0.00%

HEC Total 425 407 95.76% 403 99.02% 0.98% 0.00%

HED Total 26 25 96.15% 25 100.00% 0.00% 0.00%

HEN Total 746 694 93.03% 649 93.52% 1.73% 4.76%

HEO Total 3955 2916 73.73% 2786 95.54% 2.40% 2.06%

HEP Total 7 7 100.00% 7 100.00% 0.00% 0.00%

HHS Total 3130 1224 39.11% 1044 85.29% 13.89% 0.82%

HMR Total 120 118 98.33% 109 92.37% 7.63% 0.00%

HMS Admin Tool Total 4 3 75.00% 3 100.00% 0.00% 0.00%

HOM Total 783 755 96.42% 755 100.00% 0.00% 0.00%

Horizon Care Alerts Total 19 18 94.74% 17 94.44% 5.56% 0.00%

HPC Total 42 33 78.57% 33 100.00% 0.00% 0.00%

Open Connect Total 34 14 41.18% 13 92.86% 7.14% 0.00%

OTK Total 1244 925 74.36% 912 98.59% 1.41% 0.00%

PRCS Total 244 173 70.90% 90 52.02% 0.00% 47.98%

Reports Total 186 56 30.11% 55 98.21% 0.00% 1.79%

Reports - Oracle Total 36 23 63.89% 23 100.00% 0.00% 0.00%

Grand Total 12456 8660 69.52% 8138 93.97% 3.46% 2.56%

Test metrics must represent overall product quality and be directly related to business goals

What Should You Test?

Regarding the application(s) under test, answer these questions:

What would cause the company to lose money? What would cause the company to lose business? What is the environment in which I need to run my tests? What types of issues are “escaping” the testing process? What are the high-risk end-to-end business processes? What are the processes that your business runs every day?

Your tests should reflect “Meaningful Use” of your System

Assess Current TestsWhat do your current tests do?How are they written and do they cover what they should cover?How stable is your application?

Develop Your Plan - Assess What You Have

Staffing The Necessary Testing ResourcesWhat skills does your team have?Can you hire more resources?Can you share/integrate roles & responsibilities?

Assess Test ToolsHow and where do you organize your tests?Do you have an automated test tool? Do you use it? How often?

Getting FeedbackWhat Support issues come in?Do you have enhancement request tracking?What services do your customers ask for?

Develop the Plan – Narrow the Focus What tests do you need to meet your quality goal?

Scope

Approach

What needs to be automated vs. run manually?

Setting up the environment

Test Data

Roles, Responsibilities & Credentials

Documentation

Segment the application into components as UI screens become available

BPT supports this well, but changes happen rapidly

Organizing for Success

Test Manager/Director

XYZ Division

Test Lead

App X

Test Automation Engineer (TAE)

App X

Test Architect

App X

Business Analyst

App X

. . .

Developer

App X

Everyone on the team must be familiar with the release in a continuous delivery model

“Day in the Life”

Bu

sin

ess

An

alys

tTe

stLe

adTA

ETe

stA

rch

itec

t

DocumentsTest

Strategy & Standards

Verifies Test Artifacts /

Compliance to Standards

Test Planning Test Design

Supports DataSheet

Entry

Creates & Unit Tests

Components

Participates in Requirements

Review

Participates in DataSheet

Entry

Assembles Test Scripts

Determines Component

Needs

Test Execution

Monitors Test

Execution

Reports Test Progress /

Defects

Document Functional

Requirements

Participates in Test Design Review

Enters Data into

DataSheets

IDs Test Risks and Issues

Coordinates Test

Estimates

Assists with Test

Estimates

Provides Tech Test

Direction to TAE

Executes Test Scripts

Opens Defects &

Retests

Maintains Components

& Tests

Ensures that ALM Test

Environment is Up-to-Date

Reviews Test Results

IDs Business Risks and

Issues

Organization-wide

• SCRUM teams form for each application or functional area

Scrum Team

Application Z

Test Lead

App Z

Test Automation Engineer (TAE)

App Z

Test Architect

App Z

Business Analyst

App Z

Scrum Team

Application Y

Test Lead

App Y

Test Automation Engineer (TAE)

App Y

Test Architect

App Y

Business Analyst

App Y

Scrum Team

Application X

Test Lead

App X

Test Automation Engineer (TAE)

App X

Test Architect

App X

Business Analyst

App X

• META SCRUM meets to ensure “meaningful use” end-to-end flows

Building the Regression Suites

Release CRelease BRelease A

Automation

A

Regression

A

Regression

A

Regression

B

Automation

B

Final Release to Production

Regression

A

Regression

B

Regression CAutomation

C

Continuous Testing

How Scriptless Automation Can Help You Use the people in your organization who know the end-to-end business

flows (“meaningful use”) the most

Break up your application into components (or screens) that can be

assembled into tests and reused as needed

Allows you to run end-to-end tests

within a single application

or spanning multiple applications

Keeps your data organized and attached to the relevant tests

Allows you to easily maintain tests when things change

Maximizes your testing time to get to faster release

Continuous Test Development

Up

Dev

.Te

st/Q

AB

iz A

nal

yst

Sprint 1 Sprint 8Sprint 2 Sprint 3 . . .

• Analyze requirements• Develop first screen• Create first component

• Develop additional screens

• Develop additional components

• Begin to assemble test

• Add underlying structure

• Change interface for one screen

• Update components and data

• Execute tests • Debug tests• Fix product bugs

Requirements1 – display this2 – enter xyz in field13 – Navigation links on left4 – graphic must be centered

Test PlanTest 1: do thisTest 2: then do thatTest 3: try to break ITest 4: do it again X5

Maintaining Your Tests

Tests must be easy to maintain without re-coding and debugging

Application-aware test tools make this easier Update components and not entire test scripts Side-by-side comparison of new, deleted, or changed UI

fields Automatically identify changes in your data sheets

Building End-to-End “Meaningful Use” Tests

Requisition Requisition Summary

Individual data scenarios within your data sheet

Value from AutomationTestsPlanned

TestsExecuted

PercentTests Executed

Tests Passed

Percent of Executed TestsPassed

Percent of Executed Tests Failed

Percent of Planned Tests Passed

TOTAL 5514 5306 96.23% 5222 98.42% 1.58% 94.70%

Automating “Meaningful Use” gives your organization confidence in product quality

Regression test sets build over each sprint and release With test automation you can measure the time to

release savings Measure efficiency of scrum teams by their velocity and

defect backlog

Continuous Delivery Process

Application Development

Requirements Development

System TestingAutomated

Testing

Requirements Tracking

Automated Build

Production Release

Leverage a Mix of Tools

Agile user story tracking

Version Control System

Automated Build System

Defect tracking

Test management tool

Test automation tool

Data management tool

Use Customer Feedback in the SDLC

Measure “escapes” from each release Support Tickets User Forums

Customer Satisfaction Survey information Customer Advisory Boards

© Copyright 2015 Vivit Worldwide

Thank you

• Complete the short survey and opt-in for more information from

TurnKey Solutions.

www.turnkeysolutions.com

www.vivit-worldwide.org