10/14/2015

1

Automated TestingQUICK ORIENTATION & PRODUCT DEMONSTRATION

Zuzena automated test engine –

5) Customer Benefits

4) Scenarios & Repositories

3) Demo Core Capabilities

2) Anatomy of a Test Case

1) Customer Goals

• Data-aware

• Scenario-driven

• Repository-based

2

10/14/2015

2

Customer Goals for the QA Process

Simplicity

• A process that developers, management, & sponsors understand

Accessibility

• Developers can drive unit test cases

• System testers can add integration test cases

• SMEs can contribute acceptance test case

Labor Efficiency

• Reduce QA’s percent of budget while increasing test coverage & quality of results

Visibility

• Translate complex process into a few, intelligible graphs

3

Zuzena automated test engine –

5) Customer Benefits

4) Scenarios & Repositories

3) Demo Core Capabilities

2) Anatomy of a Test Case

1) Customer Goals

• Data-aware

• Scenario-driven

• Repository-based

4

10/14/2015

3

Anatomy of a Test Case

Container

•The Test Case

•How granular?

•Reusable components?

•Runtime parms?

Source Data

•From where?

•How much at once?

•Clean, dirty, incomplete, misorderd?

ETL Call

•EDW layer?

•Initial or incremental?

•Mid month, end of month or EOY?

Actual Results

•Collisions between testers?

•Text, DBMS?

Expected Results

•Subsets to match source data?

Assertions

•Formulas?

•Statistics?

•Comparisons?

•Further ETL?

Outcomes

•Tagged by versions, build, iterations, run dates?

•Drill down

5

Development Integration System Acceptance

Test Case Workflow

Ceregenics proprietary information. 6

Test DataActual

Results

EDW

Test

Scripts

Outcomes

Expected

Results

Assertions

10/14/2015

4

SSH &

ODBC

Engine-Driven Test Case

Ceregenics proprietary information. 7

Test DataActual

Results

EDW

Test

Scripts

Hyper Cube of

Test Scenarios

Data Set

Outcomes

Expected

Results

Assertions

Zuzena automated test engine –

5) Customer Benefits

4) Scenarios & Repositories

3) Demo Core Capabilities

2) Anatomy of a Test Case

1) Customer Goals

• Data-aware

• Scenario-driven

• Repository-based

8

Build a test case

10/14/2015

5

On-Demand Demo on YouTube

Web addresses for the demo:

• http://youtu.be/YX8lKSACELA

• https://vimeo.com/141469720

Locations of major narrative points:

• Create the test case container 0:19

• Add a input data source 2:17

• Add an ETL action step 5:56

• Add an validation assertion 9:10

• Reusable test widgets 9:50

• Run an assertion that fails 12:10

• Fix assertion 13:43

• Re-run to see it pass 14:25

9

Zuzena’s Run Modes

Via the GUI

• Developing test cases

Via the Command Line

• preparing an automated run

• https://youtu.be/Bn27KkHyCCY

Via the scheduler

• nightly regression test of customized test suite

10

Major narrative points

0:20 a command to run a dynamically defined test suite

0:45 the same command with parms annotated

1:05 viewing the results

10/14/2015

6

Zuzena automated test engine –

5) Customer Benefits

4) Scenarios & Repositories

3) Demo Core Capabilities

2) Anatomy of a Test Case

1) Customer Goals

• Data-aware

• Scenario-driven

• Repository-based

11

User-Defined Metadata Tags

Structure: <tag type>, <tag value>

12

Tag Type Sample Values

Error Class Nominal Data, Dirty Data, Missing Data

Environment Unit, Integration, System, Acceptance

Time Point Day 0, Day 1, End-of-Month, EOY

Project Master Customer, Sales, Finance

App Layer Landing, Integration, Presentation, Dashboards

Column Type Replicated, Derived, Source Metadata, EDW Metadata

10/14/2015

7

RepeatableTesting: Scenario Cycle

Ceregenics proprietary information. 13

Test DataActual

Results

EDW

Test

Scripts

Hyper Cube of

Test Scenarios

Error Class

Outcomes

Expected

Results

Assertions

Zuzena automated test engine –

5) Customer Benefits

4) Scenarios & Repositories

3) Demo Core Capabilities

2) Anatomy of a Test Case

1) Customer Goals

• Data-aware

• Scenario-driven

• Repository-based

14

10/14/2015

8

Benefit: Single Repository = Structure + Organization

15

From this...

To this...

Repository Benefits

Remember where you left it

• Far less time hunting and gathering

Know what it should look like

• Understand it when you find it

Fast find + fast recognition

• faster overall testing

Lower risk

• Less project management stress

16

10/14/2015

9

Benefit #2

With a repository,

you can count what

you should have

17

Benefit #1: Visible Progress to Goal

18

10/14/2015

10

Benefit #3

Repository + User-

Defined Tags

Categorize,

visualize,

and understand

19

Benefit #1 – Drill Down on Problem Areas

20

Project: Master Customer

(Part 1)

Huh?

10/14/2015

11

Benefit #1 – Drill Down on Problem Areas

21

Project: Master Customer, Landing Zone

(Part 2)

OK

Benefit #1 – Drill Down on Problem Areas

22

Project: Master Customer, Integration Layer

(Part 3)

Hmmm....

10/14/2015

12

Benefit #1 – Drill Down on Problem Areas

23

Project: Master Customer, Integration Layer, Derived Columns

(Part 4)

Hmmm....

Benefit #1 – Drill Down on Problem Areas

24

Project: Master Customer, Integration Layer, Derived Columns, Global Accounts Subject Area

(Part 5)

There’s the

problem

10/14/2015

13

Zuzena automated test engine –

• Data-aware = Elements for source and expected data

• Repository-based = Structured & organized

• Scenario-driven = User tags allow subsetting

25

Customer Goals for the QA Process

Simplicity

• A process that developers, management, & sponsors understand

Accessibility

• Developers can drive unit test cases

• System testers can add integration test cases

• SMEs can contribute acceptance test case

Labor Efficiency

• Reduce QA’s percent of budget while increasing test coverage & quality of results

Visibility

• Translate complex process into a few, intelligible graphs

26

Structured test definition with reuable components

GUI makes defining test cases intelligible, trainable, & repeatable

More tests defined in less time, far more runs within each project

Repository, subset reporting yields trend lines for team, PMs, and sponsors

10/14/2015

14

Acquiring Zuzena

First repository: usually a DEV host for programmers

Second repository: usually an SIT host for system testers

Discounts for UAT and PROD hosts

Procurement Patterns

• Host licenses alone

• Hosts + installation & training

• Comprehensive QA method & plan + hosts + installation & training

27

Comprehensive QA Planning

Identify the best of 40 test types to use

Negotiate a unit test pattern with programmers

Define a funnel & promotion process to SIT, UAT, & PROD

Establish reusable test widgets

Document who, when, where

Enumerate tests by requirement & component project targets

Derive user metadata tags, create compliant repository

Prioritize and implement repository reporting for dev, PM, sponsors

28

10/14/2015

15

999 18th Street, Suite 3000

Denver CO 80033303.274.9101

WWW.ZUZENA.COM

29