Test Automation

Principles, Terminologies & Implementations

Steven LiSteven004@gmail.com

2

Outline

•Test Automation Principles & Terminologies Why Test Automation How to do Test Automation Terminologies about Test Automation (Sycamore)

• Implement the Terminologies in Automation Scripts General Conventions An Example - PET

Why Test Automation

4

Top 10 Reasons for Automating Tests - What people said

1. Manual testing sucks.

2. Tool vendor said “Capture replay works.”

3. Fun to watch the dialogs popping on the screen.

4. “Afterwards we can fire all those pesky testers.”

5. Everybody else is doing it.

6. “No humans were harmed in the testing of this software.”

7. Big bucks already spent on the test tool.

8. Looks good on the resume.

9. No Testing for Dummies book ... yet.

10.Keep the intern busy.

5

Reasons for Automating – What people want • Speed up testing• Allow more frequent testing• Reduce manual labor costs• Improve test coverage• Ensure consistency

• Simplify testing• Define the testing process• Make testing more interesting and challenging• Develop programming skills• Justify cost of the tools• Of course we’ll have automation!

How to do Test Automation

7

Keep it Simple, Sustainable

•Uniform/Common infrastructure

•Appropriate tools

•Manage source code, test data, tools

•Agreed Conventions within the team

•High-quality Scripts

•Gradual Test Automation

8

Quality of Automation Scripts•Maintainability

Will the tests still run after product design changes? Will tests for 1.0 work with 2.0? And can be easily updated?

•Reviewability Can others review and understand what is being covered? Are the scripts documented?, can we make sure it matches the original design? How about the coverage? How easy another engineer use the scripts for the next release?

•Reusability To what degree the assets can be reused to create more, different tests? Can they be reused in different releases, in different environments?

•Dependability Integrity Reliability

•Repeatability•Independence•Performance•Simplicity

Terminologies for Test Automation

10

Architecture for AutomationTest Planning Test Preparation Test Perfecting Test Performing

Test Suite

Logical TestBed

Test Case

Test Case

…

Test Case

Test Suite

…

Physical TestBed

Test Case

Test Case

…

Test Case

Test Case

Logical TestBed

Test Case

Test Case

…

Test Case

Test Case

Test Plan

…

Test Step 1Test Step 2…Test Step n

Test Task

…

…

Test Report

Specify TestBed & TestCase ListFor a release Set a task to run a plan

ScheduleRuntime Options +

ScheduleRuntime Options +

Detailed Runtime InformationDetailed TestBed Information

Test Case Result

Test Case Result

…

Test Case Result

Detailed Runtime InformationDetailed TestBed Information

Test Case Result

Test Case Result

…

Test Case Result

Revised Test SuitesTest Plans

Logical TestBed

Test CaseTest Case…

Test Case

Physical TestBed

Test Case

Test Case

…

Test Case

Execute a task

11

Test Bed•Briefly, a test bed is the definition of the environment for a specified test. A test bed include the hardware/software and configuration/provisioning.

•There are two kinds of test beds: Logical TestBed. A logical test bed depicts all the elements and

configuration required for the test, but not indicate which exactly elements will be used (i.e. the index or addresses are not required.)

Physical TestBed. A physical test bed is an instantiation of a logical test bed. A physical test bed exactly indicates which elements are used in the test, and a test task can be really run on the test bed.

• Base State of a test bed The initial state of the test bed, ready for test case (group) start to

run with it. A self-cleaning test case (group) should get the test bed back to the

base state when the execution completed.

12

Test Case (group)•A test case is a sequence of operations against a system whose results can be used to determine if a requirement or use case upon an application of the system is partially or fully satisfied.

A test case contains a set of test steps, and based on a particular test bed.

A test step is an atomic statement of a test case, can have pass/fail result. Usually it contains one operation and the expected result.

A set of test cases can be grouped in a test case file.

•Test Case File (test group): A test case file defines one or multiple test cases, which is based on

a clean test bed for running, and do self-cleaning after performing all the cases one by one.

A test group are a set of test cases saved in one test case file.

13

Test Suite

•A suite of test cases, which share the same test bed to verify a specified feature in a specified test area. Usually, there are multiple test cases in one test suite.

•In automation scripts, a test suite includes: A test suite file: to describe the basic information of this test suite A logical test bed: to define the test bed which the test suite is based

on One or more test case files: each test case file is self-cleaning, so

that each file can run independently.

•Usually, test suite is feature oriented. It can be created once the feature is introduced, and keep being improved along the product life, or even product family life.

14

Test Plan•A test plan is an instantiation of a subset of a test suite, in which a set of test cases selected for a particular test for a feature in a program; and indicate the physical test bed for test performing.

E.g. we have a BLSR_4F_2Node test suite, then we can make a plan for SN16K R9.0 BLSR 4F 2Node Regression test, targeted run it in two days.

•In automation scripts, a test plan includes: Test suite path/name The physical test bed, which is an instantiation of the logical test bed A test case file list, selected from the test suite

•Usually, a test plan is feature and release oriented. For different releases, one can have different test plan for one feature. And, a test plan should be defined in the test preparation stage.

15

Test Task•Test Task is a runtime concept, it defines when and how to execute a test plan. Once a test task is executed, a test report should be reproduced

•In automation scripts, a test task could be a command in CLI, an item in TMS, or a file, which includes:

The test plan path/name A schedule for the test plan running Runtime options, e.g. Halt or skip if any failure? Wait or fail if the test

bed is not available? Recycle or one-time running?

•Multiple test tasks can be created for a test plan, that means, a test plan can be executed multiple times in a release.

•Usually, a test task is created in test performing stage by the tester for one-time running, and will not be reused.

16

Test Report

•A test report is a result record of a test task.

•Test reports can also be organized by test plans or test suites for broadened view to analyze requirements satisfaction.

•A test report should have the following information Runtime information: e.g. who/where/how had run this task with

what options? Detailed test bed information: e.g. build no. for each node, serial

numbers of elements, software versions Test results for each case and each step.

•A test report can be valuable just with a specified test bed on which the test was running.

17

Outline

•Test Automation Principles & Terminologies Why Test Automation How to do Test Automation Terminologies about Test Automation (Sycamore)

•Implement the Terminologies in Automation Scripts General Conventions An Example - PET

18

Classic Reasons for Automation Failure10. No clear objectives for automation. 9. Minimal or No attention of non GUI automation. 8. Lack of methodology to success of the project. 7. Automation is easy Just record and playback. – 6. Aim for 100% automation. 5. Testing for automation scripts. 4. No clarity of real cost involved in automation. 3. Automation is part time job of testers. 2. Wrong selection of Test cases. 1. Not treating automation as software development.

General Conventions

20

A flexible framework•General framework allows users define format and language

•User defined languages for: Test bed Test cases Test report Test task Scheduler ….

•General format XML Perl, Python Simply BDD language, Spreadsheet

21

Plug-in technology

•A platform should support a general framework

•All support simple plug in

•

22

An example: PET

•Pet: Peal Engine for Test It is not just for Perl testing, but Implement a framework to support different test format It is independent to run as an application Or, to be used as a library for your bigger framework

•Key engine: Test Recorder

•Download from: https://github.com/steven004/Pet

An example: PET

24

What is Pet?•Pet – Perl Engine for Test

Perl: Pet is based on Perl. It can run on almost any platforms Engine: It will speed up your work, and provide lots of benefit Test: It is designed for test, not just for test execution, but test

management as well

IQstream

WebUI

SN9K/3K/16K

…

Support different products

Console

HTML

TMS

E-Mail

…

TEA XML Perl …

Support different format/language

Support different outputs

Test Plans/Suites

Pet Engine

Library

Key-Words

Objects

…

Test Cases

Test Beds

Test Recorder

25

Why you need Pet? •TestRecorder Module embedded in Pet

You will get test results via different channels Console output, HTML files for Web, E-mail notification, TMS database, and more if you request All the channels can be configured, you can select any of them as you like/need when running your scripts.

A bunch of test recorder functions provided

•Multi-format supported for scripts writing Pure-perl TEA format compliant Or mix-programming

You can use Perl in your TEA case file; or use TEA format in Perl scripts

Spreadsheet cases

• Scripts running control A mechanism provided to run scripts case by case, or step by step Runtime control provided, by “Ctrl-c” Failure handling, for collect information in your scripts. Error handling, If there is some syntax problem in your scripts, Pet will

let you know.

26

How to run Pet – the 1st step

• Pet looks complicated, but we make it simple

• Let’s start① Log into an automation Server using your unix account

• 172.21.7.200 (Shanghai) or 172.31.115.6 (Chicago)• Run Pet

-bash-3.2$ pet/usr/bin/pet requires arguments. Try ‘pet –h’ for help.-bash-3.2$ pet -h

Usage: Pet [-cnhpv] [testplanfile] [switches] [arguments] -u UserName indicate the who is running the test scripts. Pet will get the user name from OS if no this option -h print Usage hints -l OutputDir indicate the output directory of log. \tmp\pet\username\ will be used for the default output directory -m the mail address for test result reports -v print version, subversion <includes VERY IMPORTANT Pet functionality info> -d [:debugger] run program under debugger (future) -b BuildNumber indicate the build number of the system/software under test -n new a test plan file in current directory based on the existing test bed and test case files

-p parse the test plan and print the case list to the standard output -r ReleaseName indicate the release name/number of the system/software under test -o [tms/console/html/xml/txt] indicate the environment of the Pet. Local run by default -c Pet will just check test bed availability, instead running all cases in the test case list -t TaskID indicate the test task ID for TMS. This option is just required by TMS to run the Pet -f [KEEP_RUNNING/STOP_ON_STEP_FAIL/STOP_ON_CASE_FAIL/PROMPT_ON_FAIL/PROMPT_BY_STEP/PROMPT_BY_CASE/

TEST_QUIT] continue the program or stop it when there is a case fail

27

Pet running examples

•Run a Perl file $ pet example.pl –r 1.0

•Run a Perl-format test plan $ pet example_pl.xml –r 1.2 –b B2.8

•Run a Tea-format test plan $ pet example_tea.xml –r 1.2 –b B2.7 –m steven004@gmail.com

•More parameters $ pet example_tea.xml –r 1.2 –b B2.7 –u xli –m

steven004@gmail.com –t 53432 –o CONSOLE – l /tmp/xli

•Parse TestPlan $pet –p exampletp.xml

28

Outline•Pet Basic

•Quick Start Understand test plan, test suite, test bed and test suite Create your own test task

Testbed modificationTestcase files selection

Where you get the results?Email, HTML, TMS?

• I want to do some change

•Advanced: How can I use Pet to write scripts

•Dig into Pet

29

Terms – Test Plan/Bed/Suite/Task/Report•See details in

Previous slides

•Pet is to run a test plan in which Test Bed, Test Suite, Test Case list are specified

•A test plan example<?xml version="1.0" encoding="UTF-8" ?> <!-- Test Engineer: Guo, Hugh --><!-- Program: Pet Example --><!-- Description: Tea-type example test plan --><TestPlan

SuiteFile="../exampleSuite.pl" PhysicalTestBedFile="../PhysicalTestBeds/example_tb.ttb"><TestCases>

<File>../TestCases/example_tc.ttc</File></TestCases>

</TestPlan>

！ Use Relative paths in your test plan, instead of absolute ones！

30

Create your own test task

•Create/Modify TestBed file

•Create/Modify test plan file Indicate the test suite file Indicate the test bed file TestCase files selection Again, Use the relative paths, instead of absolute ones

•Demo …

31

Where is my test results?

•Important Environment Variables Get it by running ‘set’ in Linux-bash-3.2$ set…TESTMAILTO=steven004@gmail.com #results will be sent to this mail.TESTRECOPT='TESTSUMMARY|TESTDETAIL|TESTLOG|ERRORLOG|WARNINGLOG|INFOLOG|DEBUGLOG‘

# All logs will be recordedTESTRECPATH=/shared1/users/IQstream_Result/auto_pet/output/ #The place to save output resultsTESTRECTO='CONSOLE|HTML|EMAIL‘ #You can get results through Console, Email and HTML files.…

•Demo…

32

Outline•Pet Basic

•Quick Start

•I want to do some change Use default output options Set Pet Environment Variables

TESTRECOPT, TESTRECPATH, TESTRECTO, TESTMAILTO Set Pet Options in your scripts

•Advanced: How can I use Pet to write scripts

•Dig into Pet

33

All about TestRecorder

•TestRecorder Manual https://github.com/steven004/Pet/blob/master/Documents/

TestRecorder.htm •TestRecorder Training Slides

https://github.com/steven004/Pet/blob/master/Documents/Use%20PET%20step%20by%20step.ppt

•Default TestRecorder Options had been set in automation servers for all users

For TestRECTO, TESTRECPATH, TESTRECOPT

•You need to set TESTMAILTO

•You can also set your own Recorder Options in your profile

34

Use different options for a particular test

•Two ways to go 1) Specify your options as arguments when you run your test

pet myplan.xml –m notme@abc.com –r 1.2 –b 1 –l /tmp/xli/ This time, notme@abc.com will get the result, and all results

files will be save in /tmp/xli in the automation server

pet myplan.xml –u goldman –r 1.2 –b 3.8 –f STOP_ON_CASE_FAIL This time, pet will think the goldman is running the scripts, instead of

you, the one log into the server, and scripts will stop once there is a case fail.

2) Set in your scripts (not recommended) See more in TestRecorder Training Slides

35

Outline•Pet Basic

•Quick Start

• I want to do some change

•Advanced: How can I use Pet to write scripts Call Flow in Pet TestSuite file writing TestBed & TestCase file

Pure Perl programming TEA format programming Mix-mode programming

•Dig into Pet

36

Call Flow in Pet

•Typical call flowImport TestSuite(run testsuite file)

Import TestBed(run PhyTestBed file)

TestBed Setup

Test Start

Test End

TestBed Cleanup

Run CaseFiles #1 #2

… #n

Case, Step & TestLog related functions (defined in TestRecorder) can be used here, and all activities will be recorded by Pet.

37

TestSuite file•TestSuite file has to be a Perl file•It could be very simple•The simplest example: # A Test Suite Example

use IQsteam::Objects; # it means you will test a IQstream system

•The TestSuite.pl template: ## use the necessary packages here

use IQsteam::Objects; sub TestBedSetup {

#add your scripts here#

1;

}

sub TestBedClearup {

#add your scripts here#

1;

}

1;

38

TestSuite file is serving the test suite

•Anything related to the test suite should be defined in it

•It may contain The packages to be used in the test suite The attributes of the test suite

Test Bed base state definition (TestBed Setup) How to recover the test bed after test completes Any variables/data to be used in the test suite, but not related to a

particular physical test bed Any functions definition for test cases & Test bed initiation

E.g. the testsuite.pl defined for Lab Pcap running

39

TestBed & Test Case files•Test Bed design

Define variables/data related to particular test beds, i.e. Any parameters may change when you change your test bed to do the same test

•Test Case file design Test Case file is just to define test procedures Test Case files should not be modified when you run your test on

different test beds Case / Step / TestLog related functions defined in TestRecorder will be

used in TestCase Files

Test Cases defined in one TestCase file should be self-clean for better test plan making

40

TestBed & TestCase Files – Pure Perl•TestBed Example (from WebUI test suite)

our $BrowserType='*iexplore';our $BrowserURL='http://172.21.5.109';our $CNIPAddress='172.21.5.109';our $client_IP='172.21.160.154';our $PortNumber='4444';

All the definitions are about the particular test bed, and can be easily changed when you run it in another test bed.

•TestSuite ExampleCaseStart(“Just an example");{

InfoMsg(“All the following steps should pass!");Step(3*2==6, "3*2==6");Like(“Shanghai”, “Sh”, “Shanghai =~ Sh");Is(3*2, 6, "3*2 is 6?");

} CaseEnd();……CaseStart(“Other Cases….”); {…} CaseEnd();

More Functions, please see TestRecorder Manual