Date post: | 10-May-2015 |
Category: |
Technology |
Upload: | eurostar-conference |
View: | 886 times |
Download: | 0 times |
www.eurostarconferences.com
Today’s webinar is presented by Hans Buwalda and he will discuss BIG Testing.
Welcome to the EuroSTAR July Webinar Series
This webinar is due to start at 4pm. Make sure you stick around at the end for the Q&A session and continue the conversation with the speaker on Twitter after the show!
@esconfs#esconfs
www.eurostarconferences.com
BIG Testing
Hans Buwalda, CTO, LogiGear (USA)An internationally recognized expert in testing, Hans Buwalda is the pioneer of keyword-driven test automation, an approach now widely used throughout the
testing industry. Working in The Netherlands for CMG (now CFI) he helped develop the successful TestFrameTMframework. He now is CTO of LogiGear Corporation in California, responsible for the keyword-driven Action Based
TestingTM method, and the tools that support it: TestArchitectTM and TestArchitect for Visual Studio®. Hans speaks frequently at international conferences and is
coauthor of Integrated Test Design and Automation: Using the TestFrame Method.
@esconfs#esconfs
Q&A
www.eurostarconferences.com
If you have a question, the control panel on the right has a Questions box. All questions will be answered at the end of the presentation.
The GoToWebinar attendee interface is made up of two parts. The Viewer Window shows the presenter’s screen. The Control Panel is where attendees can interact and ask questions.
The webinar will be recorded and you will be notified via email when it is available.
@esconfs#esconfs
www.eurostarconferences.com
How can I get the slides?
Slides and all materials will be posted on the EuroSTAR website as well as emailed to you.
@esconfs#esconfs
www.eurostarconferences.com
Join the conversation on Twitter
#esconfs@esconfs
@hansbuwalda
Continue the conversation with the speaker on Twitter after the show!
@esconfs#esconfs
BIG Testing
Mr. Playback
Hans BuwaldaLogiGear
Tutorial excerpts
Keywords, Test Design, Automation,
Virtualization, Globalization
© 2013 LogiGear Corporation. All Rights Reserved
About LogiGear
Software testing company, around since 1994
Testing and test automation expertise, services and tooling- consultancy, training- test development and automation services- "test integrated" development services
Aims to be thought leader, in particular for large and complex test projects
Products:- TestArchitect™, TestArchitect for Visual Studio™- integrating test development with test management and automation- based on modularized keyword-driven testing
www.logigear.comwww.testarchitect.com
© 2013 LogiGear Corporation. All Rights Reserved
What is "BIG"
Big efforts in development, automation, execution and/or follow up
It takes a long time and/or large capacity to run tests (lot of tests, lot of versions, lot of configurations, ...)
Scalability, short term and long term
Complexity, functional, technical
Number and diversity of players and stakeholders- pigs, chicken, elephants, ankle biters, ...
Various definitions of "big" possible... and relevant...- "10 machines" or "10 acres"- "1000 tests" or "1000 weeks of testing"
Big today means: big for you- "non trivial", you need to think about it
"Windows 8 has undergone more than 1,240,000,000 hours of testing"
Steven Sinofsky, Microsoft, 2012
© 2013 LogiGear Corporation. All Rights Reserved
Keywords, essential for scalability
Distinguish tasks for test development and for automation
The test developer creates tests using "actions". Each action consists of a keyword ("action word") and arguments
Automation focuses on automating the actions, each action is automated only once
however…. actions by themselves are just a format, not a method….
© 2013 LogiGear Corporation. All Rights Reserved
Action Based Testing
Test Development Plan
Test Cases
TestObjectives
Test Module 1
Test Cases
TestObjectives
Test Module 2
Test Cases
TestObjectives
Test Module N
Actions
. . .
ACTION AUTOMATION
Break down
Automate
© 2013 LogiGear Corporation. All Rights Reserved
Test Design
Effective test breakdown (into test modules)- make sure test modules (test clusters) have a clear focus- keep different kinds and levels of tests separate
Right level of actions- as “high level” if possible, hiding as many details as much as possible- ...but not if the details are relevant for the test
It is my believe that successful automation is not a technical challenge. It is most of all a test design challenge.
© 2013 LogiGear Corporation. All Rights Reserved
What's the trick...
© 2013 LogiGear Corporation. All Rights Reserved
What's the trick...
Have or acquire facilities to store and organize you content
Select your stuff
Decide where to put what- assign and label the shelves
Put it there
If the organization is not sufficient anymore, add to it or change it
© 2013 LogiGear Corporation. All Rights Reserved
Example of a business test module
Test a business functionality, doesn't show details that are not relevant to that Navigation details are avoided, unless they're meant to be tested
TEST MODULE Car Rental Payments
user
start system john
TEST CASE TC 01 Rent some cars
first name last name car
rent car John Doe Ford Escaperent car John Doe Chevvy Volt
last name amount
check payment Doe 140.4
FINAL
close application
© 2013 LogiGear Corporation. All Rights Reserved
Example of an interaction test module
In this "low level" test details are not hidden, since they are the target of the test The right level of abstraction depends on the scope of the test, and is an outcome of your test
design process
TEST MODULE Screen Flow
user
start system john
TEST CASE TC 01 "New Order" button
first name control
click main new order
window
check window exists new order
FINAL
close application
© 2013 LogiGear Corporation. All Rights Reserved
Eye on the ball, Scope
Always know the scope of the test module
The scope should be unambiguous
The scope determines many things:- what the test objectives are- which test cases to expect- what level of actions to use- what the checks are about and which events should generate a warning or
error (if a “lower” functionality is wrong)
© 2013 LogiGear Corporation. All Rights Reserved
What I have seen not work
"Over-Checking": having checks that don't fit the scope of the test
Forcing data driven: making all tests data driven (variables, data files) without clear reason
Combinatorial explosions: test all ... for all ... in all ...
All actions high level (or all actions low level)
Many tests for forms and dialogs, little tests for business processes
Abundance of irrelevant comments, and lack of relevant comments
© 2013 LogiGear Corporation. All Rights Reserved
Choosing actions
Scope of the test determines the actions
As high level as appropriate, but not too high
As little arguments as possible- use default values for arguments not relevant to the scope of the test
Clear names (usually verb + noun usually works well)- to standardize action names: standardize both the verbs and the nouns, so "check customer" versus "verify
client" (or vice versa)- tests are not C++ code: avoid "technical habits", like mixed case and underlines
Manage the Actions
Document the Actions
Actions should be by-product of the test design
© 2013 LogiGear Corporation. All Rights Reserved
Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test
key
key navigate F7key navigate 3
page tab
locate page tab Scan Criteria
w indow
wait for controls loaded search
text
check breadcrumb general functions > search
w indow control value
select search scan direction Backward
w indow control value
enter value search business date match # bus date
source control
click search go
w indow
wait for controls loaded search results
w indow control variable
get search results sequence number >> seq num
© 2013 LogiGear Corporation. All Rights Reserved
variable
get sequence number >> seq num
Example of using actionsIn this real world example the first "sequence number" for teller transactions for a given day is retrieved, using a search function• the "#" means an expression, in this case a variable• the ">>" assign to a variable for use later on in the test
© 2013 LogiGear Corporation. All Rights Reserved
Low-level, high-level, mid-level actions
Low-level: detailed interaction with the UI (or API)- generic, do not show any functional or business logic- examples: "click", "expand tree node", "select menu"
High-level: represent a business function specific to the scope of the test- hide the interaction- examples: "enter customer", "rent car", "check balance"
Mid-level: auxiliary actions that represent common sequences of low level actions- usually to wrap a form or dialog- greatly enhance maintainability- example: "enter address fields"
enter customer
enter address fields
enter select set . . .. . .
© 2013 LogiGear Corporation. All Rights Reserved
Test Development and Automation in sprints
Test ModuleDefinition(optional)
Test Module Development
Interface Definition
Action Automation
Test Execution
Sprint ProductsProduct Backlog
Test re-use
Automation re-use
product owner
teamprod owner &
team
User storiesDocumentation
Domain understanding
Acceptance CriteriaPO Questions
SituationsRelations
Agile life cycle
Test development
Main Level Test Modules
Interaction Test Modules
Cross over Test Modules
© 2013 LogiGear Corporation. All Rights Reserved
Some Tips to Get Stable Automation
Make the system under test automation-friendly
Use "active" timing- always wait for something, never just wait a fixed amount of time
("passive timing")
Test your automation
Use automation to identify differences between versions of the system under test
Keep an eye on the test design
© 2013 LogiGear Corporation. All Rights Reserved
Look for properties a human user can't see, but a test tool can This approach is a must-do for speedier and more stable automation
- interface mapping is often bottleneck, and source of maintenance problems- with predefined identifying property values an interface map can be created without "spy" tools- not sensitive to changes in the system under test- not sensitive to languages and localizations
Examples:- "id" attribute for HTML elements- "name" field for Java controls- "AccessibleName" property in .Net controls (see below)
Automation-friendly design: hidden properties
© 2013 LogiGear Corporation. All Rights Reserved
Environments, configurations
Many factors can influence details of automation- language, localization- hardware- version of the system under test- system components, like OS or browser
Test design can reflect these- certain test modules are more general- others are specific, for example for a language
But for tests that do not care about the differences, the automation just needs to "deal" with them- shield them from the tests
© 2013 LogiGear Corporation. All Rights Reserved
Capture variations of the system under test in the actions and interface definitions, rather than in the tests (unless relevant there).Can be a feature in a test playback tool, or something you do with a global variable or setting.
Variation Variation Variation
"Variations"
"Master Switch"
Actions, Interface Definitions
. . .
© 2013 LogiGear Corporation. All Rights Reserved
Possible set up of variations
linked variation
keyworded variation
Specify for example in a dialog when you start an execution:
© 2013 LogiGear Corporation. All Rights Reserved
Virtualization
Virtual machines rather than physical machines- allow "guest" systems to operate on a "host" system- host can be Windows, Linux, etc, but also a specialized "hypervisor"- the hypervisor can be "hosted" or "bare metal"
Main providers:- VMWare: ESX and ESXi- Microsoft: Hyper-V (readily available in Windows 8)- Oracle/Sun: Virtual Box- Citrix: Xen (open source)
Hardware support gets common now- processor, chipset, i/o- for example "slat" (second level address translation, helps map addresses to memory)
For most testing purposes you need virtual clients, not virtual servers- most offerings in the market currently target virtual servers, particularly data centers
Virtual clients will become more mainstream with the coming of VM's as part of regular operating systems- Windows 8: Hyper-V- Linux: KVM
© 2013 LogiGear Corporation. All Rights Reserved
Virtualization, a testers dream...
In particular for functional testing
Much easier to define and create needed configurations- you basically just need storage- managing this is your next challenge
One stored configuration can be re-used over and over again
The VM can always start "fresh", in particular with - fresh base data (either server or client)- specified state, for example to repeat a particular problematic automation situation
Can take "snap shots" of situations, for analysis of problems
Can use automation itself to select and start/stop suitable VM's- for example using actions for this- or letting an overnight or continuous build take care of this
© 2013 LogiGear Corporation. All Rights Reserved
Virtualization, bad dream?
Performance, response times, capacities
Virtual machine latency can add timing problems- see next slide- can be derailing in big test runs
Management of images- images can be (very) large, and difficult to store and move around
• there can be many, with numbers growing combinatorial style• configuration in the VM can have an impact, like fixed/growing virtual disks
- distinguish between managed configurations and sandboxes- define ownership, organize it- IT may be the one giving out (running) VM's, restricting your flexibility
Managing running tests in virtual machines can take additional efforts on top of managing the VM's themselves- with the luxury of having VM's the number of executing machines can increase rapidly- one approach: let longer running tests report their progress to a central monitoring service (various
tools have features for this)
© 2013 LogiGear Corporation. All Rights Reserved
Virtual machines, capacity
Key to pricing is number of VM's that can run in parallel on a physical machine
An automated test execution will typically keep a VM more busy than human use
Factors in determining VM/PM ratio:- memory, for guest OS, AUT, test tooling- storage devices (physical devices, not disk images)- processors, processor cores- specific hardware support (becoming more common)
• processor, chipset, I/O
We started regression with 140 VMs.Very slow performance of
Citrix VM clients.
© 2013 LogiGear Corporation. All Rights Reserved
Globalization
Three Challenges:- other countries, other cultures- geographic distances- time differences
Seven "Patterns":- "Solution"- "Push Back"- "Time Pressure"- "Surprises"- "Ownership"- "Mythical Man Month"- "Cooperation"
other country
surprise
© 2013 LogiGear Corporation. All Rights Reserved
Three Challenges
Other country- tricky, but can be managed- study and prepare (culture, governance, etc)- have a clear company culture to complement regional
(and professional) culture
Distance- cumbersome- have good (great) communication features, at all levels
Time difference- this is the biggest problem, comes back every day- as much as possible have local management and responsibility
company culture
© 2013 LogiGear Corporation. All Rights Reserved
Seven patterns
"Mythical Man Month"- more (inexpensive) resources does not mean faster timelines
"Solution"- team "solves" problems rather than asking
"Push Back"- on-shore side dictates work and practices in detail- tell teams to push back if necessary
"Time Pressure"- deadlines are quite often imposed without need
"Surprises"- bad news is communicated too late
"Ownership"- establish clear ownership: shared responsibility is no responsibility
"Cooperation"- distance creates confusion and conflict (human nature)- tip: work with briefs and agendas for meetings
© 2013 LogiGear Corporation. All Rights Reserved
Industrial Organization
Large scale testing can move from a "design" to a "production" focus- mostly applies to test execution, but also seen for test development- this not black and white, both paradigms can occur in the same projects
A production organization is different a development organization- different professional culture- emphasis more on delivery and scale, "thinking big"- discipline rather than creativity, "get stuff done"- activities are like planning, control, logistics, information
Responsibilities- Keeping the tests running- Allocating resources- Respond to hick-ups- Analyze and address automation issues
© 2013 LogiGear Corporation. All Rights Reserved
Summary
Not all "big project" challenges are the same
Think before you do. Best results come from combining effective concepts, tricks and tools
Consider tests and automation as products that need planning and design
Virtualization can be a solution, but also a challenge
Globalization can help scalability, but needs knowledge and attention
Big testing projects typically have a "production" focus, in addition to test development and automation
www.eurostarconferences.com
Questions?
@esconfs#esconfs
EuroSTAR Webinar Archive
Access almost 70 on-demand software testing webinars by the world's leading testing experts. Topics include….
www.eurostarconferences.com
Agile, Automation, Test Management, Process Improvements, Risk-based Testing, Performance Testing, Test Documentation, Cloud Testing, Session-based Testing, Weekend Testers
Visit the archive!
http://www.eurostarconferences.com/community/member/webinar-archive
@esconfs#esconfs
The EuroSTAR Community.Are you a member?
Did you know we have an online resource area with almost …
www.eurostarconferences.com
600 software testing presentations, 70 topical webinars, over 40 videos and podcasts and 25 expert eBooks
Do you know that membership is completely free and you can access these premium testing resources anytime?
Join today! http://www.eurostarconferences.com/
community/member
@esconfs#esconfs
www.eurostarconferences.com
Are you coming to Gothenburg?
Join us in the Swedish Exhibition & Congress Centre, Gothenburg,
Sweden from the 04-07 November for the 21st annual EuroSTAR
Conference.
Visit www.eurostarconferences.com for more information on what you
can expect to see at Europe’s largest software testing event!
@esconfs#esconfs
Book before Sept
Have you seen the latest competition?
You and 3 of your team could be joining us at the 2013 EuroSTAR Conference!Entering is easy! Just send us a short video (2 minutes approx.) demonstrating how your team works together to meet
goals or solve problems.
Deadline for entries is Wednesday, August 14th! More information here: http://www.eurostarconferences.com/content/teamstar-2013@esconfs
#esconfs
Don’t miss these great savings, book your ticket today!
Plus Make even bigger savings when you register for the Group discount where every 5th attendee goes free!
For more information on the Conference Programme and pricing options go to www.eurostarconferences.com@esconfs
#esconfs