Post on 13-Apr-2018
transcript
Fitting Solutions
Test automatisering and Exploratory Test
Christian Nørlyng
Software Test Developer
3 of April 2014
Who am I?
Fitting Solutions
•Software Test Developer at Oticon
•ISTQB Certified Test Analyst (adv. level)
•ISTQB Certified Technical Test Analyst (adv. level)
•Certified Agile Tester
•Certified Scrum Master
10+ years of test experience from
• Small & Large Teams
• Agile & Waterfall Projects
• Medical – Directory – Telecom – Defense – Maritime
Have been mostly Test Automation
The Testing Challenge at Oticon Fitting Solutions
20.000+ different HI’s since 2011, which must be tested at least once pr release
Different Windows platforms. Even ”XP”
Tons of un thought of hardware combinations
Regression testing the old platforms and HI’s
Multiple programming devices
Infinite amount of ways through the application
Safety Critical
High Volume Market
At least 4 Releases pr year
What bugs are you looking for??
What is your test mission?
What kind of bugs are you looking for?
What concerns are you addressing?
Who is your audience?
Make automation serve your mission.
Expect your mission to change.
Possible Missions for Test Automation
Find important bugs fast
Measure and document product quality
Verify key features
Keep up with development
Assess software stability, concurrency,
scalability…
Provide service
Efficiency
Reduce testing costs
Provide proof
Reduce time spent in the testing phase
Automate regression tests
Improve test coverage
Make testers look good
Reduce impact on the bottom line
Reduce time spent in the testing phase
Automate regression tests
Improve test coverage
Make testers look good
Reduce impact on the bottom line
Tighten build cycles
Enable “refactoring” and other risky
practices
Prevent destabilization
Make developers look good
Play to computer and human strengths
Increase management confidence in the
product
Automation and bugs
Test Automation are only looking for defects found during ”regression testing”
• All automated tests, except model based test are ”second time arround” a feature
Feature Picking
”Wants”
”Seems Quick”
”Critical”
”Features”
Starting with the
inter section!
Provide the best value to the project!!!
Feature Picking
”Planned”
”Unsertain”/ ”Sort of Planned”
Pick the stable features which is not part of unsertain
”Features”
”Stable Early”
Choosing Best Automation Candidates
Good candidates
• Short or simple transactions
• Many data combinations
• Expected results are stable or easy to generate at runtime
• Tests that are executed regularly
• Tasks that are difficult to do manually
• Highest priority features
Poor candidates
• Long or complex transactions
• One-offs
• Unstable or difficult to predict results
• Tests that cross multiple applications
Designing Good Test Cases
”Structural Test Patterns”
Test cases should have a single objective
Test cases should result in one of two dispositions: pass or fail
Test cases are independent
No test case relies on the successful completion of another test
Test cases start and stop at a known ‘base’ state
Approaches and ROI
Automation MUST have ROI
ROI = (Savings + Total Investment)/ Total Investment
Quick and Dirty
• Implement, Run and Discharge
• Intinial High ROI. Decreases over time with change
Build up test method framework
• Design, Impliment, Improve, Maintain and Execute
• Intinial medium ROI. Increases over time with repeated runs
Model Based Testing
• Design, Model, Impliment, Improve, Maintain and Execute
• Initial low ROI. Increases over time, if managed correctly
13
Framework Design
Traditional
Sequential
TestCases
A pseudo test case broken down into ActionWords, each KeyWord is unique to the SUT (Software Under Test)
KW 6
KW 5
KW 4
KW 3
KW 2
KW 1
Example: Framework Design
KW = KeyWord
KW 7
KW 6
KW 3
KW 2.1
KW 1
KW 7
KW 5.1
KW 4
KW 3
KW 2.2
KW 1
KW 7
KW 5.2
KW 4
KW 3
KW 2.3
KW 1 1
2.1 2.2 2.3
3
6 4
5.1
7 Saving above 50% of work for just this example.
5.2
KW = KeyWord An Advantage
Module Based Test Design
The End for Test Automation
What I did not talk about
Selecting the right Test Tool
As of 2 days ago the 3 best tools are:
• TestComplete - frequently upgraded, the 3 platforms, Not expensive
• HP Functional Tester – not so frequently upgraded, the 3 platforms, expensive
• Selenium - frequently upgraded, web only, free. tons of add-ons.
Continueous Integration
Test Architecture
Daling With change
Automation Approach: ”Quick And Dirty”
Practical Implementaion: ”Capture Replay”, ”Objects Mapping”
End Times
What ET is…
Simultaneous learning, test
design and test execution
Unscripted
Targeted
Repeatable
Relevant
A rapid cycle of
- Design a test
- Execute it
- Observe results
Until we've characterized the capabilities
and limitations of the system with respect
to a charter
Charter Pile
Do a Charter in Session
Report generarion
New Ideas for Charters
•Strategy •Modeling •Decision Making •Configuring •Operation •Observation •Evaluation
The Test Session
Uninterrupted
e-mail, meetings, telephone calls etc.
Reviewable
A report should be produced
Chartered
Mission associated with this session;
• What are we testing?
• What are we looking for?
Timeboxed
ET Execution 19
From Rapid Software Testing, copyright © 1996-2002 James Bach
20
Charter: A clear mission for the session
A charter may suggest what should be tested, how it should be tested, and what problems to look for.
A charter is not meant to be a detailed plan.
General charters may be necessary at first:
“Analyze the Insert Picture function”
Specific charters provide better focus, but take more effort to design:
“Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”
From Rapid Software Testing, copyright © 1996-2002 James Bach
Doing Exploratory Testing
Keep your mission clearly in mind.
Keep notes that help you report what you did, why you did it, and support your assessment of product quality.
Keep track of questions and issues raised in your exploration.
To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.
21
From Rapid Software Testing, copyright © 1996-2002 James Bach
When are you DONE ?
Testing does not reveal NEW useful information.
Experienced tester feels comfortable.
Critical risk features have had time to go through exhaustive
testing.
AND: Finalising Notes, learnings and ideas
Then: Debrief with the “Manager”
From Rapid Software Testing, copyright © 1996-2002 James Bach
Session Based Test Management
Brief enough for accurate reporting.
Brief enough to allow flexible scheduling.
Brief enough to allow course correction.
Long enough to get solid testing done.
Long enough for efficient debriefings.
Beware of overly precise timing.
Time Box/Session 24
Short: 60 minutes (+-15)
Normal: 90 minutes (+-15)
Long: 120 minutes (+-15)
From Rapid Software Testing, copyright © 1996-2002 James Bach
Measurement begins with observation
The manager reviews session sheet to assure that he understands it and that it follows the protocol.
The tester answers any questions.
Session metrics are checked.
Charter may be adjusted.
Session may be extended.
New sessions may be chartered.
Coaching happens.
Debriefing 25
From Rapid Software Testing, copyright © 1996-2002 James Bach
ET Documentation
Planning:
Charter – overview of what to test (plan)
Migh be a flip chart on the wall
Mission – What are we looking for?
Execution
Notes – what happened during testing?
What did I do? Why did I do it?
Used to assess product quality after test.
Data files – input data used for testing
Bug reports – enough details to recreate the test / bug
Track questions and Issues
26
From Rapid Software Testing, copyright © 1996-2002 James Bach
Why does it Work?
Testers use results of previous results to guide their future
testing on the fly
They do not have to complete a series of scripted tests
before focusing in or moving on to explore a more target rich
environment
This accelerates bug detection when used intelligently
27
From Rapid Software Testing, copyright © 1996-2002 James Bach
The Theory
Discovery helps you choose relevant things to do among the
infinite number of possibilities
The variations possible are “learned” as other tests are run
You run tests you didn’t think of initially – can’t anticipate
everything!
“Sampling” vs complete coverage of conditions where complete
coverage is not practical
Tests drawn from using a testing technique the team normally
doesn’t apply
Tests are not written ahead of time – fast, flexible
You log as you go – requires practice, new habit
Does knowing more about the intended use help you do
exploratory testing better? More efficiently?
28
From Rapid Software Testing, copyright © 1996-2002 James Bach