Date post: | 23-Dec-2015 |
Category: |
Documents |
Upload: | coleen-french |
View: | 213 times |
Download: | 0 times |
FoxPro User Group World TourFoxPro User Group World Tour
Testing Computer SoftwareTesting Computer Software
Mike StewartMicrosoft
Mike StewartMicrosoft
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Who Am I and Why Should You Care?Who Am I and Why Should You Care?
Software Test Lead for the Visual FoxPro team Numerous Knowledge Base articles and white
papers for Microsoft Consultant developing custom applications
using VFP, C++, Delphi and VB Unix sysadmin and systems programmer
using C++
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
What this Session is NotWhat this Session is Not
A lot of code examples – no example can fit every situation
Testing the “Microsoft way” – this is all based on software QA industry methodologies
One size fits all – everyone’s testing needs are different
For people with aversions to blue Power Point slides
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Why Test?Why Test?
Customer satisfaction – testers are customer advocates
Cost – the later a bug is found, the more it costs to fix
Quality – the earlier a bug is found, the more likely it is to be fixed
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
What is the goal of testing?What is the goal of testing?
Find bugs – more specifically, get bugs fixed Verify that it works Prevent errors before they get into the product Provide information on risk – how solid is the
product?
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Important Things to RememberImportant Things to Remember
“A test that reveals a problem is a success. A test that did not reveal a problem was a waste of time.” - Kaner
You will not find all of the bugs Myers’ 20 line program: 100 trillion paths Program takes two inputs of four-digit
number: almost 400 million possibilities Test something 10,000 iterations: what if it
fails at 10K + 1?
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Exhaustive TestingExhaustive Testing
“The only exhaustive testingthere is, is so much testing that the tester is exhausted!” — Hetzel
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
“White Box” vs. “Black Box” Testing“White Box” vs. “Black Box” Testing
White box testing treats the software as a box that you can see into.
You are looking into the box to see how it works
Assumes you have source code Black box testing treats the software as a box you
cannot see into. Put in data (input) and see what comes out of
the box (output). No source code needed.
Black box testing on finds about 55% of the bugs
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Why White Box Testing isn’t CoveredWhy White Box Testing isn’t Covered
Doesn’t test the software in the manner in which it will be used
It doesn’t do spec testing – it has no way of knowing if it is functioning as designed or not
It does not find missing code – some estimate this accounts for 30% of bugs
Not enough time! It’s a discipline of its own, and we’d be here all day.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Steps in testingSteps in testing
Test planning Write test plans Spec and code reviews
Build verification – is it even stable enough to test?
Function and system testing – run the full suite of test cases
Media verification No viruses Bits on the media are the release bits The product can be installed from the media
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Essential Testing IssuesEssential Testing Issues
Establish a bug tracking system ATS Web – free at http://msdn.microsoft.com
/vstudio No sticky notes
Establish processes Use cases Test plans Test cases Bug entry and resolution
Automate, automate, automate!
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Bug TrackingBug Tracking
Accountability Assures that the bug does not fall through the
cracks When closed, we are assured that it is fixed.
Metrics How great is the severity of bugs found? How fast are we closing bugs? Are we ready to ship?
Liability CYA – cover yourself against legal liability (I
am not a lawyer, however)
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Test PlansTest Plans
A test plan is a general “plan of attack”, not a detailed list of test cases.
List areas to test, and how they will be tested. Notepad File/Open example
“Notepad’s File/Open functionality will be tested for various file types, file size limits, file name limits and regional encoding. Various media need to be considered, such as removable media and networks.”
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Test CasesTest Cases
Each task can be compartmentalized into separate automation programs, or conglomerated into one big program.
Keep test cases atomized: test one particular feature at a time.
Create a matrix, and build test cases from that Could be something as simple as an Excel
spreadsheet
Example: the Notepad File/Open dialog
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Test Cases for File/Open/FilenameTest Cases for File/Open/Filename
Long file names – boundary testing Single letter file names – boundary testing Long extensions – boundary testing Edit box text length – boundary testing Invalid characters in file names – functional Valid file names – functional Wild card characters – functional Edit keys (Ctrl-C, etc.) – functional or spec Paste in binary from clipboard - pathological MRU dropdown – functional Path completion – functional Large fonts – functional Gain and lose focus – functional High ASCII chars - functional
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Bug Entry and ResolutionBug Entry and Resolution
Minimum to track: Severity Priority The tester who found it The dev who fixed it
Enter good bugs: The easier to reproduce, the more likely it is to
get fixed Check on a second machine – could be a
configuration/OS issue If it only happens on one OS, say so
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Bug Entry and ResolutionBug Entry and Resolution
If you’re org is tracking metrics, close resolved bugs as quickly as possible
Set severity appropriately, and leave the priority to the project manager(s)
Your org will have to decide how to set severity and priority – everyone has different needs
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Smoke testing – basic functionality Exploratory testing
Testing while learning the product Misses large areas of the product
Boundary conditions Cannot test every value, so test for the most
important values Hex boundaries and the boundary of field
length Boundary -1, boundary, and boundary +1 Also test largest value allowed, if it is not a
hex value
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Stress What happens when a large number of users
hit it? Low memory, low disk space, large amounts
of data, repetitive Bugs can often not be reproduced Web Application Stress Tool is great for web
apps.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Functional Does it do what the spec says it should do? How does it handle error conditions?
Performance A discipline unto itself, worthy of an entire
session Regression testing
Make sure nothing got broken Does not find a lot of bugs
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Code Coverage – for VFP, use the Coverage Profiler
Code coverage testing tells you what code has not been hit, and therefore needs to have test cases that hit this code.
You are unlikely to get 100% coverage The VFP Coverage Profiler can also reveal
bottlenecks.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Race conditions Two pieces of code are colliding Especially important in multi-threaded and
multi-user apps Error conditions - does it fail gracefully? UI
Tab order is easy to miss if you are a mouse-addicted developer
When testing as a dev, do not use the mouse, because you know the mouse will work.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Testing MethodologiesTesting Methodologies
Ad Hoc testing – “guerrilla testing” Great for checking basic functionality Can give broader testing than more formal
means Hard to verify coverage
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
AutomationAutomation
The Active Accessibility test harness included with VFP 7.0
Other automation products: Rational – Visual Test Mercury Interactive – Testing suite with
various products for web and thick client applications
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Automation TipsAutomation Tips
The test should be self-documenting Do not use “pass” or “fail” as the test result Test should say what it expected and what it
received, and whether it passed or failed. Write “drivers” or hooks into your software if
needed – for example, VFP’s Active Accessibility or Project Manager hooks
Tests should be self-sustaining – create your own data, and clean up. Expect nothing.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
When Do You Ship?When Do You Ship?
You will ship with bugs if your testing is adequate.
Define show-stoppers Will customers find the bug? Does it crash your product? Will only pathologic testers find it?
Define risk Fixing a bug has a 60% risk of introducing new
bugs When the show-stoppers are fixed, ship
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Selling Testing to ClientSelling Testing to Client
Bulk contracts – sell the job, not the hours to develop it, and the hours to test it.
Once the software is in the customer’s hands, bugs are much more expensive to fix
How much does it cost to have 30 users find a bug vs. a qualified, dedicated test effort at $x/hour, that assures quality?
Other ideas?
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
“Things They Should Teach in the Testing Stuff Class, but Don’t”
“Things They Should Teach in the Testing Stuff Class, but Don’t”
Run a high-color scheme like Marine High-Color
Use large fonts (Display/Settings/Advanced) Throw away your mouse
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
More informationMore information
Testing Computer Software – Kaner The Art of Software Testing - Myer Software Quality Testing Engineering
magazine: http://www.stickyminds.com STAR Conference – the DevCon for
professional software testers (www.stqe.com)
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
ADVISOR DEVCON Web Update PageADVISOR DEVCON Web Update Page
http://www.Advisor.com/CMF0109update
This session WILL have updates.
FoxPro User Group World Tour 2002FoxPro User Group World Tour 2002
Thank you!
Please remember to fill out your evaluation.