The future of testing

James A. WhittakerSoftware ArchitectVisual Studio Team TestMicrosoft

http://blogs.msdn.com/james_whittakerjw@microsoft.com

“Any sufficiently advanced technology is indistinguishable from

magic.”

- Arthur C. Clarke (1917-2008)

In 1953 Francis Crick and James Watson described the structure of

deoxyribonucleic acid (DNA)

From 1990-2003 the Humane Genome Project mapped the entire genetic blueprint of a human being

In 1969 Peter van de Kamp

explained wobbles in

the motion of Barnard’s star

by the presence of a

planet 1.6 times the mass of Jupiter

By 2006 Astronomers confirmed the existence of over 200 exoplanets

As of January 2009 that number has climbed to 334

It wasn’t until 2003 that the first exoplanet was confirmed to orbit the

star Gamma Cephei

That’s one new planet every 2.3 days … how long until we find one that

might support life?

We’ve always considered the autistic to lack the ability to communicate

Until, through the magic of software, a courageous autistic girl corrected

ushttp://www.youtube.com/watch?v=JnylM1hI2jc&eurl=http://well.blogs.nytimes.com/2008/02/28/

the-language-of-autism/?apage=2

We’re going to need the magic of software

Global climate change

We’re going to need the magic of software

Alternative energy

We’re going to need the magic of software

Managing global finance

Software is the one tool we need to save ourselves

But what’s going to save us from software?

• “A second V-22 crash in December 2000 killed four Marines. The accident was pegged to a ruptured hydraulics line and a software failure.” - http://community.seattletimes.nwsource.com/archive/?date=20030226&slug=osprey26

• “Fans of the rock band Pearl Jam got a taste of cyber-gridlock Saturday, when a software failure disabled phone lines set up for ticket requests for shows in Seattle, Toledo, Ohio, and other cities.” - http://community.seattletimes.nwsource.com/archive/?date=19960814&slug=2344161

• “Patriot Missile's Tragic Failure Might Have Been Averted -- Computer Glitch Probably Allowed Scud To Avoid Intercept, Army Says” - http://community.seattletimes.nwsource.com/archive/?date=19910815&slug=1300071

• “Software Failure Halts Big Board Trading for Over an Hour” - http://query.nytimes.com/gst/fullpage.html?res=9C05E0D7173EF93AA35755C0A9679C8B63&scp=1&sq=%22software%20failure%22&st=cse

• “This is what happened in 2003, when a software failure near Toledo, Ohio, combined with sagging power lines and a simple error by a control room worker to cause a blackout in eight Northeastern and Midwestern states and Ontario.” - http://www.nytimes.com/2007/07/08/opinion/nyregionopinions/08CIperrow.html?_r=1&scp=14&sq=%22software%20failure%22&st=cse&oref=slogin

• “Nike last year had to reduce prices and fly some products to customers after a software failure resulted in excess inventory and missed shipments.” - http://query.nytimes.com/gst/fullpage.html?res=9407E2DB133EF93BA15755C0A9649C8B63&scp=24&sq=%22software%20failure%22&st=cse

fee = 8.95fee = 0.00

fee = -100.00

If quality is this low with today’s software, what about tomorrow’s?

The future I see

http://blogs.msdn.com/james_whittaker

Information

• For testers, it’s everywhere– On your desk– On your drive– On your network– In your head– In other people’s heads– …

• We need information to be available, actionable and alive

Information

The future I see

http://blogs.msdn.com/james_whittaker

Virtualization

• Hard problems:– Ain’t got no repro– The Vista lesson

• Solution:– Control the environment– Easier said than done … until now

VM’s everywhere

• Virtualization for user machines– Think Watson … only better– Fool proof debugging– Libraries of virtual test machines

• Virtualization for tester machines– Virtualized test libraries, just add apps– Environment-carrying tests

A New Test Market

• Virtual test machines would have value– Buy them, sell them, rent them

• Testers no longer test, they design– Determining what needs to be tested supersedes how

to test it – Why select among tests when you can have them all?

• The result– Apps released after centuries of accumulated testing

against hundreds of thousands of possible environments

– Most apps don’t require dedicated QA staff

The future I see

http://blogs.msdn.com/james_whittaker

Visualization

Visualization

What does software look like?

• Well, there’s:– Input– Output– Data flow– Control flow– Modules– Dependencies– Environment variables– Files– Interfaces– Bugs

Conclusion

• I’ve painted my vision of the future• A future where– late cycle heroics are cause for shame– testing is thorough, predictable,

repeatable– software just works– quality is an expectation– innovation is an epidemic

Parting thoughts

• 20 years from now–Will the quality of software be taken for

granted? Will users be genuinely surprised when it fails?

–Will researchers look back in wonder that there was ever even a need for dedicated bug finders?

What needs to be done now?

James A. Whittakerhttp://blogs.msdn.com/

james_whittakerjw@microsoft.com