A Publication
VOLUME 5 • ISSUE 9 • SEPTEMBER 2008 • $8.95 • www.stpmag.com
BESTPRACTICES:
Post-Deployment
Tuning
Tools For.NET YouDidn’t KnowYou Had QA and The
Role Played
By Artifacts
Earn Your Chops Testing Microsoft Apps page 14
Build a Framework
For Sturdy, Repetitive
Data-App Testing
FALLFALL
COME AND GET CRAZY WITH US!HEAR WHAT ATTENDEES HAVE TO SAY ABOUT STPCON! CHECK OUT A VIDEO FROM THE LAST EVENT AT WWW.STPCON.COM
COME AND GET CRAZY WITH US!
IS IT CRAZY TO WANT TO MEETTHE MAKERS OF THE LATESTSOFTWARE TESTING TOOLS?WE DON’T THINK SO, which is why we’ve gatheredthe industry’s TOP COMPANIES in our Exhibit Hall!
Learn about their new products and features, test themout, and talk to the experts who built them.
Exhibit Hall Hours:Thursday, September 25 / 3:00 pm – 7:00 pmFriday, September 26 / 9:30 am – 1:15 pm
September 24-26, 2008Marriott Copley Place, Boston, MA
www.stpcon.com
REGISTER
ONLINE FOR
FREE ADMISSION!
WWW.STPCON.COM
SEPTEMBER 2008 www.stpmag.com • 5
Contents A Publication
VOLUME 5 • ISSUE 9 • SEPTEMBER 2008
1144CCOOVVEERR SSTTOORRYYDisciplines for Attacking Apps Running on the Microsoft Stack
While Windows, .NET, IIS and SQL Server might themselves seem stable,once one fails, the rest can shatter like cement blocks. By Pete Jenney
DDeeppaarrttmmeennttss7 • EditorialWhy the AOL development team takes a different tack on testing.
8 • ContributorsGet to know this month’s experts and thebest practices they preach.
9 • FeedbackIt’s your chance to tell us where to go.
11 • Out of the BoxNews and products for testers.
13 • ST&PediaIndustry lingo that gets you up to speed.
40 • Best PracticesOne of the .NET results of post-deploymententerprise testing. By Joel Shore
42 • Future TestA trainer of Microsoft testers breaks out ofthe black box. By Bj Rollison
Bring the use of artifact traceability toyour organization and use it as a QAmanagement tool.
By Venkat Moncompu and Sreeram Gopalakrishnan
3311A Sturdy DataFramework For
Repetitive Testing
3377 The Role Of Artifacts in QA
2244 .NET Tools You Already Have
Here’s a test harness that your teamcan build that will simplify manipula-tion and maintenance of your test data.
By Vladimir Belorusets
Learn how to simplify debugging byextracting diagnostics and data fromevent logs, using some of the many class-es included within the .NET FrameworkClass Library.
By Stephen Teilhet
Empirix gives you the freedom to test your way.Tired of being held captive by proprietary scripting? Empirix offers a suite of testing solutions that allow you to take your QA initiatives wherever you like.
Download our white paper,“Lowering Switching Costs for Load Testing Software,” and let Empirix set you free.
www.empirix.com/freedom
As surely as the SummerGames come every fouryears, I also find myselfcomplaining about theU.S.-centric television cov-erage. Most frustrating arechoices by the network toprovide no coverage at allof certain events or match-es, such as soccer andweightlifting. Granted, Am-erica’s soccer teams don’tusually advance into the final rounds,but fans of the sport still might want towatch Brazil vs. Germany. And justbecause the U.S. has only seven athletessnatching or doing the clean and jerkdoesn’t mean Americans aren’t interest-ed in weightlifting.
But this Olympics is different, andI’ll be the first to give NBC its due. Not for its television coverage does the network deserve praise, but forNBColympics.com, its terrific OlympicsWeb site. Thanks to video streaming,viewers were allowed to choose whichlive events to watch as they happened.An enhanced viewer (based onMicrosoft’s Silverlight) permitted up tofour streams at any one time; three inthumbnail boxes alongside one larger16x9 box with sound. There were nocommentators, but live mics at theevents kept you feeling connected asyou heard shouts from coaches, cheersfrom the crowd and splashes from the“Water Cube.”
And just as the “Bird’s Nest” andother Olympic arenas sprouted fromthe ground like winter wheat aroundBeijing, so too has a crop of Chinesecoders risen through the ranks of appli-cation developers in China’s emergingtech sector. “The Chinese, according toTopCoder’s software development com-petitions, are producing the top qualitydevelopers,” said Nick Schultz, aspokesman for TopCoder (www.topcoder.com), which has built a market inwhich developers compete for prizes
based on the quality oftheir code. “Check out thetop standings among thecommunity for developercompetitions. You’ll noticethat every single one of thetop ten ranked developersis from China.”
Developers (as well asdesigners and testers)stand to earn big buckschanneled to them from
companies using TopCoder as an out-source development service provider.One such company is AOL’s productsdivision, which uses TopCoder forsome of its customer-facing Web appli-cations, such as those of AOL Mail. NicPerez, technical director at AOL,praises Top Coder for its app-testingcapabilities. “At the very start, there’s aQA plan. We think of it as an integra-tion plan for how they’re going toattack the component.” The plans arereviewed and signed off on by AOL.“Then we’re quiet as they go do it.When we get the final code, we justlook at the test cases.” Perez said thequality is so good, he has come to relyon TopCoder’s quality reviewers. “Be-cause we’ve done so many componentswe don’t see the need [to perform fur-ther testing]. It meets our require-ments.”
Best Practices ReduxThis month I am pleased to introduceJoel Shore, who takes over the BestPractices column. He replaces GeoffKoch, who ably contributed sincebefore my time here and has movedon to pursue a career in marketing.Joel holds a special place in my per-sonal history. As director of the CRNLabs in 1995, he hired me as an edito-rial assistant for the labs, my first full-time editorial job. He is obviously aman of great foresight and vision. I amdelighted to have him writing forST&P. ý
NBC and China’sTop Coders
SEPTEMBER 2008 www.stpmag.com • 7
VOLUME 5 • ISSUE 9 • SEPTEMBER 2008
Ed Notes
President Ted Bahr
Executive Vice President Alan Zeichick
Software Test & Performance (ISSN- #1548-3460) ispublished monthly by BZ Media LLC, 7 High Street,Suite 407, Huntington, NY, 11743. Periodicals postagepaid at Huntington, NY and additional offices.
Software Test & Performance is a registered trade-mark of BZ Media LLC. All contents copyrighted2008 BZ Media LLC. All rights reserved. The priceof a one year subscription is US $49.95, $69.95 inCanada, $99.95 elsewhere.
POSTMASTER: Send changes of address to SoftwareTest & Performance, PO Box 2169, Skokie, IL 60076.Software Test & Performance Subscribers Servicesmay be reached at [email protected] or by calling 1-847-763-9692.
Cover Photograph by Tom Schmucker
Director of Circulation
Agnes [email protected]
EDITORIAL
SALES & MARKETING
READER SERVICE
Art DirectorLuAnn T. Palazzo
ART & PRODUCTION
BZ Media LLC7 High Street, Suite 407Huntington, NY 11743+1-631-421-4158fax [email protected]
EditorEdward J. Correia+1-631-421-4158 [email protected]
Copy DeskAdam LoBeliaDiana Scheben
Editorial DirectorAlan [email protected]
Contributing EditorsMatt HeusserChris McMahonJoel Shore
Publisher
Ted Bahr+1-631-421-4158 x101
Associate Publisher
David Karp+1-631-421-4158 x102
Advertising Traffic
Liz Franklin+1-631-421-4158 [email protected]
List Services
Lisa [email protected]
Reprints
Lisa [email protected]
Accounting
Viena Ludewig+1-631-421-4158 [email protected]
Customer Service/Subscriptions
Edward J. Correia
PETER JENNEY is a 20-year veteran ofsoftware testing, during which time hehas held positions with Rational,Dataware, Ipswitch and Legato. In hiscurrent role as VP of products at riskassessment consultancy SecurityInnovation, he directs the company’scommercial technology.
In our lead feature, Pete takes apartthe Microsoft stack to its componentparts, analyzing their interactions andthe impact of each on application stability. Turn to page 14.
Contributors
TO CONTACT AN AUTHOR, please send e-mail to [email protected].
8 • Software Test & Performance SEPTEMBER 2008
VENKAT MONCOMPU (right) andSREERAM GOPALAKRISHNAN areproject managers at Intellisys Tech-nology, an international IT servicesconsultancy. Venkat has a master’sdegree in engineering and more than12 years in the IT industry as a devel-oper, designer, business analyst and
testing coor-dinator.
Sreeramhas a master’s degree in businessadministration, is a certified PMPand has 12 years of experience as aQA analyst, business analyst and prac-tice lead. Beginning on page 37, theyexplain how traceability of artifactsused in testing and development canbe used to improve quality.
STEPHEN TEILHET has been workingwith the .NET platform since the pre-alpha version of .NET 1.0. He co-authored C# 3.0 Cookbook, ThirdEdition, (O’Reilly, 2008) with JayHilyard. He currently works for OunceLabs enhancing the company’s staticcode analysis tools.
Beginning on page 24, Stephenexplains how—using tools included withthe .NET framework—you can simpli-fy QA through analysis of event logs and debugging displays.
Certified by American Software Test-ing Qualifications Board, VLADIMIRBELORUSETS is an SQA manager atXerox, and is responsible for the qual-ity of its enterprise content manage-ment applications.
Vladimir explains his frameworkfor storing and reusing test data devel-oped during more than two decadesof software development, test automa-tion, test management, and softwareengineering experience. Turn to page 31.
WHAT’S YOUR FUNCTION(TESTER)? Regarding “Which Functional Testers Dothe Best Job?' (Test & QA Report, July 15,2008), I have read with interest your arti-cle on which functional testers do thebest job. It was the last line of your arti-cle that saddened me, however; you seemto be jumping from the goliaths of theindustry straight to open source with noconsideration for the smaller companiesthat compete so well against the big gunsof the industry.
Obviously the heavyweight corpora-tions such as HP/IBM/Compuware andBorland dominate this area; however Ido think it is a mistake to forget thesmaller niche players such as OriginalSoftware, Parasoft, Worksoft, etc., asmore often than not, it is in these com-panies where the real innovation withinthe industry is happening. Although Ican only talk for Original Software, someof the innovations, such as data extrac-tion, data scrambling, self-healing scriptsand the advance in assisted manual test-ing, cannot be found in the solutions ofHP/IBM etc. We are growing at a pacethat is far outstripping the market growthrate, and we are actively taking customersfrom the big guys. I am sure this storyis the same with other test software ven-dors of a similar size and agility toOriginal Software.
It is at the smaller end of the industrywhere innovation is happening. It is an
exciting place to be, and I think you aredoing your readership a disservice byignoring this area.
Scott AddingtonOriginal Software
I am doing a search for good QA testtools for a department that I have recent-ly taken over. Here is a good website witha broad list of QA test tools; many areopen source:
http://www.softwareqatest.com/qatweba .html–this page is for website testingtools. There are other pages at this site. Iam taking a deeper look at OpenSTA,which is open source. I have a subordi-nate looking at TestComplete Enterprise($2k per seat, not open, but COM basedand extensible). Another subordinate islooking into a C# web-crawler to see if wecan integrate testing into it.
BTW, thanks for the lead on pywin-auto; we will check that one out also.
Looking forward to your next article, John Bond
I am currently evaluating AutomatedQA's TestComplete for automated test-ing of our Delphi applications. Do youhave any information on how TestComplete stacks up? BTW, your reportsare a life saver to me. I am new to thetesting business, and I need all the helpI can get.
Thanks.Ed Bosgra
Having been in QA for over 10 years, andbeing familiar with many of the offeringsevaluated by Forrester, I think it'd be illus-trative to take a peek under the covers atthe Wave report in one of your future write-ups. Looking through their criteria, I finda lot of it to be rather uninsightful (e.g. ,the ability to capture environment variableinfo automatically is a key aspect of Resultreporting? I'd have to say that isn't one ofmy key business needs with regard toresults or even with regard to trou-bleshooting app and test issues!).
Similarly, I find that the inclusion ofa "Strategy" category as part of the numer-ic rating often ends up being rather mis-leading. In fact, the first thing I did withthe 2008 report was to set the strategy cat-egory to 0% and the current offering to100%. When I did that, it became ratherclear that Empirix and Borland farenotably poorer and IBM somewhat worse.Interestingly, HP and Compuware faredbetter. Ultimately, despite some of theunderlying criteria being suspect, the cur-rent offering is and should be the focalpoint of any evaluation (futures are vaporIMO until they're in beta). The strategypart is something that each evaluatorshould press the vendor with directly(when assessing their software), ratherthan relying on a third party.
In any way, I'd love to hear a realitycheck on the Wave!
“Testing Guru”
WAVING AT OPEN SOURCERegarding “Functional Testing Tools, theOpen-Source Wave” (Test & QA Report,July 15, 2008), my organization is usingAutoIt [http://www.autoitscript.com/autoit3/] for our .NET project. Howdoes this compare to the other open-source tools[?] For me, I have to makechanges all of the time. [I am] lookingfor a tool [that] adjusts to any changes ina Windows Form application where theUI is changing but the object[s] are not!
Thanks.Charles Bytheway
Feedback
SEPTEMBER 2008 www.stpmag.com • 9
FEEDBACK: Letters should include the writer’sname, city, state, company affiliation, e-mailaddress and daytime phone number. Send yourthoughts to [email protected]. Lettersbecome the property of BZ Media and may beedited for space and style.
DOWN WILL COME SOFTWARERegarding “Software Is Deployed, Bugs and All” (Test & QA Report, July 29, 2008),
I'm a code writer, AKA senior programmer.There would not be any bugs if the com-panies were not in such a hurry to secure their market share. Plus there are CASETOOLS that can check and double-check the effectiveness of any program to see if itscales (WORKS) or not.Then you have compilers that are written by programmers thatdon't catch the bugs before the software is deployed (HITS THE STORES).
And as far as the CASE TOOLS areconcerned, they are expensive for a jun-ior programmer or an independentwriter (freelance).The BIG companieshave really no excuse; they can affordthe CASE TOOLS. If [they] followed the flow chart to the letter it would not happenas much…that's why you need an error checker. Most [errors] are syntax errors any-way. [If] your compiler does not come with one then you'll have to acquire a programspecifically for that task.They don't call me compuman2153 for nothing.
K.J. Robinson
Xenocode, which makes virtualizationand obfuscation tools, in early August wasset to begin shipping an update toPostbuild 2008 for .NET, which enablesdevelopers to deploy .NET applicationsto systems that do not have the .NETframework installed or have a mismatchedversion. The update adds support for.NET 3.0 and 3.5, Visual Studio 2008, the
Windows Presentation Foundation andthe LINQ .NET extensions for native-lan-guage queries.
Among the benefits of the tool are theability to package and distribute applica-tions, dependencies, components, DLLs,runtimes and services as a single exe-cutable. Apps can be sent via e-mail, directfile transfer, removable media such as
USB drives or any other available method. “Postbuild is primarily designed for use
in deploying applications into productionenvironments,” said Xenocode CEO KenjiObata via e-mail. It integrates directly withVisual Studio and includes a scriptablecommand-line interface. The addition toapplication footprint is minimal, he said.
When Microsoft updates its frameworkwith features developers would like to takeadvantage of, “the software publisher val-idates the application on the new runtimeand then rebuilds and updates the pack-aged application,” Obata said. This minorinconvenience is offset by the benefit ofdeploying applications bundled with aspecific version of the .NET runtime,“insulating the application against poten-tial failures due to execution of the appli-cation on an untested forward version ofthe framework,” he said.
Pricing starts at US$1599 for fivedevelopers.
Postbuild applications perform as well asthose running in the .NET Framework as nor-mal, according to the company.
Test-tools maker StackSafe in early August released an updateto Test Center, adding the ability to automatically import virtu-alized environments and their components for staging, testing,analysis and reporting, either alone or in combination with phys-ical systems.
Introduced in January, Test Center employs virtualizationtechnology to give IT operations teams an easy way to simulatemulti-tiered systems and applications for testing and perform-ance tuning, according to company claims.
As before, copies of imported environments are introducedinto a working infrastructure stack that simulates the produc-tion configuration, enabling production-safe changes, regres-sion testing, patch testing, security and risk assessment, diag-nostics and root-cause analysis, emergency change testing, appli-cation assembly and validation, and compliance reporting, thecompany says.
Test Center benefits test teams, according to claims, by pro-viding a “broad view of the entire IT operations infrastructure,”enabling testing across physical machines running Linux andWindows, virtual machines set up with VMware and external
components such as databases, mainframes and other compo-nents that cannot yet be virtualized. The update is free to cur-rent licensees.
Automated ImportOf Virtual Environs
Out of the Box
Update .NET Apps, Not the Framework
Test Center now automates VMware component imports.
SEPTEMBER 2008 www.stpmag.com • 11
BusyBox Creators SueExtreme NetworksErik Anderson and Rob Landley, cre-ators of the BusyBox toolset forresource-constrained Linux and Unixsystems, in July filed another GPLenforcement lawsuit for copyrightinfringement.
With the help of the SoftwareFreedom Law Center, the action isagainst Extreme Networks Inc., a makerof high-performance network switchesand other connectivity and communi-cations gear. Four previous cases result-ed in out-of-court settlements in favor ofAnderson and Landley. In those cases,defendants were ordered to distributesource code in compliance with the GPLv2. They’re also looking for damages andlitigation costs.
According to the five-part complaint,which was filed July 17 in the UnitedStates District Court in New York, a judg-ment is sought that Extreme Networksbe immediately “enjoined and re-strained from copying, modifying, dis-tributing or making any other infring-ing use of Plaintiff’s software.” Alsosought is that Extreme “account for and disgorge to Plaintiffs all profits derived by Defendant from itsunlawful acts.”
“We attempted to negotiate withExtreme Networks, but they ultimatelyignored us,” said Aaron Williamson,SFLC counsel. “Like too many othercompanies we have contacted, they treated GPL compliance as an after-thought. That is not acceptable to us orour clients.”
BusyBox in late July agreed to end itslawsuit against Super Micro Computer,which manufactures and distributes com-puters and PC components. In exchangefor dismissing the suit, “Super Micro hasagreed to appoint an Open SourceCompliance Officer within its organiza-tion to monitor and ensure GPL com-pliance, to publish the complete and cor-responding source code for the versionof BusyBox it previously distributed, andto undertake substantial efforts to notifyprevious recipients of BusyBox fromSuper Micro of their rights to the soft-ware under the GPL,” according to anSFLC news release. The settlement also
includes an undisclosed financial con-sideration for the plaintiffs.
OpenMake MeisterDoes the Mash (up)With the recent release of Meister 7.2,OpenMake Software adds support forcross-platform builds within Microsoft’sVisual Studio, can “mash up” such buildswith those performed in Eclipse and oth-er IDEs, and offers features to simplify theprocessing of continuous integration usedin many agile processes, the company said.
The release also enhances Meister’sManagement Control Console, a Web-based portal that the company says per-mits QA engineers, production controlstaff and other non-developers to havecontrol and oversight of builds. The toolnow includes extended reporting usingthe PostgreSQL. “Simplifying build com-plexity is the no. 1 requirement we hearfrom developers,” said OpenMake CTOSteve Taylor. “Our Management Console[provides] a push-button process for exe-cuting and viewing build results from any-where in the world.”
The news comes on the heels of the May1 release of Meister 7.0, which allowedtesters to expose the build “forensics” andlinks to production binaries, which in turnpermit root cause analysis back to theoffending source code. Beginning with ver-sion 7, the tool now links with a centralknowledge base containing build-to-releaseinformation, connecting developers withproduction results, and giving test teamsbetter traceability of failed builds.
TI’s Low Power ChipsSave Battery Life, NotThe PlanetThe low-power chips are a series ofapplication processors and digital signalprocessors announced in July by TexasInstruments, they consume significantlyless power than their predecessors andprolong battery life of the devices builtaround them. The word “green” wasnowhere to be found.
Device designers using TI processorshave been asking the chip-maker forproducts that consume less power, more
or less supplanting prior requests formore and more power. “[The] develop-ers’ first question is now, ‘This is my powerbudget; how can TI help me do more withit?’ ” That’s according to Gene Fritz, aprincipal fellow at TI. The answer, he said,is simple: “Decades of experience allow TIto cut power consumption, improve ease-of-use and drive performance within itsarchitectures through better process tech-nology, peripheral integration, parallelprocessing, analog, connectivity, andpower management software and tools.”
The results is a series of about 15new devices in four product lines to bereleased over the coming year that itclaims will increase battery life to daysand weeks without sacrificing applica-tion performance.
Aimed at audio, medical and industri-al applications needing a high-accuracyfloating point unit is the 674x DSP, withTI says consumes one-third the power ofits rivals. In sleep mode, it sips as little as6 mW of power, according to claims, and420 mW in active mode. Using about halfthe power (415 mW) of existing chips inthe line is the 640x DSP, which TI says isintended for software-defined radio andother industrial instrumentation. It’splanned for early 2009.
Planned in the same time frame isthe latest in TI’s ARM-based applicationprocessor/DSP series, the OMAP L1x. Itwill run Linux or TI’s DSP/BIOS real-time kernel and is pin-for-pin compati-ble with devices in the 674x and 640xchips. Power consumption in activemode is rated at 435 mW.
For maximum battery life, develop-ers should look to the 550x, which usesjust 6.8 uW in deep sleep and 46 mW inactive mode. That’s half the power ofTI’s C5000-series chips, and is suited forportable music recording, noise-reduc-tion headphones and multi-parametermedical devices. The 550x includeslarge on-chip memory and an opti-mized FFT coprocessor. Availability isscheduled for early 2009.
Prices have been disclosed only forthe 674x, which will be sampling beforethe end of this year. In quantities of 100,pricing will start at less than US$9.
Send product announcements to [email protected]
SEPTEMBER 2008 www.stpmag.com • 12
The first day on the job at a Microsoft shop,you might hear something like this: “Thegraphic designers are using Expression Weband Silverlight, but the developers will justtake the HTML and put it in ASP, which we’lltest with Watir. Of course, we all collaboratewith VSTS.”
“Er, I’m sorry, what?” might be your feebleresponse, unless you’ve read this issue ofST&Pedia. The Microsoft Technology stack isa complex and sophisticated environment.Having a knowledge of this environment—even a surface knowledge—can make you amore effective and more valuable softwaretester.
Here’s a broad introduction to theMicrosoft environment and some of its relatedproducts. If you’re already .NET-savvy, feelfree to skip around.
Microsoft Tools and .NET ComponentsADO.NET / A technology anyone can use toconnect programming code to a database.
ASP.NET / A framework that allows thedeveloper to embed ‘snippets’ of code in Webpages, for example, to populate a table from adatabase. Visual Studio provides additionaltools to view and edit the page without look-ing at code.
C# (see-SHARP) / An object-oriented pro-gramming language based on C++ with influ-ences of Delphi and Java. Its code runs in amanaged runtime environment.
EXPRESSION WEB / Microsoft’s profes-sional Web layout tool; formerly known asMicrosoft FrontPage.
IIS / Internet Information Server. Micro-soft’s Web server manages requests for Webcontent by executing code or ‘serving up’data files.
SILVERLIGHT / Microsoft’s competitor toQuick Time and Shockwave, Silverlight is aWeb browser plug-in that supports animation,graphics, audio and video.
SQL SERVER / A database engine similar toAccess designed to scale to massive numbers
MS Terms of Endearment
ST&PediaTranslating the jargon of testing into plain English
SEPTEMBER 2008 www.stpmag.com • 13
Matt Heusser andChris McMahon
Matt Heusser and Chris McMahonare career software developers,testers and bloggers. They’re col-leagues at Socialtext, where theyperform testing and quality assur-ance for the company’s Web-based collaboration software.
Q:What would your answers be?
Did you exhaustively testthis? Are we doing an SVTafter our BVT? Does theperformance testing pass?What are your equivalenceclasses? Which heuristicsare you using?
A: ST&Pedia will help you answer questions
like these and earn therespect you deserve.
Upcoming topics:
OctoberSecurity & Code Analysis
NovemberTesters Choice Awards
DecemberTest Automation
January 2009Application Life CycleManagement
FebruaryDefect Tracking
MarchWeb PerformanceManagement
of simultaneous connections. Developers typi-cally write programs in C# or VB and connectto SQL Server via ADO.NET.
TEAM FOUNDATION SERVER (TFS) /A collaboration tool for the entire develop-ment team that includes version control,reporting, analytics, and process management.
VISUAL BASIC .NET (VB.NET) / The.NET version of Visual Basic, an easy-to-useprogramming environment that allows pro-grammers (and testers) to be productivequickly.
More MS StuffMSTEST / Integrated with Visual StudioProfessional, enables programmers to writelow-level tests in their language of choice toperform API testing. http://msdn.microsoft.com/en-us/library /ms182489(VS.80).aspx
WEBTEST / Software that records theHTTP traffic that goes from the server tothe browser – it does not test the GUI.http://msdn.microsoft.com/en-us/library/ms364082 (VS.80).aspx
LOADTEST / Simulates simultaneous userson a Web site using unit tests or existingWebTest scripts. http://msdn.microsoft.com/en-us/ magazin /cc163592.aspx
TEAM FOUNDATION BUILD / A contin-uous integration feature of TFS. http://msdn.microsoft.com/en-us/library/ms181710(VS.80).aspx
Free StuffNUNIT / Similar to MSTest, NUnit is a portof the JUnit framework for .NET languages;open source. http://www.nunit.org/
WATIR / Short for “Web Application TestingIn Ruby,” Watir is a set of Ruby libraries thatdrive Internet Explorer and monitors tests asthey run. Open source. Other implementa-tions include those for Firefox (FireWatir)and Safari (SafariWatir), .NET (WatiN) andJava (WatiJ). Watir - http://wtr.rubyforge.org/WatiN - http://watin.sourceforge.net/ ý
remote systems, from the very lowesttransport protocols to rich browser UIcomponents and data storage mecha-nisms. And, for the most part, we oftendon’t have a clue whether the code is anygood. These quality issues are staggeringand should be keeping you up at night.
But for applications running onWindows, quality can reasonably beassumed at several levels. For example,it’s pretty safe to assume that networkdrivers and other core operating systemservices are stable and secure. To a cer-tain extent the same assumptions canbe made about applications higher inthe stack, but these assumptions have tobe tempered with a dose of reality.Consider first the resources on which atypical Windows application depends,as illustrated in Figure 1.
Failure can start at any level, with theultimate result being failure of the appli-cation. In a compound/n-tier environ-ment, failure of an application may repre-sent the failure of a server, which, in turn,may result in failure of another applica-tion, and so on down the line. In all cases,failure will have some effect on systemsthat are dependent on it, and therein liesthe issue that we’re here to consider.
Application stability depends on theavailability and correctness of resources
that are consumed. When either is com-promised, the resulting behavior isundefined—read “unstable.” In addi-tion to making users unhappy, unstableapplications are the fodder of attackersas instability typically leads toexploitable vulnerabilities; hence, the
goal of hackers is generally to destabi-lize applications, which may be donesimply by attacking them via theresources they depend on.
Application security depends a greatdeal on application stability. Stabilitydepends a great deal on how applica-tions manage the resources they dependon, specifically how they handle excep-tions caused by missing resources or cor-rupted data delivered by resources.Consider the following code snippet:
BOOL getPdata(char* buf, DWORD*cnt) {m_hPipe = CreateFileA( szPipeName,
GENERIC_ALL | SYNCHRONIZE,
FILE_SHARE_READ,
NULL,
OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL,
NULL );
if( m_hPipe ==INVALID_HANDLE_VALUE )
{// TODO: Come up with a
good failure timeout in 2.0}
ReadFile( m_hPipe, &Ack, sizeof(Ack),&dwBytesRead, NULL );memcpy(buf,Ack,VdwBytesRead);*cnt = dwBytesRead;return( TRUE );}
Note that return values are notchecked at the time the pipe is created,nor when it’s been read. Also note thatthe read data is never checked in anyway, but immediately copied into thecalling buffer and that the size of the
Disciplines for Testing AppsRunning on the Stack That IsWindows, .NET, Internet Information Server and SQL Server
Pete Jenney is director of technology devel-opment at Security Innovation, an applicationsecurity and risk assessment consultancy.
By Pete Jenney
A typical Web application, regardless of development language, consumes hundreds of thousands lines of code from local and
www.stpmag.com • 15
Pho
togr
aph
by G
abri
el M
oisa
calling buffer is unknown. Developersand testers familiar with secure codingpractices will find this rife with defects,but the two critical ones are buffer over-flow and data corruption.
Any developer writing code like thisshould be taken out and shot at dawn,naturally, but it is a good example andit’s not made up. No kidding, this isactual code. Any application calling thisfunction will have to be prepared tohandle all types of failures dependingon what the caller needs to do with it.Now consider this code snippet:
try{char* buf = new char[MAX_PATH];int bytesRead, realLen;
BOOL pipeOpened = getPdata(buf, &bytesRead);
if(pipeOpened){if( (realLen = dataValid( buf )) != 0 ) // call ourdata cleanser{parseAndSend(buf, realLen);return( TRUE );}else{throw(“Malformed Data Error”);}}}
catch(…){logerror(_LINE_, “Pipe OpenFailure”);}
return( FALSE ); // Default, return failure
This code is prepared for failure andwill do the right thing if the data is cor-rupt or if anything messes around withmemory. Hence, it is hardened and willstand up to fairly rigorous testing. Mostcode however, will not.
Testing applications for fragility incompound environments is similar totesting those in an isolated environment.The same exploratory and fault injectiontechniques apply. The difference is that
when you crash anapplication in isola-tion, it only takesdown the system it’srunning on. In acompound applica-tion, it not only takesitself, but potentiallymany others too.And, of course, eachof those otherservers may bedepended on byother compound
applications, etc. You get the picture.This article will highlight some spe-
cialized testing techniques and postu-late mechanisms for testing com-pound applications running on theMicrosoft stack. It will get you thinkingabout what error handling means inan interdependent runtime environ-ment, how forcing errors during run-time is a critical activity in developingsecure and stable applications, andhow you might apply these techniquesto your own environments.
Testing on the StackCompound applications are deployedusing stacks of other applications, withthe Microsoft implementation beingamong the most popular. Its primarycomponents are IIS for Web services,.NET framework for application servic-es and SQL Server for data services(see Figure 2). Each component in thestack is, in itself, quite stable. However,as we’ve seen, the applications thatdepend on them are the primary con-cern, and forcing failures on one serv-er will help to uncover dependencydefects in others.
Techniques for testing compoundapplications vary, and the approach wewill take here will be to simulate envi-ronmental conditions that lead to appli-cation failure. Discovery and executionwill leverage several off-the-shelf tools—some freeware and some commercial—and target the platforms, including IIS,.NET and SQL Server, with the goal offorcing flaws that propagate unstablebehavior across the servers in a com-pound application.
Breaking Applications The most effective approach to breakingapplications is to force them to respondto hostile conditions with appropriateerror/exception handling mechanisms.If no such mechanism exists, the appli-cation will generally fail, or at leastbecome unstable and display unexpect-ed behaviors. The attack vectors that willprovide the best results are typicallythose that applications consume regu-larly and that developers are least likelyto worry about when developing excep-tion handlers. Specifically, these are:
• Registries• File streams• Network streams• Memory• Libraries (DLLs)Attacks that focus on these items
will yield some pretty spectacularresults in many cases. In terms of thecompound application, simple appli-cation failures could cause ripplesthroughout the entire system. Forcingfailures in each of these areas is two-phased, requiring discovery and actionusing tools or done manually. In eachcase, the activity is slightly different,and non-exhaustively described here.
STACK ATTACK
Application
.NET Framework
OS/Services (Win 32)
Application
Storage Memory Network Registry DLLs
FIG. 1: APP DEPENDENCIES
TABLE 1: PLAN OF ATTACK
Iteration
1
2
3
4
5
6
7
8
Test
Catastrophic
Catastrophic
Catastrophic
Catastrophic
Memory Constraint
Registry KeyCorruption
Memory Constraint
Network I/OCorruption
Goal
Force other server failure
Force other server failure
Force other server failure
Force other server failure
Force I/O & processing slowdown
Force application failure
Force application failure
Force client/caller failure
Server
Web
Database
Application 1
Application 2
Web
Application 1
Application 2
Database
16 • Software Test & Performance SEPTEMBER 2008
STACK ATTACK
Attacks Using the RegistryApplications may rely heavily on the reg-istry for runtime support. Interestingitems such as configuration, file locationand license information are good exam-ples of things that applications regularlystore and consume from there.Applications typically assume that theyhave access to the registry and that thedata stored there is correct. Therefore,the registry is a clear point of failure anda prime attack vector for attackers.Successfully attacking applications is amatter of denying access to the registrydata, changing it or corrupting it some-how. Consider replacing a temp file pathwith garbage like X:\\#$ERW—&UD^\\@#!D.FOOF, and the applicationtrying to open or create it. Naturally, theoperating system would reject it as hav-ing invalid characters, but what does theapplication do with it? What happens ifthe UI window width is set to 224 or someother ridiculously large number?
Attacks Using File StreamsApplications typically assume that thefiles they create are perfect and willsometimes consume anything as long asthe file extension is correct. Corruptingfiles, also referred to as “Fuzzing,” isbecoming very popular in testing circlesand is an integral part of the SecurityDevelopment Lifecycle (“The SecurityDevelopment Lifecycle,” Microsoft Press,2006). With its increasing popularity, it isalso beginning to attract tool developerswho see automating the process as anopportunity. Fuzzing the file stream canlead to application failures of all sorts.For example, if the pagination data in aword processing document header is offby a million pages or the value is corrupt,the application will likely die a horribledeath. File stream fuzzing attacks quicklyand simply, and hence is popular withthe hacking community. In real life, acci-dental file corruption can lead to appli-cation stability issues.
Attacks Using Network StreamsLike file streams, applications tend toassume that the network I/O they per-form always yields perfect data in andperfect data out. Fuzzing the datacoming off the stream can certainlylead to application failures, both ingeneral and in conjunction with spe-cific activities, such as SQL queriesand result-data processing. For exam-ple, what happens if a query is fuzzedand the result data is not checked, or
the resul -data is fuzzed? There are sev-eral methods to fuzz effectively, includ-ing on-the-wire protocol fuzzing andnetwork I/O virtualization. Each hasits advantages and the result is gener-ally the same—sometimes spectacular
and always interesting application fail-ures that allow attackers to rip into thesoft underbelly of poorly written appli-cations and pull out data.
Attacks Using MemoryDereferencing NULL pointers in C/C++is a classic faux pas and has lead to thou-sands of failures in software of all types.Those of you old enough may rememberthe awful feeling in your stomach as youwatched your hard disk formatting aftera crash while developing C programs onDOS 3.x. Memory errors are not just lim-ited to C/C++ though. Canonically, alllanguages have to call the operating sys-tem memory allocator at some point,and they all have to do it through thesame interface(s). Hence, memoryerrors can be forced regardless of theplatform by using an API intercept sys-
tem. Errors can be simple allocation fail-ures, or they can be corrupted pointersor other local or heap manipulations. Inany case, forcing memory allocationerrors is great way to quickly force inter-esting failures in applications of all types.
Attacks Using LibraryDependenciesFailing to check the return value ofLoadLibrary() calls is another exam-ple of assumed ownership, and onethat makes it easy to quickly knockover applications and potentially pro-vide an obvious attack vector. Considera difference between security andfunctional testing. In functional test-ing, if you pull a DLL away and theapplication crashes, that’s a bad thing.In security testing, if you pull a DLLaway and the application doesn’tcrash, that’s a bad thing. Why?Because the application that loads itdoesn’t check to see if the call suc-ceeded. So, an attacker can just write areplacement DLL, replace the originalwith it and own the application. Theunchecked load behavior will naturally
Internet Explorer (IE)
Global Failure Conditions • Server Unavailable • Server Unstable • Server Data Corrupt • ...
User Interface
Web Server
Application Server
Data Server Local Failure Conditions • Memory Unavailable • Registry Corrupt • File Stream Corrupt • Network Stream Corrupt • Library Missing • ...
Internet Information Server (IIS)
SQL Server
.NET Framework
FIG. 2: DIVIDE AND CONQUER
SEPTEMBER 2008 www.stpmag.com • 17
18 • Software Test & Performance SEPTEMBER 2008
lead to a crash as the application callsany of the methods in the missing DLLand immediately fails over, which pro-vides another juicy attack vector toexploit.
Breaking Compound ApplicationsThe five attacks described above may beapplied in several ways on individual sys-tems, and in most cases have similarapplicability in compound applications.For example, a named pipe is still anamed pipe whether it’s connecting twolocal processes or two remote servers.The bottom line is that interdependentservers need to incorporate all theerror/exception handling mechanismsfor all services, not just those they own.The tack we’ll take here is to attack thecompound application componentsand destabilize them.
Interactions between the servers dur-ing testing may be subtle or dramatic,and collecting runtime data from all ofthe participants is key to success. To thatend, it makes sense to borrow severaltools and techniques from the networkmanagement world that are focused oncollecting runtime information from allthe devices. Yet, instead of using them tolocate the root cause of a problem, we’relooking for how a root cause affects thesystems around it. Key assets to collectduring each test run from every serverare those that capture time-stampedruntime information, for example:
• Event logs• System/application logs• API call logs• First chance exception logs• Network traffic logs, etc.Test planning is based on an imagi-
nary application that consists of an IISWeb server, two simple .NET applica-tions and a SQL Server. One of the .NETapplications is ours, which digitizes
water for delivery over the Internet. Theother is a third-party application fordoing left handed smoke shifting andpresents as a Web service.
Each sample test run will describe adifferent fault or set of faults along withtooling and techniques for applyingthem. The goals of each are first tounderstand the behavior of our applica-tion in relation to the others in the sys-tem, and then the reciprocal.
The toughest tests target the appli-cation servers—we’re trying to causethem to fail based oninput from the others—and any direct or indirectinteractions between the two. The tests willsimulate resource con-straints and data corrup-tion from the databaseand Web servers, which ismore typical of a real-lifesituation, and the resultsare visible to the otherservers.
Several tools from thenetwork managementrealm may be useful inthis process, where man-aging multiple disparatedevices is what they do.Applying some of thattooling to the testingprocess will greatly speedthings up, especiallythings that do automaticevent correlation acrossmultiple servers. Whenyou start trying thesetechniques, it’s usually a good idea to talk to your local network adminis-trators and find out and borrow whatthey use to monitor and managethings. This could save a ton of analy-sis time.
Getting Started Understanding what’s going on behindthe scenes between the servers requireslogging and keen observation. Beforestarting anything, make sure that all thelogging capabilities are started and in asverbose a mode as possible to supportfuture correlation analysis. Also makesure that all the servers are synced tothe same timeserver (e.g., time.win-dows.com) and are current, again to aidin future correlation analysis.
Our sample plan (see Table 1, page16) will pass through several iterations,with the first being dramatic and repre-senting catastrophic failure of one serv-er at a time. It will then proceed tomore granular tests, with specific goalsin each.
Iterations 1 – 4Force applications to manage a missing majorresource by taking it away unexpectedly.
A missing server may cause severalvery real problems, and it’s the job ofthe application that depends on it tomanage loss of access and fail securely.The process is straightforward: If theservers all run on a single server, kill
one after the other with-out allowing them to usetheir shutdown handlers.For servers running ondifferent machines, youcan just disconnect themfrom the network or givethem the old “FullNyweide” and kick theirpower cords out of thewall. Again, don’t let thesystems execute any oftheir normal shutdowncode; it has to be abruptto make the AUT reactrealistically.
During the test, it isuseful to sniff and cap-ture the network trafficbetween the applicationsand the downed server tounderstand which typesof recovery activities areattempted, if any. Thisinformation may be cor-related with that in thesystem and event logs of
the still standing servers. Collectingtraffic and logs is a best practice for alltests of this nature and will show inter-esting connections, e.g., how a memo-ry allocation failure caused an applica-tion server to fall over.
STACK ATTACK
FIG. 3: AN EYE ON EVENTS
•Collecting
traffic and logs
is a best practice
for all tests of
this nature
and will show
interesting
connections.
•
SEPTEMBER 2008 www.stpmag.com • 19
Iteration 5Force memory constraints on the Web serverwith the goal of putting it into some form offailure state that will slow it down and doodd things.
The conditions we want to createwill force the Web server to hit theswapper and start to thrash. This mayor may not cause failures in the con-suming application, but it will be a datapoint either way.
There are a couple of ways to creatememory constraints and failures. Thefirst is to tear open the machineand start pulling chips out. Thesecond is to start a lot of applica-tions that will compete for mem-ory. But the easiest is to use afault injection tool to set up avery low value for the availablesystem memory or to set up spo-radic allocation failures. A test-ing tool that virtualizes applica-tions is a good choice herebecause it allows testers tomanipulate their runtime envi-ronment and allows you toquickly manipulate the memoryavailable to just the applicationunder test (AUT), so memory con-straints won’t affect any other programsrunning on the system.
Using the tool to simulate networkbandwidth limitations and random cor-ruptions in the network stream will alsocause the Web server to react in inter-esting ways and, in turn, drive interest-ing behaviors in dependent systems.Again, as the application is virtualizedthe rest of the system is not exposed tothe faults the AUT sees.
While the Web server is being beatenup, pay careful attention to the systemand event logs on all the systems in thetest and the applications. After the testrun is complete, all of the logs shouldbe saved for later reference, or, if some-thing crashed, immediately reviewed forcoincident events.
Iteration 6Force a failure in an application by corrupt-ing registry keys it consumes and make itcrash or perform non-deterministically.
Finding the registry keys an appli-cation uses is pretty easy using com-monly available free and commercialtools. Several run in parallel to appli-cations and capture all of their reg-istry, allowing you to get a good feelfor how it is used and what keys mightbe interesting. Select a key that seems
to be used a lot and jump over to it inyour favorite registry editor. Save theentire section of the registry you’regoing to work on before actuallytouching it, and then try to changethe values of the interesting key(s). Ifyou can’t change the values, stop theAUT, change things around andrestart, keeping an eye on the key tosee if the AUT alters it/them duringstartup or shutdown.
If the AUT does manipulate the key,and you still can’t edit it at run-time, try
a fault injection tool that virtualizes theapplication and allows real-time manipu-lation of the keys. Failing that, just deletethe darned things. If the application isfragile, missing keys will send it reeling.
Try several avenues to cause failurewhen working the registry such as:
Corrupting key values—if the appli-cation uses data to control executionflow or configuration, out of range orcorrupted values may destabilize things
Forcing error return values—ifyour tools allow registry call return val-ues to be overridden, try returningREGISTRY_CORRUPT or ACCESS_DENIED
Changing paths—if the applicationuses keys to point to configuration ortemp files, change them and try redi-recting data to non-standard places,pipes or shares
Changing the key type—try chang-ing the type of data the key stores, forexample, a string to binary or othertype.
While you’re manipulating the reg-istry, pay attention to the system andevent logs and watch for odd behaviorfrom the other servers. Failures in reg-istry reading may be pretty dramatic insome cases and subtle in others. Lossor corruption of data may cause com-plete failure or allow redirection of
sensitive data or affect the way that theapplication communicates with otherservers and destabilize entire com-pound applications.
Iteration 7Force a failure in an application by fuzzingfile streams it consumes and make it crash,perform non-deterministically or forward cor-rupted data.
File fuzzing is becoming popularamong testers, and there are a lot of free-ware and several good commercial tools
available that do it well. The trickto success is to find all the filesthat applications consume andwhen they use them, and thencorrupt them in a meaningfulway that will cause failures.Finding them is simple withsome commonly available tools;fuzzing them is best done by oneof the commercial or freefuzzing frameworks and/orproducts. For cases like theMicrosoft SDL, where it’s arequirement to open 100,000corrupt files of the type theapplication creates and reads,
consider a tool that virtualizes a singlefile and re-corrupts it on every open.Consider the following pseudo codesnippets:
// Open and use the virtual file 100000 times, if itfails, bail out For( I = 1 to 100,000 )App.FileOpenUseAndClose(“myfile
.mine_I_own_it.you_cant_have_it”)Next I
Return “All Good”
:ErrorTrap
Return “Failed on “ + I
//—————————————————————
// Open and use 100000 different files, if it fails, bailout Array String[100000]// 100000 files I hand built …took me forever// I hate my job and my boss
LoadNames(Array)
For( I = 1 to 100,000)App.FileOpenUseAndClose(Array[I])
Next I
:ErrorTrapReturn “Failed on “ + I
Follow the same steps as in iterationsix to set up the AUT and locate targetsto manipulate. Files are slightly differ-ent than registry values because aprocess may create a file at runtime,keep it open and locked, and destroy it
STACK ATTACK
•File fuzzing is becoming popular
among testers, and there are a
lot of tools that do it well.
•
22 • Software Test & Performance SEPTEMBER 2008
at shutdown. So you’ll need to find filesthat are available at runtime and thatthe application really uses. Fuzzingcomes in two basic flavors: random andparametric. Random just corrupts byteswherever; parametric allows specificparts of files to be changed/corruptedin specific ways. It’s best to experimentwith both. And while the tests are beingexecuted, monitor all channels out ofthe application for corrupted data too.
Iteration 8Force a failure in an application by fuzzingthe network streams it consumes and make itcrash, perform non-deterministically or for-ward corrupted data.
Fuzzing network I/O is similar tofuzzing files, and it’s more significant asthe majority of data consumed bytoday’s applications is network-based,not file-based. Fuzzing channels can bedone in two ways, by generating it on asystem and sending it to a targetmachine, or by virtualizing the networkI/O channels the application consumesand corrupting them on the fly. Bothwork well and should be used in con-junction with a network sniffer to moni-tor the response packets from other sys-tems in the application. Try several dif-ferent attacks on the stream like:
• Randomly corrupt data from thestream before it gets to the app
• Randomly corrupt data going tothe stream
• Insert long strings of AAAAs orsome other unexpected characters
As with file fuzzing, the results can bedramatic and can easily destabilize theapplication under test. But unlike files,you can also target other machines forabuse and send dependent machinescorrupted (or otherwise altered) data totest various different failure scenarios,simulating various ways your AUT canfall over and forcing responses.
Simple Correlation AnalysisFor every system crash or instability, therest of the components of the applica-tion need to be evaluated for events.The simplest method is a temporalsearch (see Figure 3, page 18), whereyou take the timestamp of the crashfrom the log and search for timestampsin the same range in all the other logfiles. Network sniffer logs are a valuableresource in the analysis and a weathereye kept out for high traffic levels thatmight indicate recovery attempts orother situational activity. File, API and
registry I/O logs are useful in theprocess, as they may point to specific fileoffsets, call patterns and other interest-ing things that you might normally asso-ciate with normal server operation.
Testing applications completely re-quires that all the runtime conditionsin which they may fail are executed andtested. Understanding where applica-tions are vulnerable or prone to insta-bility is challenging, but using faultinjection and fuzzing techniques allowstesters to force applications to exerciseerror handlers and quickly exposeproblem areas.
In larger, compound applications,this testing is more critical as theresources an application consumes orprovides may be shared or dependent,
and its loss might have broad impact onsystems outside the expected applicationboundaries. Testing in a typical lab envi-ronment does not generally include thedepth needed to test all the needed con-ditions, as it’s difficult to do, but usingthe techniques described here will helpget you moving in the right directionand quickly discover lurking problems.
Testing on the Microsoft stackrequires an additional layer of effort inthe quality process and carries a gooddeal of manual labor to do properly.Much of the process can be accelerateddramatically with the use of tools thathelp with discovery and analysis [Seesidebar for suggestions], though thereis no all-encompassing framework forsetup and execution …yet. ý
STACK ATTACK
There are several tools both commercial and free that are great for implementing the test-ing described in this article. Most have overlapping functionality, but in many cases thestandalone implementations are best for a specific task.
RegMon for Windows v7.04 by Microsoft [SysInternals]technet.microsoft.com/en-us/sysinternals/bb896652.aspxRegMon is a free and very useful discovery tool that monitors all registry interaction fromany and all running applications, and allows users to quickly jump to Regedt32 to manip-ulate registry values.
FileMon for Windows v7.04 by Microsoft [SysInternals]technet.microsoft.com/en-us/sysinternals/bb896642.aspxFileMon is a free and very useful discovery tool that monitors all file activity for any and allrunning applications, and allows users to quickly jump to files in Explorer for management.
Peach Fuzzing Platform 2.0 peachfuzzer.comPeach is a free and comprehensive fuzzing platform that allows on-the-wire fuzzing of net-work I/O and files of almost any type.
Defensics 2.0 by Codenomiconwww.codenomicon.com/defensicsDefensics is a powerful commercial fuzz testing platform for on the wire fuzzing of mostall protocols.
Wireshark 1.0.0 www.wireshark.org/about.htmlWireshark, formally known as Ethereal, is a free and powerful tool for real-time monitor-ing and analysis of network traffic.
WhatsUp Gold by Ipswitchwww.whatsupgold.comWhatsUp Gold is an inexpensive professional-grade network discovery and managementtool that provides real-time server and service monitoring.
Event Analystwww.eventanalyst.com/index.htmEvent Analyst is an inexpensive tool that allows the consolidation and correlation of serv-er log files.
Holodeck Enterprise Edition v2.8 by Security Innovation Inc.www.securityinnovation.com/holodeckHolodeck is a professional-grade discovery and fault injection tool that virtualizes anapplication’s runtime environment and allows testers to completely control its resources.
A STACK OF TESTING TOOLS
JUST BECAUSE YOUR INFRASTRUCTURE SURVIVED THE LOAD TEST DOESN’T MEAN
THE CUSTOMER EXPERIENCE DID TOO.With Gomez Reality Load™ XF you can test from outside the data center — from machines in the real world — where the unpredictability of network and user conditions can challenge today’s complex web applications.
Unlike competing offerings, Gomez Reality Load XF delivers a global network of over 40,000 backbone and desktop testing locations, more than 100 browser and operating system combinations, fast results and encyclopedic detail.
So, even if your infrastructure survived a traditional load test, with Gomez you’ll know if the customer experience did too.
Gomez® and Gomez.com® are registered service marks, and ExperienceFirstTM, Active NetworkTM XF and Active Last MileTM XF are service marks of Gomez, Inc. All other trademarks and service marks are the property of their respective owners.
With no software to buy or install, or advance reservations, you can start load testing fast.
And, you can easily combine load testing with our Active Network™ XF or Actual Experience™ XF products, for 24x7 production monitoring.
For details in the United States call +1 877.372.6732, in the United Kingdom +44 (0)1753 626 632, or in Germany +49 (0)40 53299 207. Or visit www.gomez.com.
Gomez. Ensuring Quality Web ExperiencesSM.
> Gomez, Inc. 10 Maguire Road Lexington, Massachusetts 02420
24 • Software Test & Performance SEPTEMBER 2008
nostic information about an application and the environmentit is running in. This article will address specific solutions toproblems that both developers and QA personnel can use tomake monitoring and debugging an application easier andadd to your arsenal of tools to make locating and fixing prob-lems in your applications much quicker and easier.
This code is written to run under C# 3.0 and the .NETFramework v3.5. However, nearly all of the code—except forthe code that uses LINQ (Language Integrated Query)—canbe compiled under C# 2.0 and the .NET Framework v2.0 andv3.0. Also, some knowledge of LINQ is presumed.
Using Event Logs in Your ApplicationTaking advantage of the built-in Microsoft Windows event logmechanism allows your application to easily log events thatoccur, such as startup, shutdown, critical errors and even secu-rity breaches. Along with reading and writing to a log, theevent log APIs provide the ability to create, clear, close andremove events from the log.
You should use the event log mechanism to record specificevents that occur infrequently. You should also try to minimizethe number of entries written to the event log, because writingto the log causes a performance hit. Writing too much informa-tion to the log can noticeably slow your application. Pick andchoose the entries you write to the event log wisely. If you needto create a detailed log of all the events that occur in your appli-cation, such as for debugging purposes, you should use theSystem.Diagnostics.Debug or System.Diagnostics.Trace classes.
To easily add event logging to your application, simply addthe AppEvents class below, which contains all the methodsneeded to create and use an event log in your application.
LISTING 1using System;using System.Diagnostics;using System.Collections.Generic;
public class AppEvents{
public AppEvents(string logName) :this(logName, Process.GetCurrentProcess().ProcessName, “.”) {}
public AppEvents(string logName, string source) :this(logName, source, “.”) {}
public AppEvents(string logName, string source, string machineName){
this.logName = logName;this.source = source;this.machineName = machineName;
if (!EventLog.SourceExists(source, machineName)) {
EventSourceCreationData sourceData =new EventSourceCreationData(source, logName);
sourceData.MachineName = machineName;
EventLog.CreateEventSource(sourceData);}
log = new EventLog(logName, machineName, source);log.EnableRaisingEvents = true;
}
private EventLog log = null;private string source = “”;private string logName = “”;private string machineName = “.”;
public string Name {
get{return (logName);} }
public string SourceName {
get{return (source);} }
public string Machine{
get{return (machineName);}}
public void WriteToLog(string message, EventLogEntryType type,CategoryType category, EventIDType eventID)
{if (log == null){
throw (new ArgumentNullException(“log”,“Open the event log before writing to it.”));
}
log.WriteEntry(message, type, (int)eventID, (short)category);}
public void WriteToLog(string message, EventLogEntryType type,CategoryType category, EventIDType eventID, byte[] rawData)
{
By Stephen Teilhet
T he .NET Framework Class Library containsmany classes that allow testers to obtain diag-
Pho
togr
aph
by S
teve
Dib
blee
A longtime .NET developer and author, Stephen Teilhet currently worksfor security tool maker Ounce Labs.
26 • Software Test & Performance SEPTEMBER 2008
.NET TOOLBELT
if (log == null) {
throw (new ArgumentNullException(“log”,“Open the event log before writing to
it.”));}
log.WriteEntry(message, type, (int)eventID,(short)category, rawData);
}
public EventLogEntryCollection GetEntries(){
if (log == null){
throw (new ArgumentNullException(“log”,“Open the event log before retrieving its
entries.”));}
return (log.Entries);}
public void ClearLog(){
if (log == null){
throw (new ArgumentNullException(“log”,“Open the event log before clearing
it.”));}
log.Clear();}
public void CloseLog(){
if (log == null){
throw (new ArgumentNullException(“log”,“The event log was not opened.”));
}log.Close();log = null;
}public void DeleteLog(){
if (EventLog.SourceExists(source,machineName))
{EventLog.DeleteEventSource(source,
machineName);}
if (logName != “Application” &&logName != “Security” &&logName != “System”)
{if (EventLog.Exists(logName,
machineName)){
EventLog.Delete(logName,machineName);
}}
if (log != null){
log.Close();log = null;
}}
}
The EventIDType and CategoryType enumerationsused in this class are defined as follows:
public enum EventIDType
{
NA = 0,Read = 1,Write = 2,ExceptionThrown = 3,BufferOverflowCondition = 4,SecurityFailure = 5,SecurityPotentiallyCompromised = 6
}
public enum CategoryType : short{
None = 0,WriteToDB = 1,ReadFromDB = 2,WriteToFile = 3,ReadFromFile = 4,AppStartUp = 5,AppShutDown = 6,UserInput =7
}
The AppEvents class provides applica-tions with an easy-to-use interface for cre-ating, using and deleting one or moreevent logs in your application. Your appli-cation might need to keep track of sever-al logs at one time. For example, yourapplication might use a custom log totrack specific events, such as startup andshutdown, as they occur in your applica-tion. To supplement the custom log, yourapplication could make use of the securi-ty log already built into the event log sys-tem to read/write security events thatoccur in your application.
Support for multiple logs also comesin handy when one log needs to be cre-ated and maintained on the local com-puter and another duplicate log is need-ed on a remote machine. This remotemachine might contain logs of all run-ning instances of your application oneach user’s machine. An administratorcould use these logs to quickly discoverif any problems occur or if security hasbeen breached in your application. Infact, an application could be run in thebackground on the remote administra-tive machine that watches for specificlog entries to be written to this log fromany user’s machine. More about watch-ing event logs for specific events later.
Keeping duplicate copies of event logs
will also help during a forensics investiga-tion after a security breach occurs. Thelogs can be compared to determine notonly if the attacker had access to modifythe event log, but also which events weremodified. Unless the attacker had accessto the local and remote event logs, aforensics investigation can easily uncoverthe motives of the attacker and the extentof the damage.
Let’s dive into the specifics of theAppEvents class. The methods of theAppEvents class are described in Table 1:
An AppEvents object can be addedto an array or collection containingother AppEvents objects; eachAppEvents object corresponds to aparticular event log. The followingcode creates two AppEvents classesand adds them to a generic Dictionarycollection:
public void CreateMultipleLogs(){AppEvents appEventLog = new
AppEvents(“AppLog”, “AppLocal”);AppEvents globalEventLog = new
AppEvents(“System”, “AppGlobal”);
Dictionary<string, AppEvents>logList = new Dictionary<string, AppEvents>();
logList.Add(appEventLog.Name,appEventLog);
logList.Add(globalEventLog.Name,globalEventLog);
}
To write to either of these two logs,obtain the AppEvents object by namefrom the Dictionary object and call itsWriteToLog method:
logList[appEventLog.Name].WriteToLog(“App
startup”,EventLogEntryType.Information,
CategoryType.AppStartUp,EventIDType.ExceptionThrown);
logList[globalEventLog.Name].WriteToLog(“Appstartup security check”,
EventLogEntryType.Information,CategoryType.AppStartUp,
EventIDType.BufferOverflowCondition);
TABLE 1:THE APPEVENTS CLASS
Method
WriteToLog
GetEntries
ClearLog
DeleteLog
CloseLog
Description
This method is overloaded to allow an entry to be written tothe event log with or without a byte array containing raw data.
Returns all the event log entries for this event log in anEventLogEntryCollection object.
Removes all the event log entries from this event log.
Deletes this event log and the associated event log source.
Closes this event log, preventing further interaction with it.
Storing all AppEvents objects in aDictionary object allows you to easilyiterate over all the AppEvents objectsthat your application has created. Usinga foreach loop, you can write a singlemessage to both a local and a remoteevent log:
foreach (KeyValuePair<string,AppEvents> log in logList)
{log.Value.WriteToLog(“App startup”,EventLogEntryType.FailureAudit,
CategoryType.AppStartUp,EventIDType.SecurityFailure);
}
To delete each log in the logList Dictionary object,you can use the following foreach loop:
foreach (KeyValuePair<string,AppEvents> log in logList)
{log.Value.DeleteLog();
}logList.Clear();
You should be aware of several keypoints. The first concerns a small prob-lem with constructing multipleAppEvents classes. If you create twoAppEvents objects and pass in the samesource string to the AppEvents con-structor, an exception will be thrown.Consider the following code, whichinstantiates two AppEvents objects withthe same source string:
AppEvents appEventLog = newAppEvents(“AppLog”, “AppLocal”);
AppEvents globalEventLog = newAppEvents(“Application”, “AppLocal”);
The objects are instantiated withouterrors, but when the WriteToLogmethod is called on the globalEventLogobject, the following exception isthrown:
An unhandled exception of type
‘System.ArgumentException’ occurred in system.dll.Additional information:The source
‘AppLocal’ is not registered in log ‘Application’.(It is registered in log ‘AppLog’.) “.
The Source and Log properties must be matched,or you may set Log to the empty
string, and it will automatically be matched to theSource property.
This exception occurs because theWriteToLog method internally calls theWriteEntry method of the EventLogobject. The WriteEntry method checks tosee whether the specified source is regis-tered to the log you are attempting towrite to. In this case, the AppLocal sourcewas registered to the first log it wasassigned to—the AppLog log. The sec-ond attempt to register this same sourceto another log, Application, failed silent-ly. You don’t know that this attempt faileduntil you try to use the WriteEntrymethod of the EventLog object.
Another key point about theAppEvents class is the following code,placed at the beginning of each method(except for the DeleteLog method):
if (log == null) {
throw (newArgumentNullException(“log”,
“Open the event log before writ-ing to it.”));
}
This code checks to see whether theprivate member variable log is a nullreference. If so, an ArgumentExceptionis thrown, informing the user of thisclass that a problem occurred with thecreation of the EventLog object. TheDeleteLog method does not check thelog variable for null, since it deletes theevent log source and the event log itself.The EventLog object is not involved inthis process except at the end of thismethod, where the log is closed and setto null, if it is not already null.
Regardless of the state of the log vari-able, the source and event log shouldbe deleted in this method.
The DeleteLog method makes a crit-ical choice when determining whetherto delete a log. The following code pre-vents the application, security and sys-tem event logs from being deleted fromyour system:
if (logName != “Application” &&logName != “Security” &&logName != “System”)
{if (EventLog.Exists(logName,
machineName)){
EventLog.Delete(logName,machineName);
}}
If any of these logs are deleted, soare the sources registered with the par-ticular log. Once the log is deleted, thedeletion is permanent; and believe me,it’s no fun trying to re-create a log andits sources without a backup.
As a last note, the EventIDType andCategoryType enumerations aredesigned mainly to log security-typebreaches as well as potential attacks onthe security of your application. Usingthese event IDs and categories, theadministrator can more easily trackdown potential security threats and dopostmortem analysis after security isbreached. These enumerations can eas-ily be modified or replaced with yourown to allow you to track differentevents specific to your application.
Searching Event Log EntriesNow that your application supports writ-ing events to the Event Log, it is possiblethat the application might have addedquite a few entries to the log. To performan analysis of how the application oper-ated, how many errors were encounteredand so on, you need to be able to per-form a search through all of the entriesin an event log. You will eventually haveto sift through all the entries your appli-cation writes to an event log to find theentries that allow you to perhaps fix a bugor improve your application’s security sys-tem. Unfortunately, there are no goodsearch mechanisms for event logs.
To fix this we have built the EventLogSearch class, to which you’ll add stat-ic methods, allowing you to search forentries in an event log based on variouscriteria. In addition, this search mecha-nism allows complex searches involvingmultiple criteria to be performed on an
SEPTEMBER 2008 www.stpmag.com • 27
.NET TOOLBELT
TABLE 2: OTHER SEARCH METHODS
Searchmethodname
FindCategory (overloaded to accept astring type category name)
FindCategory (overloaded to accept ashort type category
FindEntryType
FindInstanceID
FindMachineName
FindMessage
FindSource
Entry property searched
Category == categoryNameQuery
Category == categoryNameQuery
EntryType == entryTypeQuery
InstanceID == instanceIDQuery
MachineName == machineNameQuery
Message == message.Query
Source == sourceQuery
28 • Software Test & Performance SEPTEMBER 2008
.NET TOOLBELT
event log at one time.
using System;using System.Collections;using System.Diagnostics;
public sealed class EventLogSearch {
private EventLogSearch() {} // Prevent this classfrom being instantiated.
public static EventLogEntry[] FindEntryType ( EventLogEntryCollection logEntries,
EventLogEntryType entryTypeQuery) {
var entries = from EventLogEntry entry inlogEntries
where entry.EntryType ==entryTypeQuery
orderby entry.TimeGeneratedascending
select entry;
return entries.ToArray();}
public static EventLogEntry[]FindTimeGeneratedAtOrAfter(
EventLogEntryCollection logEntries, DateTimetimeGeneratedQuery)
{var entries = from EventLogEntry entry in
logEntrieswhere entry.TimeGenerated >=
timeGeneratedQueryorderby entry.TimeGenerated
ascendingselect entry;
return entries.ToArray();}
}
What makes this class so flexible isthat new searchable criteria can beadded to this class by following the samecoding pattern for each search method.For instance, the following exampleshows how to add a search method tofind all entries that contain a particularusername:
public static EventLogEntry[]FindUserName(EventLogEntryCollection logEntries,
string userNameQuery)
{var entries = from EventLogEntry
entry in logEntrieswhere
entry.UserName == userNameQueryorderby
entry.TimeGenerated ascendingselect entry;
return entries.ToArray();}
The methods shown in Table 2 (page27) list other search methods that couldbe included in this class and describewhich property of the event log entriesthey search on.
The FindCategory method can beoverloaded to search on the categoryname, the category number or both.
The following method makes use ofthe EventLogSearch methods to findand display entries that are marked asError log entries:
public void FindAnEntryInEventLog()
{EventLog log = new
EventLog(“System”);
EventLogEntry[] entries =EventLogSearch.FindEntryType(log.Entries,
EventLogEntryType.Error);
// Print out the informationforeach (EventLogEntry entry in
entries){
Console.WriteLine(“Message:“ + entry.Message);
Console.WriteLine(“InstanceId:“ + entry.InstanceId);
Console.WriteLine(“Category:“ + entry.Category);
Console.WriteLine(“EntryType:“ + entry.EntryType.ToString( ));
Console.WriteLine(“Source:“ + entry.Source);
}}
Note that this search mechanism cansearch within only one event log at a time.
To illustrate how searching works,let’s assume that you are using theFindInstanceID method to search onthe InstanceID. Initially, you would callthe FindInstanceID search method,passing in the EventLogEntryCollectioncollection (which contains all entries inthat event log) or even an array ofEventLogEntry objects. A LINQ query isused to search through theEventLogEntryCollection for specificevent log entry objects (EventLogEntry)that satisfy the where clause of theLINQ query. The ToArray method isused to convert the resulting LINQquery results into an array ofEventLogEntry object. TheFindInstanceID method will thenreturn an array of EventLogEntryobjects that match the search criteria(the value passed in to the second argu-ment of the FindInstanceID method).
LINQ is used in the event logsearch because of its power and sim-plicity. Its declarative syntax is easier toread and understand, while at thesame time providing a wealth of oper-ations that can be performed on yourquery. LINQ allows not only sorting,but also grouping, joins with otherdata sets, and multiple search criteriain the where clause. LINQ also allowsthe use of set operations such asunion, intersect, except and distinct,on your data. There are aggregateoperators, such as Count and Sum, aswell as quantifier operations, such asAny, All and Contains. This is but asampling of the operations a LINQquery can perform on your data. Bycombining these operations, you cancome up with your own custom searchqueries for your event log entries.
The real power of this searchingmethod design is that the initial searchon the EventLogEntryCollectionreturns an array of EventLogEntryobjects. This EventLogEntry array maythen be passed back into anothersearch method to be searched again,effectively narrowing down the searchquery.
For example, the EventLogEntryarray returned from the FindInstanceIDmethod may be passed into anothersearch method, such as the FindEntryType method, to narrow down thesearch to all entries that are a specificentry type (informational, error, etc.).This can continue until the search hasbeen sufficiently narrowed down. Thefollowing method finds and displays
FIG. 1: CUSTOM DEBUGGER DISPLAY
SEPTEMBER 2008 www.stpmag.com • 29
.NET TOOLBELT
entries generated at or after 5/3/2008,marked as an error type entry, and con-taining an event ID of 3221232483 bysimply passing in the results of onequery into another:
public void FindAnEntryInEventLog() {
EventLog log = newEventLog(“System”);
EventLogEntry[] entries =EventLogSearch.FindTimeGeneratedAtOrAfter(log.Entries,
DateTime.Parse(“5/3/2008”));entries =
EventLogSearch.FindEntryType(log.Entries,EventLogEntryType.Error);
entries =EventLogSearch.FindInstanceId(log.Entries,3221232483);
// Print out the informationforeach (EventLogEntry entry in
entries){
Console.WriteLine(“Message:“ + entry.Message);
Console.WriteLine(“InstanceId:“ + entry.InstanceId);
Console.WriteLine(“Category:“ + entry.Category);
Console.WriteLine(“EntryType:“ + entry.EntryType.ToString());
Console.WriteLine(“Source:“ + entry.Source);
}}
Watching the Event Log for aSpecific EntrySometimes a way to search your eventlog for specific events of interest is notenough; you need a mechanism toalert you when highly important eventsoccur, such as when an application ter-minates unexpectedly or a critical secu-rity event is logged. What you need is amonitoring application to watch forspecific log entries to be written to theevent log and then send an alert notifi-cation to the administrator. For exam-ple, you might want to watch for a logentry that indicates that an applicationencountered an error from which itcould not recover gracefully, or that amalicious user is trying to attack yourapplication by feeding it malformeddata. These types of log entries need tobe reported in real time to the appro-priate person or persons.
Monitoring an event log for a specif-ic entry requires the following steps:
• Create a method to set up the eventhandler to handle event log writes:
public voidWatchForAppEvent(EventLog log)
{log.EnableRaisingEvents = true;
// Hook up theSystem.Diagnostics.EntryWrittenEventHandler.
log.EntryWritten += newEntryWrittenEventHandler(OnEntryWritten);
}
• Create the event handler to exam-ine the log entries and determinewhether further action is to be per-formed. For example:
public static voidOnEntryWritten(object source,
EntryWrittenEventArgs entryArg){
if (entryArg.Entry.EntryType ==EventLogEntryType.Error)
{
Console.WriteLine(entryArg.Entry.Message);
Console.WriteLine(entryArg.Entry.Category);
Console.WriteLine(entryArg.Entry.EntryType.ToString());
// Do further actions here asnecessary...
}}
This code revolves around theEntryWrittenEventHandler delegate,which calls back to a method wheneverany new entry is written to the event log.The EntryWrittenEventHandler dele-gate accepts two arguments: a source oftype object and an entryArg of typeEntryWrittenEventArgs. The entryArgparameter is the more interesting of thetwo. It contains a property called Entrythat returns an EventLogEntry object.This EventLogEntry object contains all the information you need concern-ing the entry that was written to theevent log.
The event log that you are watching
is passed as the WatchForAppEventmethod’s log parameter. This methodperforms two actions. First, it sets log’sEnableRaisingEvents property to true.If this property were set to false, noevents would be raised for this eventlog when an entry is written to it, effec-tively turning off the delegate. The sec-ond action this method performs is toadd the OnEntryWritten callbackmethod to the list of event handlers forthis event log.
Note that the Entry object passed tothe entryArg parameter of the OnEntryWritten callback method is read-only, sothe entry cannot be modified before itis written to the event log.
Creating Custom DebuggingDisplays for Your ClassesMoving on to something quite differ-ent, here’s a useful debugging featurethat you can add to your classes. Thisfeature makes it a breeze to see at aglance in the debugger what particulardata is contained within each classinstance. Today, the default debuggerdisplay doesn’t show any useful infor-mation for your class. The onus is onyou to drill down into your class tofind the data you are looking for.Wouldn’t it be much easier if thedebugger just displayed this data upfront?
The solution is to add a DebuggerDisplayAttribute to your class to makethe debugger show you something youconsider useful about your class. Forexample, if you had a Citizen class thatheld the honorific and name informa-tion, you could add a DebuggerDisplayAttribute:
FIG. 2: DEBUGGER OPTIONS
30 • Software Test & Performance SEPTEMBER 2008
.NET TOOLBELT
[DebuggerDisplay(“Citizen Full Name = {_honorific}{_first}{_middle}{_last}”)]
public class Citizen{
private string _honorific;private string _first;private string _middle;private string _last;public Citizen(string honorific,
string first, string middle, string last){
_honorific = honorific; __first = first;_middle = middle;_last = last;
}}
Now, when instances of the Citizenclass are instantiated, the debugger willshow the information the way theDebuggerDisplayAttribute on the classdirects it to. To see this, instantiate twoCitizens, Mrs. Alice G. Jones and Mr.Robert Frederick Jones, like this:
Citizen mrsJones = newCitizen(“Mrs.”,”Alice”,”G.”,”Jones”);
Citizen mrJones = new Citizen(“Mr.”,“Robert”, “Frederick”, “Jones”);
When this code is run under thedebugger, the custom display is used, asshown in Figure 1 (page 28).
It is nice to be able to quickly see thepertinent information for classes as you
debug them. But the more powerfulpart of this feature is the ability for yourteam members to quickly understandwhat this class instance holds. Thispointer is accessible from theDebuggerDisplayAttribute declaration,but any properties accessed using thethis pointer will not evaluate the prop-erty attributes before processing.
Essentially, if you access a propertyon the current object instance as part ofconstructing the display string (assum-ing that property has attributes) it willnot be processed, and therefore youmay not get the value you thought youwould. If you have custom ToString( )overrides in place already, the debuggerwill use these as the DebuggerDisplayAttribute without your specifying it, pro-vided the correct option is enabledunder Tools/Options/Debugging menuitem in Visual Studio 2008; see Figure 2(page 29).
By using the event log mechanismbuilt in to Windows, you can keep trackof issues that occur while your applica-tion is running in a production envi-ronment. You now know how to createand use a class to manage event logs aswell as to write data (events) to one ormore event logs.
You’ve also learned a mechanismthat allows an administrator or otherperson to be notified as highly criticalevents are occurring in a system, suchas a network connection going downor an attacker trying to break throughyour application’s defensive perimeter.
And finally, you’ve seen a cool way tomake debugging much faster by takingadvantage of custom debugging dis-plays—introduced in the .NET Frame-work v2.0—that you can use to auto-matically display the relevant informa-tion about your classes within thedebugger window.
So instead of spelunking through allthe various items within your class,searching for information while debug-ging code, you can have the pertinentdata contained within your object auto-matically bubble up to the top level with-in the debugger window. You’re in goodshape to take better advantage of the.NET Framework for debugging. ý
REFERENCES• This article and its code have been adapted from “C#
3.0 Cookbook,” Third Edition, written by StephenTeilhet and Jay Hilyard and published by O’Reilly(ISBN: 0-596-51610-X). Some of the code has beenmodified slightly to fit the context of this article.
• Download the source code for Listing 1 at: stpmag.com/downloads/stp-0809_teilhet.zip
Without oversight, software projects can creepout of control and cause teams to freak. But withSoftware Planner, projects stay on course.Track project plans, requirements, test cases,and d efects via the web. Share documents, holddiscussions, and sync with MS Outlook®. VisitSoftwarePlanner.com for a 2-week trial.
scripts and test data. Script execution is frequently support-ed by software, often called a test automation framework.The framework is a common structure into which you plugin scripts and data from independent test automationtools. It is up to script developers to decide how to organ-ize test data and how the scripts read them. Meanwhile, theease of test data manipulation and maintenance is one ofthe key aspects of framework viability.
This article presents an approach to test data man-agement derived from a small number of simple datadesign rules. I have successfully implemented this archi-tecture for test data management at Xerox and othercompanies. Intuitive design and easy access to test datafor modification and maintenance allow significantimprovements to application test coverage.
Global and Local Test DataRelated test scripts are usually assembled into logicalgroups, called test sets, to cover a specific functional areain the application under test. A test set defines a list ofscripts that have to be executed by the test automationframework in a particular order as a batch. Popularexamples of the test sets are the smoke test set and theregression test set.
A script can belong to multiple test sets and run withdifferent instances of test data. The automation frame-work extracts copies of the scripts (along with the asso-ciated data files) from the script repository and runs
www.stpmag.com • 31
By Vladimir Belorusets
T est automation architecture defines how tostore, reference, group, share, and reuse test
Rules That Give RepetitiveTests A Global Dimension
Vladimir Belorusets is SQA manager for DocuShare at Xerox.
Construct A DataFramework ForSeamless Testing
32 • Software Test & Performance SEPTEMBER 2008
them on distributed hosts in asequence defined by the test set.
Test data can be classified accord-ing to scope. Global test set data areshared and visible to all scripts withina test set, and local data are visibleonly within the originating script.Global test data usually represent theconfiguration parameters, such asserver name, starting Web page URL,and other items, which are commonto all scripts. Figure 1 illustrates theorganization of the test sets.
Every test automation frameworkarchitect should answer the follow-ing questions:
• How will we implement globaldata?
• How will we change the script’slocal data?
• Will the local data be overwrit-ten for every new test set?
• If we want to rerun the scriptlater with data from a previoustest set, do we need to reenter it?
• How can we preserve the origi-nal test data?
• Do we need to keep multipleinstances of the data files foreach script?
Successful adoption of a testautomation framework depends onhow effectively it supports the easeof test data access and modificationand avoids conflicts of overwritingscript data in the different test sets.
Design RulesBased on industry practices, I havedeveloped a list of six general designrules that have proven to be efficientfor organizing test data within theframeworks. These rules should beconsidered as functional require-ments when developing test automa-tion frameworks in-house or whenevaluating commercial frameworks.
Here are the rules and the advan-tages that they bring to the testautomation framework users:
Rule 1: Test data must be separatedfrom the test scripts.
Advantages. This is the most fun-damental principle in code designapplicable to any code including thetest scripts. You will not believe howmany times I have seen this rulebeing violated. Because of that, thecode had to be rewritten later withthe missing deadlines. Once the pro-gram is debugged and released youshould avoid code change unless it’sabsolutely necessary. Any code mod-ification is error-prone.
If the data are hard-coded, youmay change them in one place andforget to change them in the otherplaces. Another reason is code inter-nationalization; all human-readablestrings in the code should be repre-sented by the vari-ables and storedin the separateresource files. Tochange the locale,all you need to dois change the ref-erence to the newlocale directory,and no code edit-ing is required.The main advan-tage of this rule intest automation isthat the samescript can be
reused, without changes, for testinga different functionality in a productjust by modifying the test data.Separating the data from the scriptswill also significantly reduce thetotal number of scripts that must besupported.
Rule 2: Test data should be presented intables.
Advantages. Presenting data intables facilitates design of data-driv-en tests. A data-driven test is a tech-nique that allows one automationscript to implement many test casesby iteratively reading through a datatable one row at a time, using thevalues in that row to drive the test.
Rule 3: Data tables should be providedin external files and be easily availablefor review and modification to the testcase consumers.
Advantages. I divide the users ofthe scripts into two broad categories:test automation engineers and sub-ject matter experts. Users belongingto the latter category usually do nothave programming skills, but theydo have a deep understanding of thebusiness that an application undertest automates. They know whichdata to apply to verify the subtletiesof the application’s functionality. If ascript is properly designed, the sub-ject matter experts should be able toexecute it easily without looking inthe code. All they need to do is mod-ify test data. If it takes considerableeffort to find the data file navigatingthrough the tens of subdirectories,then the test automation frameworkis inefficient and unusable.
Rule 4: Global data common to allscripts in a test set must be separated
DATA STRUCT
FIG. 1:TWO DIMENSIONS
FIG. 2: SCRIPT/DATA CATALOG
SEPTEMBER 2008 www.stpmag.com • 33
from the local script data.Advantages. If script data files in a
large test set contain both globaland local data, it takes more time tomodify the same data in all datafiles. This procedure is inefficientand error-prone. If we instead have acentral repository for global data, weneed to do the modification onlyonce per test set and it will be imme-diately propagated to all the scripts.Everyone will be able to execute testsets in their environment just bymodifying global settings andreusing the original local data with-out changes if they are still valid fortheir tests.
Rule 5: Local test data should beuniquely associated with both the testscript and the test set that contains thescript.
Advantages. An association with thetest set is necessary to run the samescript in the multiple test sets but withdifferent data. Note in Figure 1, Iassigned two dimensions to all localdata: one index for the script, and theother one for the test set.
Rule 6: Local data for each test setshould be separated and coexist withinthe same data file.
Advantages. To run the script, thetest automation framework extracts thedata file provided by the script devel-oper from the script repository. Tochange data for the subsequent test set,you need to overwrite that originaldata file. This creates a data conflict.One possible solution is to have onescript and multiple data files corre-sponding to every test set. Such anapproach creates overhead for data filemaintenance and for dynamically map-ping the right data file to the script. Ifwe have only one data file with coexist-ing local data for every test set, it sim-plifies data inventory and search. Datacoexistence also protects against dataconflicts and overwriting.
The following is an example of aflexible and efficient architecture
for test data organizationthat abides by the presenteddesign rules. The solution isgiven for Windows, but thesame ideas can be applied toany platform, and the datadesign rules are platformindependent.
Data OrganizationIn this implementation, each scriptis associated with only one data file.It uses an Excel spreadsheet for pre-senting test data in the tables follow-ing Rule 2. According to Rule 3,these data files should be easilyaccessible by subject matter experts.
This raises another funda-mental question on how toeffectively group scripts and their data files for easylocation.
When an engineer beginsa test automation project, thefirst task he is faced with ishow to arrange scripts anddata. To conduct this task, I
promote a practice of creating theapplication’s Functional Decomp-osition Model, where all of the appli-cation’s functionality is decomposedinto a hierarchy of functional areasand subareas. This structure is thenmapped to a directory tree, whichstores test cases and scripts under thematching functional area directories.The subject matter experts do notneed to review the individual scripts,but they do need to know what thescript does and where its data arelocated.
Easy access to data files can beprovided by creating a script/datacatalog in MS Excel using its “Groupand Outline” feature (Figure 2). Foreach script record, there is a link tothe corresponding data file that youcan open, modify, and save directlyfrom the spreadsheet.
Each data file has multiple work-sheets with one mandatory work-sheet, named “Default” (Figure 3).That worksheet contains theoriginal test data providedby the script developer.
All worksheets have thesame structure: the first rowcontains headers (parameternames), and all other rowscontain test data values.Multiple rows indicate a data-driven test. To modify the
original local test data, you shouldcreate a worksheet with the test setname and enter the new test datathere (Figure 4). This way, local datafor different test sets are associatedwith the test set names (Rule 5) andcan coexist in one data file (Rule 6).
We use the following simple algo-rithm for accessing local test data.Each script reads the associated datafrom the worksheet that has the testset name the script belongs to. Ifthere is no such worksheet, the dataare automatically read from the“Default” worksheet. If subject mat-ter experts want to modify the origi-nal test data, all they need do is cre-ate a worksheet with the test setname in the data file.
Global Data ImplementationLet’s look at a typical situation.You’ve assembled a test set of 100scripts developed by others. With thescripts, you inherited their test datafiles. Every script in the test set usesServer_Name as an input parameter.You are comfortable with all test dataexcept that the server’s name in yourtest environment differs from theone in the original data files. Howcan you avoid the error-prone workof editing all 100 data files to changethe server’s name to the same value?The solution is global data for the testset (Rule 4).
There are three options for where tostore the global variables: Windows reg-istry, environment variables and files.One convenient way to implementglobal test data is through the environ-ment variables that one can easily view
DATA STRUCT
FIG. 3: FIRST TEST SET
FIG. 4: SUBSEQUENT SETS
FIG. 5: GLOBAL VALUES
34 • Software Test & Performance SEPTEMBER 2008
and edit with theSystem tool from theWindows ControlPanel. Every test setstarts with a Setupscript that createsthe environmentvariables for all glob-al data within thetest set. Global vari-able values aredefined in the SetupExcel data file.
Test set name isone of the global variables whosevalue is used by every script to deter-mine which worksheet contains thescript’s local data (see an example inFigure 4). In some test automationframeworks, such as HP QualityCenter, the script can use the frame-work API to get the test set name itbelongs to. Here is a simple solutionthat is applicable to any framework.This algorithm can be easily extendedfor more complex cases.
Unlike other script data, the Setupdata file has only one worksheet, withcolumns named after the global vari-ables. Two columns, “Order” and“Test_Set,” are mandatory. The “Order”
column describes the sequence inwhich the test sets are assembled to beexecuted by the test automation frame-work. The “Test_Set” column containsthe names of the test sets. Each row inthe worksheet presents the values ofthe global variables for one test set(Figure 5).
To run a sequence of test sets, weneed to define one more environmentvariable, TS_CURRENT, and manuallyassign it initial value 1. The Setup scriptin the test set Smoke from Figure 5reads TS_CURRENT and creates envi-ronment variables with values from therow whose Order number equates toTS_CURRENT. Now, all scripts in the
test set executedafter Setup haveaccess to globaldata. Since theTest_Set variable isdefined, the indi-vidual scripts alsoknow from whichworksheet to readtheir local data.
We end every testset with a Resetscript that deletes allenvironment vari-
ables for the current test set and incre-ments TS_CURRENT for the next testset. When the Regression test set starts,the value for TS_CURRENT will be 2.
Thus, in our implementation, everytest set contains the scripts presentedin Table 1.
By using this test data architectureand modifying it for your environ-ment, you will be able to better andmore efficiently manage your test datain a way that lends itself to standardi-zation and reuse. In my experience,managing data in this way—using atest automation framework—makesmy life as a tester much easier and farmore enjoyable. ý
DATA STRUCT
TABLE 1: UNIVERSAL SCRIPTS
Scripts Description
Setup Sets global variables for the test setsStart Starts the application under testAutomated test cases Scripts to exercise the functionality of the
application
Finish Closes the applicationReset Deletes global variablesClose Tool Releases the automation tool license once all test
sets are completed (included in the last test set)
OCTOBER 28–30, 2008 • HYATT REGENCY RESTON • RESTON, VA
REGISTRATION NOW OPEN!Sign Up Online To Receive the Latest Updates!
EclipseWorld: the first and only technology conference dedicated 100% to Java development using Eclipse.• Become a better Java
developer—today!
• Stay on top of the newest trends and developments in the Eclipse ecosystem
• Learn how to use Eclipse to build desktop, server and Web Java applications
• Understand how to extend and customize Eclipse for your organization
• Explore the most popular free and commercial Eclipse-based tools
• Gain knowledge that you canbring back to your departmentand team
www.eclipseworld.net
A BZ Media Event
w w w . c o d e - m a g a z i n e . c o m
REGISTER BY SEPT. 26 FOREarly-BirdSavings!
SAVE $200!
PLATINUM SPONSORS GOLD SPONSORS SILVER SPONSOR MEDIA SPONSORS PRODUCED BY
EW08ad STP Sept 8/4/08 11:00 AM Page 1
envelope on the best practice of testingearly and testing often. Keeping pacewith the quick development turn-around and shorter time to market, andbeing adaptive to late changes inrequirements itself, requires effectivemanagement of quality process.
The use of traceability of test arti-facts—test cases, test defects, test fix-tures—mapped to the requirements—needs, features, use cases, supplemen-tary requirements—as a QA schedulingand planning tool, though mentionedin passing and claimed to have beenpracticed, has been largely overlookedby the industry. This article explores astudy of software that involves iterativeapplication development practices,bringing traceability as a QA manage-ment tool into focus.
Many software methodologies havecome to be classified under the hood of“Agile Methodology.” These methodscame about in response to the need foradaptive design and development tech-niques as opposed to predictive tech-
niques to meet the changing or evolvinguser requirements and needs. Softwaredevelopment is not a defined process, atthe very least because the main inputsto the process activities are people.
Agile methods are people-orientedrather than process-oriented. Agile meth-ods are iterative. Iterative developmenttechniques adapt to changing require-ments by focusing on the product devel-opment with “good enough” require-ments. However, there’s still an elementof planning involved per iteration wherea subset of the required features are bro-ken down into tasks, estimated in detailand allocated to programmers.
Use case modeling is a popular andeffective requirements managementtechnique. Use cases capture most of thefunctional requirements of a softwaresystem. They describe the user goals andthe sequence of interactive steps toachieve the goal. Use cases are widelyadopted in iterative software develop-ment methodology, such as the unifiedprocess and other agile techniqueswhich are iterative or evolving in nature.
Verification techniques to derive testcases from use cases are well established.So planning testing cycles entails effectivetraceability of test artifacts to require-ments planned for the iteration. Thoughthe emphasis in agile development is onpeople rather than on process, on work-ing software over comprehensive docu-
SEPTEMBER 2008 www.stpmag.com • 37
Pho
togr
aph
by K
eith
Bin
ns
The authors are project managers at IntellisysTechnology, an IT services company based inOak Brook, Ill.
Uncover Buried
Quality By
Digging Up
The Hidden
Traceability Of
Your Artifacts
By Venkat Moncompu andSreeram N. Gopalakrishnan
Rapid prototyping and de- velopment techniques com-
bined with agile developmentmethodologies are pushing the
OCTOBER 28–30, 2008 • HYATT REGENCY RESTON • RESTON, VA
REGISTRATION NOW OPEN!Sign Up Online To Receive the Latest Updates!
EclipseWorld: the first and only technology conference dedicated 100% to Java development using Eclipse.• Become a better Java
developer—today!
• Stay on top of the newest trends and developments in the Eclipse ecosystem
• Learn how to use Eclipse to build desktop, server and Web Java applications
• Understand how to extend and customize Eclipse for your organization
• Explore the most popular free and commercial Eclipse-based tools
• Gain knowledge that you canbring back to your departmentand team
www.eclipseworld.net
A BZ Media Event
w w w . c o d e - m a g a z i n e . c o m
REGISTER BY SEPT. 26 FOREarly-BirdSavings!
SAVE $200!
PLATINUM SPONSORS GOLD SPONSORS SILVER SPONSOR MEDIA SPONSORS PRODUCED BY
EW08ad STP Sept 8/4/08 11:00 AM Page 1
mentation, and responding to changerather than following a plan, a QA man-agement process needs to remain nimbleto the changing and evolving needs andrequirements. This is precisely where thetraceability matrix can be leveraged toperform optimal QA activities that givethe most value.
Agile TestingAgile QA testing involves closer andtighter feedback within each cycle ofiteration, defining levels and types oftesting in each cycle of iteration. Howcan planning of requirements testingwork with iterations? User needs in anagile process are defined by a story(sometimes captured as use-cases andfeatures) planned to be implementediteratively. Work breakdown for devel-opment (in iterations) of these use-cases and features is defined in terms oftasks. As a logical extension, the QAeffort can also be tasked for planningand scheduling purposes.
The scope of testing in iteration isusually a set of unit and (build) accept-ance tests to verify the requirementsand features planned for the iteration.The need for constant and continuousregression testing is warranted as thesoftware construction evolves and bugsget fixed, just as it scopes the featuresand use cases that go into the currentiteration or development cycle. Itera-tions, being time-boxed, do not wait forthe exit or entry criteria to be met norare they predefined.
Agile testing leaves a lot of room forexploratory and ad hoc testing that isn’tnecessarily captured in the use cases
and/or features (remember “just enoughdocumentation to develop software”). Inagile methodology, the emphasis is onsoftware construction rather than docu-mentation, unlike the traditional water-fall model of software development. Thetwo main premises of being agile are:
• The ability to welcome and adaptto requirement changes later inthe development life cycle.
• Testing often and testing early (initerative cycles).
Apart from these two basic tenets,the other difference from a waterfallmodel is that the requirements arenever really “frozen” in developmentsuch that it becomes an entry criterionfor the software construction phase.Prototyping is the key aspect of agiledevelopment techniques that help ingetting user feedback early and contin-uously in the development life cycle.This reduces the “dreaded integrationphase” late in the software developmentphase, minimizing the risk of fallingshort of user needs or ending up withunfulfilled requirements.
User acceptance tests serve as exit(or acceptance of the build) iterationcriteria and to measure progress (orburn rate) of the project. So, in tech-niques such as feature-driven develop-ment and test-driven development, themapping of the features and use casesto test cases—traceability—serves as avaluable tool to effectively plan andschedule testing just as features andcards are used to plan development initerative cycles.
And just as use cases provide a userperspective for developers and design-ers; testers have the onus of ensuringthe software meets the user require-ments adequately. This can be effectivelyachieved by mapping test artifacts torequirements that are modeled as usecases and testing the intended function-ality independently.
Scheduling and Iteration PlanningThe agile techniques for software devel-opment uses tasks in place of workbreakdown structures referred in tradi-tional project planning tools. To effec-tively understand the use of tasks andplanning of effort from a QA perspec-tive, it is useful to breakdown the QAwork product into iterations based onthe features and functional specifica-tions that are planned for the iteration.
Traceability matrices provide a con-venient way of ensuring the intended
features are tested and verified. Thisfurther provides valuable feedback tothe project team (including the end-user stakeholder) about the softwareconstruction progress. To be effective,therefore, it is important that thetraceability is mapped thoroughlymaking the features provided trans-parent to all stakeholders. And for theQA manager, it provides a good substi-tute from “traditional” selection crite-ria for regression and acceptance tests.It plays an important role in providinga basis for statistical information suchas burn rates and velocity for the teammanagement.
Multi-Dimensionality of TraceabilityFor the sake of clarity, a case study show-ing traceability to map test cases relatingto use cases and features of a student reg-istration system is discussed here.Consider a student-course registration sys-tem. It should have the following features:
• Users (Students, Registrars andProfessors) should be able to regis-ter with the system.
• Users should be able to create,update or delete their profiles andpreferences.
• Users (Students) should be able toregister for classes and securely payfor courses.
• Users (Students, Registrars andProfessors) should be able to viewthe student transcripts based onaccess restrictions.
• Users (Registrars and Professors)should be able to create courseofferings and the system shouldprovide a catalog of courses.
As with any system of moderate com-plexity, the set of requirements cannever be really termed “complete.”Therefore the process should be adap-tive to changing user needs. But for thesake of this example, these require-ments will suffice. A set of possible usecases identified for the system are:
The use case descriptions in Table 1(see References) define the main suc-cess scenarios of the system. However,not every use-case scenario ends in suc-cess for the user. While elaborating theuse cases using the descriptive text tocapture these alternate paths, newaspects of the systems come to lightwhen exceptions are encountered (non-happy path behavior of the system isbeing captured).
Spence and Probasco refer to themas overloading the term requirements, a
TRACEABILITY DIG
38 • Software Test & Performance SEPTEMBER 2008
SEPTEMBER 2008 www.stpmag.com • 39
common point of confusion withRequirements Management. These maynot be clear from the user needs andsystem features captured, but they are avital and essential aspect of the systembehavior. To ensure that the systemmeets these requirements and for cov-erage to be effective, these have to beelicited clearly and traced completely.Alternate paths may also be capturedusing a usability (scenario) matrix asseen in Table 2 (see References). Whilethe use cases are mapped against fea-tures (or cards) that are planned for theiteration, so can the use cases, the use-case scenarios that stem from these andso on, cascading to the test cases (andtest artifacts).
Note that the usage of the applica-tion flow, even though captured, couldend up varying the application flowbased on the data. For example, a stu-dent logging into the system would beprovided with a different set of featuresand screen flows compared to a profes-sor or a registrar who uses the system.Supplementary requirements corre-sponding to the architectural require-ments for the system cannot be mappedunless captured separately. Theseremain outside the functional require-ments modeled by the use cases as seenin the Table 3 (see References).
A sample list of business require-ments that have to be followed couldbe summarized as:
• BR1: Students without pre-requi-sites defined for the course theyseek to enroll in should be pre-vented from trying to register, i.e.check out the course.
• BR2: Students checking out cours-es have to register within two work-ing days from the time of initiationcheckout, otherwise the seats shallnot be guaranteed and released tothe general pool.
• BR3: If the courses are outside ofthe student’s planned Major depart-ment, then such courses shouldrequire an advisor override and can-
not exceed two courses outside theMajor program of study.
In the above case, when the map-ping of the test case flows across func-tionality is carried, it becomes evidentthat the granularity of detail falls shortwhen mapping the coverage of the testflows against the business rules as canbe seen in Table 4 (see References).
Based on the feature set as set out,it’s possible that any one of the flowsused to ensure coverage of businessrequirement 1 could as well serve forbusiness requirement 2. However, oncloser scrutiny, the test-case flow thattests the unhappy path scenario of busi-ness requirement 2 requires a furtherelaboration of the test flows against fea-ture set. Such gaps and inadequacieswill come to light in a traceabilitymatrix that is not granular and, conse-quently, the test coverage falls short.
Tracing every business, non-busi-ness and non-functional requirementto test cases and scenarios shouldincrease the confidence and coverageof testing and QA activities that can beperformed. The usability flows andconcrete test cases that cover therequirements and needs can be formu-lated, and with each iteration, targetedtest cases could be identified to be runor executed to address within the spe-cific build. Traceability is really multi-dimensional, and to be effective QAartifacts, they have to transcend thevarious phases of the developmentprocess—initiation, elaboration, con-struction and transition (see Figure 1).Further, it has to be a “living” artifact,one that is updated with each iteration.
Within iterations, a set of acceptanceand regression tests have to be sched-uled and performed to meet the exitcriteria. Features and stories (in theform of cards) are planned in iterationsin an agile methodology. With traceabil-ity matrix and mapping of the test casesto features, use cases and defects, opti-mum test planning assuring the soft-ware quality within each build/release
becomes effortless and convenient.By establishing effective traceability
matrices, the tool helps to answer some ofthe following questions apart from achiev-ing the traceability of requirements todesign and construction of the software:
Apart from the base code smoke andbuild-acceptance tests, which test casesshould be selected to run for the cur-rent build: Verify fixed defects orregression suite for the current fixes?
What impact does change in a spe-cific set of non-functional and function-al requirements have on the QA testingprocess in arriving at test estimates?
How can defects identified bemapped to the requirements that theiteration was scoped to achieve?
What surround testing and re-testinghave to be carried out for validationbefore the defects can be closed out ornew but related ones identified?
What change requests were broughtabout by the most recent build or itera-tion, and what impact on quality doesthis new change entail?
Establishing and maintaining trace-ability provides a hidden but valuablebenefit—one of serving as a tool forplanning the testing tasks in the itera-tion during iterative development.Traceability also establishes trackingback to the exact requirements beingimplemented in the iteration improvingcoverage and confidence in the qualityprocess. This is of greater significance inagile projects where requirements docu-mentation isn’t complete as require-ments continue to evolve with eachbuild or iteration. Agility ensures theprocess (and the product) is adaptive tochanging requirements and using trace-ability for QA activities ensures that ver-ification keeps up with these changes. ýREFERENCES• Tables 1 to 4 can be found at www.stpmag.com
/downloads/stp-0809 _moncompu.pdf• Kurt Bittner, Ian Spence, Use Case Modeling,
Addison-Wesley, 2003.• Alistair Cockburn, Writing Effective Use Cases,
Addison-Wesley, 2006.• Dean Leffingwell, Applying Use Case-Driven Testing
in Agile Development, StarEast 2005.• Dean Leffingwell, Don Widrig, Managing Software
Requirements – A Use Case Approach, Addison-Wesley, 2003.
• Jim Heumann, Generating Test Cases From UseCases, The Rational Edge, E-zine for the rationalcommunity, www.ibm.com/developerworks/rational/library/content/RationalEdge/jun01/GeneratingTestCasesFromUseCasesJune01.pdf, June 2001.
• Peter Zielczynski, Traceability From Use Cases toTest Cases, http://www.ibm.com/developerworks/rational/library/04/r-3217/, 10 Feb 2006.
• Bret Pettichord, Agile Testing What is it? Can itwork?, www.pettichord.com, 2002.
• Ian Spence and Leslee Probasco, Traceability Strat-egies for Managing Requirements with Use Cases,Rational Software Corp. white paper, 1998.
TRACEABILITY DIG
Establish Traceability
with UCs
Inception Elaboration Construction Transition Inception Elaboration
IdentifyUsability
flows
Map SRand
Rules
Identify &execute
test cases
UpdateTraceability
FIG. 1: ARTIFACT EXCAVATIONS
40 • Software Test & Performance SEPTEMBER 2008
Like other companies in themidst of developing a newapplication, Emergisoft, anArlington, Texas, developerof hospital emergency roompatient management soft-ware, did its homework.
With a team of 10 in-house developers building anew generation of its Web-based, EmergisoftED hospi-tal emergency departmentmanagement application, transactionresponse times and other key perform-ance requirements had been defined inthe SLA, says Godson Menezes, the com-pany’s director of architecture and prod-uct engineering. A test plan had beendeveloped, and a test environment that“closely simulated an actual productionenvironment” was called upon to makesure all aspects of the system performedproperly,” Menezes says.
Yet when it was put into full produc-tion, performance of the system, builton a foundation of Microsoft .NET, C#,ASP.NET, COM+, IIS 6, Windows Server2003 and Oracle, slid well below thethreshold of acceptability, a seriousproblem when people’s lives literallyhang in the balance.
It’s a common problem. Lookinggood on paper and performing well,even in the most rigorous of test environ-ments, are no guarantee that post-roll-outperformance will meet SLA targets.
“Look at how data is being accessedand stored,” says Dorota Huizinga, for-mer associate dean of the College ofEngineering and Computer Science atCalifornia State University, Fullerton.“Reorganize the database, then exam-ine what is being cached and what canbe cached. Even performing I/O from adesignated part of a disk with the fastestaccess can make a difference.”
Inefficient data I/O and malformeddatabase queries are the culprit far
more often than insuffi-cient bandwidth or otherhardware-related causes,Huizinga says.
But even if SQL queriesare optimized, the verydata being extracted canlead to cumulative per-formance degradation.
At one of Wall St.’s best-known brokerage firms, anew customer service appli-
cation started out well but graduallybogged down with each subsequentquery, leading to screen refresh timesexceeding two minutes. With predeploy-ment testing uncovering no apparentperformance problems and plenty ofbandwidth available, profiling toolszeroed in on data handling followingqueries and the efficiency of the userinterface as the key suspect areas, saysWalt Sully, a senior manager and experton software development methodology attechnology solutions provider Axispoint.
“The pages were designed to pre-loadwith drill-down detail that could berevealed by clicking on a tree-view con-trol in the user interface,” he says. Theproblem was that as the database grew,so too did query process time and thetime needed to assemble the extractedinformation, resulting in large streamsfor each screen preload.
The brokerage scenario highlightsthe inability of even severe predeploy-ment testing to perfectly emulate thedemands of production. “The transac-tion levels that an online retailer sees inthe Christmas season may exceed whatwas predicted, and the resulting systemsdesign may be deficient,” says MarkEshelby, product management directorat Compuware. He recommends atriage approach that looks not just atthe behavior of individual systemcomponents–-both hardware and soft-ware–but also at their interaction.
“Successful predeployment testing ofindividual processes and componentsmay not reveal deficiencies in the hand-off from server to network, network touser interface, or something else.”
The wrong approach, says RichYannetti, longtime test manager anddirector of delivery at Technisource, isblindly adding hardware. “Hardware isrelatively cheap and it’s not that hard toaddress performance issues by buyingbigger, faster servers with more memo-ry,” he says. “But if they already have 20severs and don’t realize that one is han-dling half the traffic, the analysis needsto be based on watching where packetsgo and doing load balancing.” Onlyrarely are performance problems relat-ed to insufficient bandwidth, he says.
In a post-deployment troubleshoot-ing scenario, most application testexperts recommend assembling a triageteam of experts who can examine dif-ferent aspects of the overall system.Network troubleshooting includes ananalysis of bandwidth, congestion andlatency. An internal application thatperforms well at headquarters butdegrades in branch offices needs to fol-low packets as they are assembled andtraverse the network in either directionthrough switching components andcommunications links. Likewise, a serv-er analysis will determine if perform-ance is CPU-, memory-, or disk-bound,although, most experts agree, the prob-lems most often lie elsewhere.
Examining application perform-ance with tools that track the con-stituent components of a completetransaction is crucial, says Eshelby. “It’sessential to see how each SQL state-
Best Practices
Joel Shore
Joel Shore is a 20-year industry veteran andhas authored numerous books on personalcomputing. He owns and operates ResearchGuide, a technical product reviewing and docu-mentation consultancy in Southboro, Mass.
The .NET Result Of Post-Deployment Testing
ment or HTTP request performs.” DBAs trained in the optimization of
SQL calls, whether to an Oracle,Microsoft, My SQL or other database arealmost always part of any large IT opera-tion. Specialists in HTTP opti-mization are a newer breed, butare equally valuable. Even withhighly efficient databasequeries, determining which datato return first and whether theentire screen or just a portion ofit is refreshed can make a signif-icant difference.
According to research firmGartner, the simple act ofopening a file can generate upto 100 message exchanges.Place the server and customerthousands of miles apart, and a 50-mil-lisecond network latency for each ofthose exchanges adds up to five sec-onds,before any transactions take place.Add hundreds or thousands of simulta-neous users, and the result can be unac-ceptable performance that may require
structuring of queries or adding addi-tional networking resources in strategicgeographical locations.
After troubleshooting the Emergi–softED application with the aid of a
thread analysis and application analyticstool, performance degradation wastraced ultimately to a trio of factors:SQL tuning and out-of-date or missingindexes. Once the offending SQL hadbeen captured and escalated toEmergisoft’s production DBA, correc-
tions were made that resolved all per-formance issues, says Menezes.
Though Emergisoft’s performancewoes were found relatively quickly, thatis not always the case, leading to politi-
cal stress on the IT organizationand a potential loss of business.“The entire development teamhas to understand that they areall in this together, regardlessof where the actual problem is,”says Technisource’s Yannetti.“Post-deployment problem solv-ing is an area where the testmanager needs to step up andtake the lead.”
Even with functional specifica-tions that quantify the bounds ofacceptable performance and with
a test environment that reproduces theproduction environment in terms oftransaction load and simultaneous userheadcount, a post-deployment meltdowncan still occur. Get the team togetherquickly, make sure assignments are clear,and keep management informed.ý
Best Practices
SEPTEMBER 2008 www.stpmag.com • 41
•Get the team together quickly and
keep management informed.
•
Advertiser URL Page
Automated QA www.testcomplete.com/stp 10
Eclipse World www.eclipseworld.net 36
Empirix www.empirix.com/freedom 6
Gomez www.gomez.com 23
Hewlett-Packard hp.com/go/quality 44
IBM www.ibm.com/takebackcontrol/secure 20, 21
iTKO www.itko.com 43
McCabe www.mccabe.com/stp 35
Qualitest www.QualiTest-int.com 3
Ranorex www.ranorex.com/stp 34
Reflective Solutions www.stresstester.net/stp 8
Seapine www.seapine.com/optiframe 4
Software Planner www.softwareplanner.com 30
Software Test & Performance www.stpmag.com 41
Software Test & Performance www.stpcon.com 2Conference
Index to Advertisers
In the future, the value oftesting is not going to bemeasured using simplisticand meaningless measuressuch as time spent testingan area or raw bug count. Inthe future, the value of aninternal testing organiza-tion will be determined byits intellectual contribu-tions. Those include theability to design effectivetests that accurately evaluate business-crit-ical product attributes and capabilities, itscapacity to work across disciplines, part-ner with developers and program man-agers to prevent defects, and drive quali-ty upstream, and its ability to identify andthoroughly analyze potential risks andprovide important, timely, context-sensi-tive information that is relevant, andenables the decision makers to makeinformed business decisions.
All software companies face themounting challenges of long-term main-tenance costs, governmental regulationand compliance rules, ever growing secu-rity risks, increasing complexity, the needfor greater reliability and of coursedemands from the customer that the soft-ware just work intuitively!
For the past several years, commontesting approaches for commercial soft-ware relied primarily on business domainexperts and other knowledgeable users to“shake the bugs out” after the developershave a working application. This wasdone by manually testing the usabilityand the functionality of the softwarethrough the user interface and employ-ing simple record and playback tools to“automate” simulated user actions.
An intensively manual, end-user cen-tric approach to software testing mayhave been “good enough” in the past.But, as software permeates virtuallyevery aspect of our lives, it is imperativefor commercial software companies to
improve the trustworthi-ness of their solutions.
Behavioral testing is animportant testing approachfor the ultimate success ofany software solution. Be-havioral testing is extremelyvaluable in exposing usabili-ty issues, some types of userinterface anomalies, obviousdefects and occasionallymore serious problems. But,
as Boris Beizer noted, “Testing only toend-user perceived requirements is likeinspecting a building based on the workdone by the interior decorator, at theexpense of the foundations, girders andplumbing.”
We know that behavioral testing miss-es critical issues and other functionalproblems that sometimes require expen-sive hot-fixes and service pack releases tomaintain the software over a prolongedperiod of time. The research firm IDCreported, “The increased complexity ofsoftware development environments andthe cost of fixing defects in the field(rather than early in the software cycle)combine in exorbitant ways to drainincome and to hamstring businesses as aresult of critical software downtime.”
We also know that fixing functionalproblems exposed by behavioral tests thatcould have been detected or remediedsooner in the development life cycle iscostly. Research by Victor Basili and BarryBoehm proved that defects detected in atesting phase after the implementation iscomplete can be more than seven timesas expensive compared to finding and fix-ing that problem earlier. And, a study bythe National Defense Industrial Asso-ciation (NDIA) recently concluded thatmanual testing simply does not scale wellto the increasing size and complexity ofsoftware, and is unproductive in relationto the number of resources.
Of course, the overall effectiveness of
any testing approach primarily dependson the tester’s professional skills, experi-ence and in-depth knowledge of the sys-tem. But testing complex softwareshould not only validate software froman end-user perspective, but must alsoinclude more systematic in-depth analy-sis approaches. This means that testersmust embrace new challenges and adaptto the ever-changing demands of thishighly dynamic industry in order toremain competitive in their careers.Testing complex software and criticalsystems requires testers with a broaderset of technical skills and knowledgewho can look beyond the user interfaceand perform a more in-depth investiga-tion and systematic analysis of the systemearlier in the product cycle. The testersof tomorrow must be able to engagemuch sooner in the product life cycle,participate throughout the product lifecycle and expand their roles beyond glo-rified bug finders.
The role of the tester is shifting awayfrom that being an adversarial opponentand maturing into a partner in the devel-opment life cycle. Testers will not only val-idate design models earlier, but sometesters will also work more closely withdevelopers and use their testing knowl-edge to help write more comprehensiveunit tests and participate in code inspec-tions and peer reviews. Testers developtest automation scripts to reduce long-term maintenance costs, and some teamswill “ship” their automation to help inde-pendent developers verify aspects of theirimplementation. And testers are engagedin root cause analysis efforts to identifypatterns of defects, building tools andrefining processes for defect prevention,and to help drive quality upstream. Andyes, testers and others around the com-pany will still dog-food the productsunder development to get a feel forbehavioral usage in completing dailytasks to help with behavioral testing cov-erage. Future testing professionals needto shift the testing paradigm and breakbeyond the black box barrier! ý
Future Test
42 • Software Test & Performance SEPTEMBER 2008
Break the BlackBox Barrier!
Bj Rollison
Future Test
Bj Rollison is a test architect withMicrosoft’s Engineering Excellence group,where he designs and develops the technicaltraining curriculum in testing methodologiesand test automation.
OK
Prepared by Goodby, Silverstein & Partners 2008. All rights reserved. 415.392.0669
Released on 07.24.08Print Output at 100% Reader 3
ClientJob Number
Ad Number
Ad-ID
Job Title
File Name
File Format
Start Date
Color / Media
1st Close
1st Insertion
Vendor
Pubs
B
T
L
G
S
PeopleCreative Director
Associate CD
Art Director
Copywriter
Proofreader
Account Manager
Asst. Account Manager
Print Producer
Project Manager
Client
Production Artist
Mechanical SpecsHP TSGHPTSG-322
000085
EQUP13390000
Demand Gen - Quality Mgmt REV
HPTSG322_DGen_Qual Mgmt_REV.indd
Adobe InDesign CS3
7/23/08 9:27 AM
4/c Mag
None
September
XYZ Graphics
Better Software
Software Test and Performance
8.25 in x 10.75 in
7.875 in x 10.5 in
7 in x 9.75 in
None
1 in = 1 in
For Regions Only
Keith Anderson
None
Brian Gunderson
Brian Thompson
Shannon / Sage / Jen
Liz Clark
Leigh McDaniel
Katie Borzcik
Lisa Nicolay
None
Kirsten Finkas @ 7/24/08 10:00 AM
Notes
1735207.31.08
GSPRT
n/aOrisM17352_HPTSG322DGenQtyMgmt_REV
Cyan Magenta Yellow Black
Technology for better business outcomes. hp.com/go/quality
A LT E R N A T I V E T H I N K I N G A B O U T Q U A L I T Y M A N A G E M E N T S O F T W A R E :
Make Foresight 20/20.Alternative thinking is “Pre.” Precaution. Preparation. Prevention.Predestined to send the competition home quivering.
It’s proactively designing a way to ensure higher quality in yourapplications to help you reach your business goals.
It’s understanding and locking down requirements ahead oftime—because “Well, I guess we should’ve” just doesn’t cut it.
It’s quality management software designed to remove theuncertainties and perils of deployments and upgrades, leavingyou free to come up with the next big thing.
©2008 Hewlett-Packard Development Company, L.P.
S:7 inS:9.75 in
T:7.875 inT:10.5 in
B:8.25 inB
:10.75 in
