Software Testing inSmall Teams

Don R. Gilman, P.E.: Energy Systems Lab – TEESAdvanced Gaming Systems Inc

The Presenters

Don• Professionally programming/managing IT applications since 1984• Contributed to over one million units of various computer games• Licensed Professional Software Engineer (one of about 35 in the entire US)• Member of the Software Engineering Task Force for the Texas PE Board• Manager, Software Engineering Group – Energy Systems Laboratory – TEES

Presentation Overview

1. What is testing?2. What are the levels of testing?3. Types of testing:4. Who tests?5. Case Studies

1. Unit Testing2. eCalc Automated Testing3. A Computer Game

6. Summary7. Questions

What IS Testing?

• IEEE: The process of operating a system or component under specification conditions, observing and recording the results, and making an evaluation of some aspect of the system or component.

• Boehm: Verification and ValidationVerification – are we building it right?Validation – are we building the right

thing?

Testing is NOT

• Code inspections• Design reviews• Configuration management• Bug tracking

These are, along with testing, are part of Software Quality Assurance (SQA). Together these improve the quality of

the product

Types of Testing:

ScopeUnitModule Integration InterfaceCoverage

PurposeFunctionalStress

ExecutionManualAutomated

Phases/TriggersNightly BuildsDefect RemovalChange Order

CompletionAlphaBeta

Who Tests?

DevelopersUnit testing

QA TeamModule testing, automated testing

SoftwareAutomated testing

DesignersEngineers, Analysts

UsersUnpleasant but true

Who … some thoughts

• All New Members– To train– To get fresh test paths– In Industry – not a good model– In Gaming – a great way to recruit

• Release/QA Folks– Typically Engineers or Accountants– CompSci typically make lousy testers

Case Study: Unit TestingWhat is Unit Testing?

• Lowest Level of Testing• Programmer Safety Net• Instant Feedback• Documentation

Case Study: Unit Testing Good Unit Tests:

• Tests functionality not edge cases– GOOD: 100 points damage to demon ==

death– BAD: referencing null demon throws

exception

• Capture the meaning behind the code in tests

Case Study: Unit Testing xUnit Tool:

• Unit testing tools in many languages– .NET: NUnit– Java: junit– C++: CppUnit

NUnit

Case Study: e2Calc

Case Study: e2CalcWeb Server

User Interface

Emissions Service

BDI Service

Web Server

Database Server

Reporting Service

Que M

gmt

ProjectRecording

Service

XMLSQL

Server 2000

eGrid

DB Server

CalcEngine(s)

DOE-2 v2.1EFchart

PVFChartIMT

Calc EngineWrappers

Calc Servers

WebUsers

Optional Email w results

Browser

Emissions

Jobs

BDI User

EmissionsUser

Notification Email

BTIWeather

Lookups

Project and User Data

Report

Project and User XML

Case Study: e2Calc Quick QA Information

Use this for quickly checking the emails.

Scenario Annual kWh Annual ThermsSNGFAM Express Calc 12162 158.96SNGFAM Detailed Calc 12162 158.96

MULTIFAM Express Calc 13142 221MULTIFAM Detailed Calc 13142 221

Office Express Calc 966150 7245Office Detailed Calc 966150 7245Retail Express Calc 205002 1248Retail Detailed Calc 205002 1248

Municipal 966150 7245Utility Bill* -- --

Street Lights – Design Mode* 136875(post-retrofit) --Steet Lights – Utility Bill* -- --

Traffic Lights – Design Mode* 1621(post-retrofit) --Traffic – Utility Bill* -- --

Water Supply** N/A N/AWaste Water** N/A N/A

PVFCHART High End* 805 --PVFCHART Low End* 379 --

FCHART DHW* -- 3,553FCHART POOL* -- 2827

Wind** N/A N/A

All these agree on the energy savings with Andrews old tests!!

Do not match or not found in Andrews testsClose to Andrews testsMatches with previous tests by AndrewNo previous tests to validate against

Case Study:Games

Case Study: Games

Manual TestingCoderCore team

Public Alpha/Beta TestingSometimes process andfocus areas listed on

websitesTools

Command lines for auto loggingScreen dumpsSaved gamesDebug compilesGame vs game – with new log analysis tools

Case Study: Games

Case Study: Games

Case Study: Games

Summary

• Writing automated tests is one of the most difficult programming assignments

• Design to Test• Let the computer do the dirty work• Not everyone is wired to do this – only the

successful succeed.

Maintainable Code Correct Code

Reliable Production Accurate deliveries

It’s done means it is DONE!

Resources

Software Development Magazine (www.sdmagazine.com)

CrossTalk (www.stsc.hill.af.mil/crosstalk) Better Software (www.stickyminds.com)

The Rational Edge (www.rational.com) SEI (www.seir.sei.cmu.edu) TDD, an Introduction: (

www.agiledata.org/essays/tdd.html#Figure1StepsOfTDD)

Object Mentor – TDD: (www.objectmentor.com/writeUps/TestDrivenDevelopment)

SE Books we’ve read that matter

• How to Win Friends and Influence People, Carnegie• Software Requirements, Wiegers• Agile Project Management, Jim Highsmith• Planning Extreme Programming, Beck/Fowler• Agile Project Management w Scrum Schwaber• Mythical Man Month, Brooks• Test Driven Development: By Example, Kent Beck• MS Solutions Framework Papers• Decline and Fall of the American Programmer, Yourdan• The E-Myth Revisited, The E-Myth Manager, Gerber

Questions?