Software testing techniquesSoftware testing techniques

Software Testability

Presentation on the seminar

Kaunas University of Technology

Motivation

Poor testability

Ineffective testing

Severe penalties

What is Software Testability?

•  degree to which a software artifact (i.e. a software system, software module, requirements- or design document) supports testing in a given test context.

Software Testability …

• ... is not an intrinsic property of a software artifact • …can not be measured directly (such as software size). • … is an extrinsic property which results from

interdependency of the software to be tested and the test goals, test methods used, and test resources (i.e., the test context)

Lack of testability is bad. Why?

• A lower degree of testability results in increased test effort.

• In extreme cases a lack of testability may hinder testing parts of the software or software requirements at all.

Background

• properties of the software requirements• properties of the software itself (such as size, complexity

and testability)• properties of the test methods used• properties of the development- and testing processes• qualification and motivation of the persons involved in

the test process

Testability of Software Components

• Controllability• Observability• Isolateability• Separation of concerns• Understandability• Automatability• Heterogeneity

Controllability

•  The degree to which it is possible to control the state of the component under test (CUT) as required for testing.

Observability

• The degree to which it is possible to observe (intermediate and final) test results.

Isolateability

• The degree to which the component under test (CUT) can be tested in isolation.

Separation of concerns

• The degree to which the component under test has a single, well defined responsibility.

Understandability

• The degree to which the component under test is documented or self-explaining.

Automatability

• The degree to which it is possible to automate testing of the component under test.

Heterogeneity

• The degree to which the use of diverse technologies requires to use diverse test methods and tools in parallel.

Testability of Requirements

Requirements need to fulfill the following criteria in order to be testable:– consistent– complete– unambiguous– quantitative – verifiable in practice 

Improving software testability

• Test-driven development• design for testability (similar to design for test in

the hardware domain)

So which system is “testable system”?

Roy Osherove: “For each logical part of the system, a unit test can be written relatively easily and quickly that satisfies all the following PC-COF rules at the same time:

 – Partial runs are possible– Consistent results on every test run– Configuration is unneeded before run– Order of tests does not matter– Fast run time”

Consistent results on every test run

• The test always has to fail or pass, but never switches between modes until a bug is fixed.

What’s the problem with this code?

  public bool CanPurchase(Person p){            if (!(PersonValidator.IsValid(p))){                return false;            }             if (p.SSID != null &&                p.SubscriptionType != "Canceled"

&& p.CreditOnFile > 0){

                return true;            }             return false;        }  

Interfaces make code more testable

IValidator m_validator;public void SetValidator(IValidator validator) {      m_validator = validator; }               public bool CanPurchase(Person p) {     if (!(m_validator.IsValid(p))) {            return false;     }      if (p.SSID != null &&  p.SubscriptionType !=

"Canceled" &&  p.CreditOnFile > 0) {            return true;      }             return false;}

MyFakeValidator val = new MyFakeValidator();

val.whatToReturn = true; PersonLogic logic = new

PersonLogic();logic.SetValidator(val); Person p = new Person();bool result =

logic.CanPurchase(p);Assert.IsFalse(result);

Configuration is not needed, because…

• … ability to configure a class at runtime is an important one for unit tests.

• … if code requires external configuration before it is tested, it will take more time to create tests for that code.

• … it can also make the tests less manageable and not as easy to write.

What’s the problem here?

public bool IsConnectionStringValid() {   string connString =         ConfigurationSettings.AppSettings["conString

"].ToString();         //do some stuff         return true; }

Use virtual method

Class ConnectionHelper{public bool IsConnectionStringValid() {         string connString = getConnectionString();            //do some stuff            //...            return true;} protected virtual string getConnectionString(){            return

ConfigurationSettings.AppSettings["conString"].ToString(); }

}

Derive and override it in testable class

TestableConfigBasedClass myClass = new TestableConfigBasedClass();myClass.mConnectString = "bad string";Assert.AreEqual(false,myClass.IsConnectionStringValid())

public class TestableConfigBasedClass:ConnectionHelper {     public string mConnectString;

            protected override string getConnectionString(){                return mConnectString;            }}

Fast run time

• In a non-testable system you would find it really hard, or really time consuming to write some of the tests against objects and make them run fast.

• That’s because you’d most likely have objects that do some sort of time consuming activity using their external dependencies.

What if the IsValid method…

if (!(PersonValidator.IsValid(p))){                return false;}• Calls a web service to do its bidding? • reads rules from the database?

To make it testable :• Refactor the code to use interfaces,

OR• use a virtual method to check the validation and return a true/false

result, then override it in your “testable” class.

Order of tests does not matter (Data layer problem)

bool Insert(Person p){//insert person to database} bool Delete(Person p){//Delete person from database}most unit test frameworks cannot guarantee the order in which unit tests will be

executed

Partial Runs of tests are possible

• When you have external state you’re likely to have a problem if you don’t run all the tests in your suite, since the external state between specific tests may need to be adjusted for the next test.

• This problem is solvable, but it’s not as easy as writing simple unit tests without external state

• The whole problem of rolling back external state is that of controlling the before and after states and making sure they are the same for each test run.

• If you can’t do that easily you’re either not using a testable system design, or you’re not writing a unit test.

References

• http://www.sce.carleton.ca/faculty/yee/Improving_Testability.pdf [žiūrėta 2011.04.26]

• http://en.wikipedia.org/wiki/Software_testability [žiūrėta 2011.04.26]

• http://replay.web.archive.org/20060424002143/http://weblogs.asp.net/rosherove/articles/Design4Tesatbility1.aspx [žiūrėta 2011.04.26]

• http://www.etestinghub.com/testing_requirements.php [žiūrėta 2011.04.26]