Continuous QA

Sewit Adams (Colorado State University)

Bin Gao (Michigan State University)Jerry Neal (Indiana University)

Topics

• Overview of Project Testing• KFS Testing Methodology• The Test Environment• Lessons Learned• Test Case Examples

Objective

OVERVIEW OF PROJECT TESTING

Testing Definitions(1)

• Unit Test (IEEE Definition): A set of one or more computer program modules together with associated control data, usage procedures, and operating procedures.

• Integration Test: A test that groups modules together to test their interactions.

• Conformance-Directed Testing: Intent of testing is to demonstrate conformance to required capabilities.

Testing Definitions(2)

Unit Test:

Testing Definitions(3)

• Integration Test:

Example Test Definition (1)

Example Test Definition (2)

Why Unit Test

• Exposes problems earlier in development cycle.

• Encourages Refactoring• Forces Regression Testing

KUALI TESTING METHODOLOGY

Writing Unit Tests (1)

• Who writes unit tests– API’s owner/implementer– Pair programmer

• When to write tests– Test first– Code first

• What are the criteria– functional specifications – APIs

Writing Unit Tests (2)

• What are unit tested– Service and utility methods– Application configuration

• What may not be unit tested– Third party libraries– Database

• Follow three As pattern– Assemble: prepare test data– Act: execute a tested method against the data– Assert: compare the test results with the

expected

Writing Unit Tests (3)

• How to prepare test data?– Well-maintained data set– Test fixture– Properties file– Mock objects

• How many tests for each method?– Cover all boundary conditions

Kuali Test Framework (1)

• Objectives– Ease the unit test writing– Make the unit tests maintainable

• Implementation– org.kuali.kfs.context.KualiTestBase– Extends junit.framework.TestCase– overrides TestCase#runbare()

• Test extends KualiTestBase

Kuali Test Framework (2)

• Features– Spring context startup – Transaction management– Authorization management– Unsolved issue

• Annotations – @ConfigureContext– @RelatesTo

Kuali Test Framework (3)

• Using Kuali test framework

public class ServiceTest extends KualiTestBase {

@ConfigureContext(session = KULUSER)public void testCreate() throws Exception{

// assembly – act - assert}

@ RelatesTo(KULRNE42) public void testRoute() throws Exception{

// assembly – act - assert}

}

THE TEST ENVIRONMENT

Test Package Structure

• Parallel Package Structure– work/src/org/kuali/module/someModule– test/src/org/kuali/module/someModule

How To Run Unit Test (1)

How To Run Unit Test (2)

How To Run Unit Test (3)

How To Run Unit Test (4)

• Can also run tests using ant target.

Continuous Integration With

Anthill

LESSONS LEARNED

Test Code Standards (1)

• Leads to easier maintenance

• Documents code use better

• Test code is not second class!

• Follow normal coding best practices• Establish test code specific standards

– Test case naming conventions– Best Use of assert statements– Document test case (what is it testing?)– Avoid hard coding data– Keep test methods short (one “Act”)

Test Code Standards (2)

Code Coverage (1)

• In Theory, should test all services• Or even better, all public methods• But, we must meet deadlines!• Therefore, we must decide which

tests are more important to write

• Want tests with high value but low maintenance

• Determine V/M rating for possible tests

• Start with tests that bring the most reward

• As time allows, move on to tests that reward less

Code Coverage (2)

• Establish value of test with a simple rating system like low, medium, high or 1-10

• Possible criteria for determining value:– Importance of functionality being tested– Prevents reoccurrence of a bug?– Usefulness of documentation from test

case

Code Coverage (3)

• Establish maintenance rating with similar system. Lower ratings are preferred.

• Possible criteria for determining maint. rating:– Dependencies (service, external systems, …)– Fragility of test data– Test state setup (Number of setup calls,

services used, mock objects needed, …)– Time required to write test

Code Coverage (4)

Duplicate Tests

• Cause more overhead when refactoring• Provide no additional value• More developer time when a failure

occurs• Duplication easily masked:

– Testing method that encapsulates already tested methods (indirect testing)

– Multiple tests with different parameters that are part of an Equivalence Class

Continuous Build (1)

• Frequent builds (every 15 minutes or each check-in) help identify and correct test failures more quickly.

• Important to keep tests all passing at high rate:– Pinpointing new failures easier– Better chance developer who caused failure will

fix the test

• Suites and Package Tests help identify failures before build

• Overseen by QA Manager• Notices sent out when failures occur• Jira assignments made for tests not

resolved after one day• Developers encouraged to spend

some time early in day resolving test issues

Continuous Build (2)

Sharing Knowledge

• Establish ways for sharing test related information– Testing KB– External Resources– Test Reviews

• Document test failures– CVS comments– Jira

Beyond Unit Tests

• Difficult to test web layer• Many web bugs reoccur multiple

times from framework modifications• Need mechanism for doing system

tests• Developing Http Unit tests

Continuous QA

• Test standards continue to evolve• More test recipes are developed• Testing KB grows • More test utilities, base methods,

and improvements in data handling and dependencies

• Better ability to identify tests to run before check-in

QUESTIONS?