Prioritizing User-session-based Test Cases for Web Applications Testing

Prioritizing User-session-based Test Cases for Web Applications Testing

Sreedevi Sampath, Renne C. Bryce,

Gokulanand Viswanath, Vani Kandimalla,

A. Gunes Koru

University of Maryland

University of Nevada

Sreedevi Sampath, Renne C. Bryce,

Gokulanand Viswanath, Vani Kandimalla,

A. Gunes Koru

University of Maryland

University of Nevada

Demand in Secure & Reliable Web Applications

Demand in Secure & Reliable Web Applications

A web application is a set of static or dynamic web pages that are accessible through a browser over a network

Critical part of business for many organizations Must be available 24/7 Failures in this domain result in losses of millions

of dollars

A web application is a set of static or dynamic web pages that are accessible through a browser over a network

Critical part of business for many organizations Must be available 24/7 Failures in this domain result in losses of millions

of dollars

Complications for Automated TestingComplications for Automated Testing

Fix bugs and deploy a new version within a short time frame

Number of integration technologies

- third party reusable modules

- a well-defined, layered architecture

- dynamically generated pages with

dynamic context

- extensions to an application framework Changes in database information

Fix bugs and deploy a new version within a short time frame

Number of integration technologies

- third party reusable modules

- a well-defined, layered architecture

- dynamically generated pages with

dynamic context

- extensions to an application framework Changes in database information

Previous Work: Testing with User-Session-based Test

Previous Work: Testing with User-Session-based Test

Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)

Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)

Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)

Logs of actual usage data are converted into test cases to run regression tests (Spenkle et al, 2005)

Efficient at detecting faults but unscalable with larger numbers of user sessions (Elbaum et al. 2005)

Reduced user-session-based test suites on “covering all base requests” criterion (Sampath 2007)

Previous Work: Prioritizing Test CasesPrevious Work: Prioritizing Test Cases

Schedule the test cases according to some criterion to satisfy a performance goalCriteria: semantic differences between 2 programs (Binkley 1992)

coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001)fault likelihood (Elbaum 2002)length of tests for GUI-based programs (Bryce & Memon

2007)

All strategies prioritize test cases for C and Java programs.

Schedule the test cases according to some criterion to satisfy a performance goalCriteria: semantic differences between 2 programs (Binkley 1992)

coverage of requirements (Offut 1995) fault exposure potential (Rothermel 2001)fault likelihood (Elbaum 2002)length of tests for GUI-based programs (Bryce & Memon

2007)

All strategies prioritize test cases for C and Java programs.

Prioritization for Web Application Testing

Prioritization for Web Application Testing

This work Expands upon previous prioritization work to develop

strategy specifically for web application testing Empirically evaluates strategies using user-session-based

test suites Gives guidance to testers based on the results of their

empirical evaluation

This work Expands upon previous prioritization work to develop

strategy specifically for web application testing Empirically evaluates strategies using user-session-based

test suites Gives guidance to testers based on the results of their

empirical evaluation

Motivation and Approach OutlineMotivation and Approach Outline

Criterion: Length Based

Base Request Long to Short (Req-LtoS)

Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)

Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)

Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)

Criterion: Length Based

Base Request Long to Short (Req-LtoS)

Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)

Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)

Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)

Test suite prioritization uses the entire test suite for execution but the test cases are ordered based on criteria that attempts to detect faults as quickly as possible in test execution cycle

Approach: Test Case Prioritization Criteria:Test Length

Approach: Test Case Prioritization Criteria:Test Length

Order by the number of HTTP base requests in a test case

Descending order of length (Req-LtoS) Ascending order of length (Req-StoL),

where length of a test case is defined as the number of base requests the test case contains counting duplicates.

Order by the number of HTTP base requests in a test case

Descending order of length (Req-LtoS) Ascending order of length (Req-StoL),

where length of a test case is defined as the number of base requests the test case contains counting duplicates.

Approach: Test Case Prioritization Criteria:Frequency-based Prioritization

Approach: Test Case Prioritization Criteria:Frequency-based Prioritization

Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suiteSequences are considered: in terms of base requests that involve interactions between JSP and JSP servlet

pages of size 2

Test cases that cover most frequently accessed pages/sequence of pages are selected for execution before test cases that exercise the less frequently accessed pages/sequences of pages in the entire test suiteSequences are considered: in terms of base requests that involve interactions between JSP and JSP servlet

pages of size 2

Two Ways to Prioritize Frequently Accessed Sequences

Two Ways to Prioritize Frequently Accessed Sequences

All Accessed Sequences (AAS)For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite

All Accessed Sequences (AAS)For each sequence in the application, beginning with the most frequently accessed sequence, test cases that have max. occurrences of these sequences are selected for execution before other test cases in the test suite

Most Frequently Accessed Sequence (MFAS)Orders test cases in decreasing

order of the number of times that

sequence appears in the test case

Systematic Prioritization by Parameter-ValuesSystematic Prioritization by Parameter-Values

Pages that contain parameters for which users may specify valuesPages that contain parameters for which users may specify values

Criterion:Length Based

Base Request Long to Short (Req-LtoS)

Base Request Short to Long (Req-StoL)Parameter Value Long to Short (PV-LtoS)Parameter Value Short to Long (PV-StoL)

Frequency BasedMost Frequently Accessed Sequence (MFAS)All Accessed Sequences (AAS)

Parameter Value CoverageUnique Coverage of Parameter Values (1-way)2-way Parameter-Value Interaction Coverage (2-way)

Systematic Prioritization by Parameter-ValuesUnique Parameter-Value Coverage

Systematic Prioritization by Parameter-ValuesUnique Parameter-Value Coverage

The 1-way criterion selects a next test that maximizes the number of parameter-values that have not appeared in previously selected tests

The 1-way criterion selects a next test that maximizes the number of parameter-values that have not appeared in previously selected tests

Systematic Prioritization by Parameter-ValuesParameter-Value Interaction Coverage

Systematic Prioritization by Parameter-ValuesParameter-Value Interaction Coverage

The 2-way criterion selects a next test that

maximizes the number of 2-way parameter-value

interactions between pages that occur in a test

The 2-way criterion selects a next test that

maximizes the number of 2-way parameter-value

interactions between pages that occur in a test

Systematic Prioritization by Parameter-ValuesLength by Parameter-Value Counts

Systematic Prioritization by Parameter-ValuesLength by Parameter-Value Counts

Prioritize test by the number of parameter-values in a test case

Prioritize test by the number of parameter-values in a test case

Approach OutlineApproach Outline

Criterion: Length Based

Base Request Long to Short (Req-LtoS) Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)

Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)

Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)

Criterion: Length Based

Base Request Long to Short (Req-LtoS) Base Request Short to Long (Req-StoL) Parameter Value Long to Short (PV-LtoS) Parameter Value Short to Long (PV-StoL)

Frequency Based Most Frequently Accessed Sequence (MFAS) All Accessed Sequences (AAS)

Parameter Value Coverage Unique Coverage of Parameter Values (1-way) 2-way Parameter-Value Interaction Coverage (2-way)

Experimental MethodologyExperimental Methodology

Experimental EvaluationExperimental EvaluationEvaluation is based on: Fault detection rate Average percent of faults detected Execution time of the test suite

Subject Applications (test suites): Book - e-commerce bookstore CPM - create grader accounts for TA Masplas – web app for regional workshop

Evaluation is based on: Fault detection rate Average percent of faults detected Execution time of the test suite

Subject Applications (test suites): Book - e-commerce bookstore CPM - create grader accounts for TA Masplas – web app for regional workshop

Evaluation MetricsEvaluation Metrics Rate of fault detection

Average percentage of faults detected

Rate of fault detection

Average percentage of faults detected

Study is based on:

Finding most faults in earliest tests (first 10% of tests executed)

Finding 100% of the faults earliest

Experimental Evaluation (contd…)Experimental Evaluation (contd…)

Categories of fault seeded: Data Store - faults that exercise code

interacting with data Logical - code logic errors in control flow Form based Appearance - change in perceiving user’s

view of the page Links - change of hyperlink locations

Categories of fault seeded: Data Store - faults that exercise code

interacting with data Logical - code logic errors in control flow Form based Appearance - change in perceiving user’s

view of the page Links - change of hyperlink locations

Result Analysis for CPMResult Analysis for CPM

AAS is most effective technique to find most faults in the earliest test

2-way parameter value interaction is most effective technique to find all the faults at the earliest

Result Analysis for MasplasResult Analysis for Masplas

Req-LtoS - Test length based on number of base requests (descending) is most effective technique to find most faults in the earliest test

2-way parameter value interaction is most effective technique to find all the faults at the earliest

Result Analysis for BookResult Analysis for Book

1-way parameter value interaction is most effective technique to find most faults in the earliest test

MFAS and 1-way parameter value interaction are most effective techniques to find all the faults at the earliest

Summary of resultsSummary of results

Choice of the testing technique is based on the tester’s goal and characteristics of web application The 2-way parameter value interaction and PV-LToS technique is to be

used when the primary goal is to find 100% of faults at the earliest Frequency based technique needs to be used to validate the functionality of

frequently used web pages of an application

Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact

Choice of the testing technique is based on the tester’s goal and characteristics of web application The 2-way parameter value interaction and PV-LToS technique is to be

used when the primary goal is to find 100% of faults at the earliest Frequency based technique needs to be used to validate the functionality of

frequently used web pages of an application

Choice of the correct prioritization would help in finding and fixing the faults in application quickly, thus saving cost impact

Future WorkFuture Work

Consideration of costs associated with prioritization strategies

Evaluate hybrid approaches to prioritization (combining prioritization techniques)

Extension to frequency based metrics to include other metrics

Consideration of costs associated with prioritization strategies

Evaluate hybrid approaches to prioritization (combining prioritization techniques)

Extension to frequency based metrics to include other metrics

ConclusionConclusionPrioritization criteria improve the rate of fault detection of test suites when compared to random ordering of test cases