Evaluating Web Software Reliability

By Zumrut Akcam, Kim Gero, Allen Chestoski,Javian Li & Rohan Warkad

CSI518 – Group 1

Agile Development

The Agile Development Process

What is Agile? Focus on business value Collaboration within team; meetings Communication is key Sprints, prototypes, feedback Changing requirements

The Agile Development Process

How does Agile work?

How did our class use Agile?

Three Sprints

“Stand up” meetings at beginning of each class

Retrospective at the end of each sprint

Overview

Definition of Reliability

What is reliability for Web applications?

The reliability for Web applications can be defined as the probability of failure-free Web operation completions.[2]

Failure is “the event of a system deviating from its specified behavior like obtaining or delivering information”.[1]

Failure Sources

Failures are caused from the following sources:

Host, network or browser failures: computer systems, network or software failures, etc.

Source content failures: missing, inaccessible files, JavaScript errors, etc.

User errors: improper usage, mistyped URLs.[2]

Project Goal

Attempt to extend previous work on testing the reliability of websites.

Gain experience doing a research project

Sprint 1

Sprint 1 Goals

Read relevant research papers

Identify factors that may effect reliability analysis

Gather status logs

Factors That May EffectReliability Analysis

Web Workload Characteristics Byte Count

User Count

Session Count

Error Count

System to Analyze Reliability On

Reliability analysis via status logs

Variety of reliability requirements

Commercial and non-commercial

We will try to record the technologies the websites employ (Apache, DNN, ISS, PHP, ColdFusion, etc..)

Sprint 2

Sprint 2 Goals

Collect log files for calculation

Automate processes to extra data (user, session, byte, and error counts)

Convert them into excel format

Log Parser

Sprint 2 Progress

DNN Logs (10 Websites)

PHP Logs

What is DotNetNuke (DNN)

.NET version of Drupal An open source platform for building websites and web applications based on Microsoft .Net technology. Leading open source ASP.NET web content management Has been downloaded over 6 million times ~100 employees 5th Version Founded 2006

Our DNN Logs

Logs from 10 Websites Window Server (Same Server) SQL Server 2008 ~1000 unique visitors per day Logs contain

User count Limited Error count

Major Problem

The DNN Logs do not contain Session count Byte count

Alternative

Generate our own DNN logs

Sprint 3

Server Side

Technologies Used Windows XP Professional

Microsoft Internet Information Services (IIS)

Microsoft SQL Server 2008

DotNetNuke (DNN)

Generating Logs

Clients Web-Crawlers DotNetNuke Client API

Inducing Errors

Logs contain: Client IP’s Byte Counts (Uploaded & Downloaded) Time-Taken Status Code

Description of Success Status Codes

Status Code Meaning Description200 OK The request was received, understood, and

being processed.206 Partial

ContentResponse to a request for part of a document. Just the section requested is returned.

302 Found Requested resource was moved to a new location. Response includes new location.

304 Not Modified

Response to a request for a document on the condition it is newer than the version the client has.

Description of Error Codes

Status Code Meaning Description400 Bad Request The server did not understand the request

due to bad syntax401 Unauthorized Indicates that before a resource can be

accessed, the client must be authorized by the server.

403 Forbidden The client cannot access the requested resource. Possible incorrect username & password, or permissions do not allow the action.

404 Not Found The requested resource was not found at the supplied URL.

500 Internal Server Error

Indicates that the server encountered something unexpected and was unable to complete the request.

Results

Workload Measurement Facts

Server log data consisted of 23 consecutive days of data.

Page Not Found (Error 404) is the most common type of error in our logs, with 46% of total recorded errors.

Accessing forbidden data (Error 403) follows with 41%.

72 unique IPs, 33 thousand hits total, and each hit associated with about 5kb.

Status Code-Bytes Graphic

200 302 404 403 304 401 206 500 400 -

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

160,000,000

Bytes DownloadedBytes Uploaded

HTTP Status Codes

Error and Success RatiosHTTPStatus Codes*

Description

200 OK

206 Partial Content

302 Found

304 Not Modified

400 Bad Request

401 Unauthorized

403 Forbidden

404 Not Found

500 Internal Server Error

20089%

3027%

4041%

4031% 304

1%4010%

2060%

5000%

4000%

STATUS-BYTES-Server To Client

20018%

30268%

4041%

4031%

30411% 401

0%2060%

5000%

4000%

STATUS-BYTES-Client to Server

SUCCESS

ERROR

*HTTP Status Codes definedBy W3C-World Wide WebConsortium.

500-Internal Server Error Profile

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 370

2000

4000

6000

8000

10000

12000

14000

16000

"500"-Internal Server Error Byte Transferred Over Time

sc-bytescs-bytestime-taken

# of Times "500" status happened

# o

f By

tes

Tran

sfer

red

Number of Status Codes

302 200 304 404 403 401 500 206 4000

2000

4000

6000

8000

10000

12000

14000

16000

18000

16764

81906901

514 467 89 43 1 1

Status Code Counts

HTTP Status Codes

# o

f St

atus

Cod

es

Average Time Taken By Different Status Codes

200 206 302 304 400 401 403 404 5000

200

400

600

800

1000

1200

1400

691

1171

1011 100

4 8 6

252

Average Time Taken for Each Status Code

HTTP Status Codes

Aver

age

Tim

e Ta

ken

(ms)

Identify a metric to analyze reliability

The Nelson Model

R: Reliability f : Total number of failures n: Number of workload units r: Failure rate

Mean Time Between Failures (MTBF)

MTBF = n/f

Conclusions

• By Nelson Model, the site software reliability is R = 0.966, or that 96.6% of access to website is successful.

• This model also shows that MTBF=29.6 hits or the site averages one error for every 29.6 hits.

• From the number of errors chart, we can see that Server errors are very few among the other errors which shows what the reliability of the DNN server is.

Conclusions [1] J.Tian, S.Rudraraju, Z.Li, “Evaluating Web Software Reliability Based on Workload and Failure Data Extracted from Server Logs”,2004.

Our Work Previous Work[1]

23 days data 26 days data96.6% success 96.2% success29.6 hits/error 26.6 hits/error148,579 bytes per error

273,927 bytes per error

Resources

[1] Jeff Tian, Sunita Rudraraju, and Zhao Li. 2004. Evaluating Web Software Reliability Based on Workload and Failure Data Extracted from Server Logs. IEEE Trans. Softw. Eng. 30, 11 (November 2004), 754-769. DOI=10.1109/TSE.2004.87 http://dx.doi.org/10.1109/TSE.2004.87

[2] Toan Huynh and James Miller. 2009. Another viewpoint on "evaluating web software reliability based on workload and failure data extracted from server logs". Empirical Softw. Engg. 14, 4 (August 2009), 371-396. DOI=10.1007/s10664-008-9084-6 http://dx.doi.org/10.1007/s10664-008-9084-6

[3] G. Albeanu, A. Averian, I. Duda, “Web Software Reliability Engineering”,2009.