Predictor of Customer Predictor of Customer Perceived Software QualityPerceived Software Quality

By By Haroon MalikHaroon Malik

IntroductionIntroduction• Predict the customer’s experience within

first three months

• Quantifying the relative importance of various processes and product factors on customer experience

• Deployment issues– Usage patterns,

– Software platforms &

– Hardware platform.

IntroductionIntroduction• Come up with a model that can be easily

adapted and used at other organization with little or more tailoring.

Driving forceDriving force• Techniques already exists to predict

how many faults remains in unchanging software system, changes or module will have defect and even how much effort defect repairs will require.

• Many researches examined the effect of software contents and development process on measure of customer perceived quality.

Driving forceDriving force• Most of which ignore the

– Hardware configurations,– Software platform,– Usage patterns &– Deployment issues

• End users experience the software typically experience the quality of the entire “Solution”.

• Past researches do consider the importance of these factors but no solid work is done to validate the claim or quantify these factors.

The Software projectThe Software project• Call processing software for AVYA

telephony systems– Established product

– Seven million lines of code mostly in C and C++

• Multiple releases are in field and are being used by tens of thousands of customers.

• Used by clients whose business depend on the high availability of the product.

Capturing InteractionCapturing Interaction• Four database are considered for

capturing customer interaction measures– Customer issue tracking system

• Trouble ticket database

– The equipment database

– Change management• Sablime database

Database-1

Database-2

The ProcessThe Process• Avaya uses a tiered support process.

• Trouble Ticket

• Half of the over 4 million tickets created in 2003 are related to product analyzed in this project.

• Equipment Database– Software release

– Number of licensed ports

– 4 million systems listen in equipment databases.

Customer perceived quality Customer perceived quality modelsmodels• Business activity distraction ----

Negative effect

• To major aspect of customer perceived quality:– Impact of problem occurrence

– Frequency of problem occurrence

• Interest– Rare high-impact problems

• Equipment services outages

• Malfunction resulting in software modification

Model

Model

Model

Model

Model

Model

Mod

el

Model

Customer perceived quality Customer perceived quality modelsmodels

– Frequent low impact problems• Technician dispatches

• Customer calls

• Alarm reportsMod

el

Model

Model

Model

Model

Model

Mod

el

Model

Factors to predictFactors to predict• Factors

– Deployment issues – Usage patterns– Software platform– Hardware configurations

• Prior work examines– Software product – Development process

lack empirical validation

They are not good predictors (Static. do not vary for a single release)

Predictor Measures ?Predictor Measures ?• Value Predictors

– System size.

– Operating system

– Ports

– Total Deployment time

– Software Upgrades

• Nuisance Predictors– US of international installation

– Service contracts

– Missing configuration information

System sizeSystem size

• This predictor measures– Hardware configuration factors– Software platform factors and– Usage patterns factor.

• Introduced “LARGE” variable indicator in the model

• Small and medium systems have:– Fewer customer interactions– Few settings to configure– Fewer systems to interface with – Likely not to be involved in business critical

application requiring 7x24 uptime.– Less likely to experience and report issues.

Operating SystemOperating System• Operating system predictor measures:

– Software platform factor &

– Hardware configuration factor

• Considered – An open Linux

– proprietary

– Commercial windows

• Very small number Os systems used NT/Win2000

• Off-the-shelf operating system introduce unnecessary complexity and configuration issue.

PortsPorts• The port predictor measures

– Usage pattern factor &

– Hardware configuration factors

• The number of ports indicates how many licensed end points are supported by the system.

• Model encoded the log number of ports with log(nPort) variable

Total deployment timeTotal deployment time• Deployment time predictor

measure the deployment issues

• Anticipation: fewer customer interaction as the total deployment time increases.

I need it on Timeeee!!!!

Software upgradesSoftware upgrades• The software upgrades predictor

measures the deployment issues factor.

• Encoded the existence of upgrade using an indictor variable called “Upgr”.

• Upgrade serves to keep the machine running properly by incorporating the latest fixes and refinements to the system

• Upgrades have clearly defined purpose of making the system more stable, so expect to have that effect.

Nuisance FactorsNuisance Factors• These factors are likely to identify peculiarities

of data reporting and collection process, but not necessarily differences in underlying customer perceived quality.– US of International installation

– Service contracts

– Missing configuration information

Factors Vs PredictorsFactors Vs Predictors• Predict for each

customer (outputs):– Software defects– System outages– Technician dispatches– Calls– Automated alarms

• Using Logistic regression and Linear regression

• Using predictors (inputs):– Total deployment time

– Operating system

– System size

– Ports

– Software upgrades

• For a real world software system

ResultsResults• First fit the models to test the

relationships hypothesized previously.

• Use the models to predict customer interaction for the next major releases.

• Present full results only for two measures.

Software FailureSoftware Failure

Nuisance Factors

Most important predictorMost important predictor• Total Deployment time:

– Customers who installed the application early may have detected malfunctions that are fixed by the time later customers install their systems

– The individuals performing the installation and configuration may have acquired more experience, have access to improved documentation

– The lesson from this relationship is that customers that are less tolerant of availability issues should not be the first to install a major software release

“never upgrade to dot zero release.”

Another Important PredictorAnother Important Predictor• Operating system (software

platform, hardware configurations)

– Systems running on the proprietary OS are 3 times less likely to experience a software defect compared with systems on running the open OS (Linux)

– Systems running on the commercial OS (Windows) are 3 times more likely to experience a software defect compared with systems running on the open OS (Linux)

Customer calls

Predicting customer call traffic

• Ports and NAPorts are not available

• know from talking to the customer call center that they have estimates of how many calls a call representative can handle so our prediction of the number of calls in a month can be used to plan staffing.

• The take away is that they the predictions are accurate and can be used to plan various activities

ValidationsValidations• Accounted for data reporting

differences– Included indicator variables in the models

to identify populations (e.g. US or international customers)

• Independently validated the data collection process– Independently extracted data and

performed analyses

• Interviewed personnel to validate findings– Programmers– Field technicians

Best ContributionsBest Contributions• Identified and quantified characteristics, like

time of deployment, that can affect customer perceived quality by more than an order of magnitude

• created models that can predict various customer interactions and found that predictors have consistent effect across interactions

• We learned that controlled deployment may be the key for high reliability systems