INFO 630 Evaluation of Information Systems Prof. Glenn Booker

www.ischool.drexel.edu

INFO 630 Evaluation of Information Systems

Dr. Jennifer Booker

Week 3 – Quality Management and Earned Value

1INFO630 Week 3

www.ischool.drexel.eduINFO630 Week 3 2

Quality Management System• Every organization should have a Quality

Management System (QMS)– The QMS describes the quality organization

(members, duties, and to whom they report) and activities which they need to perform

– Quality Assurance, Quality Management, and Quality Control are often used interchangeably for the QMS’ name


Typical QMS Activities• Activities by a quality management organization

typically include– Process Evaluation – to determine if process is clear,

and consistent with other related processes– Process Assessment - help ensure processes meet

quality standards (CMMI, ISO 9000)


Typical QMS Activities– Process Verification – to ensure that processes are

actually being used– Product Evaluation – to inspect documents or other

products (SW, HW) to verify their quality• Think of the tag ‘Inspected by 19’ in new clothes – that’s a

product evaluation


QMS Activities• Quality control for manufacturing,

assembly, shipping, receiving, etc.• Support definition, collection, and analysis

of metrics for each organization• The quality organization should be able to

stop any activity if the product quality is inadequate (what defines inadequate?)


Quality Road Map for Software Maintenance

• An organization’s Quality Management System may be guided by some sort of Quality Road Map (high level plan)

• The Quality Road Map is the basis for each release’s Quality Plan

• The Road Map is the highest level corporate strategy for improving the product

(Warning – IBM terminology here…)


Quality Road Map• The Road Map should systematically introduce

more comprehensive quality activities and measures• Let each team develop own quality goals• Consider process, product, people, and tools; how

will each contribute to quality?– FAA uses 3-year overall plan, with one-year goals to

help meet it


Supplier Buy-in• Your quality efforts must be matched by

your suppliers• This must be stated in the Quality Plan or

equivalent• Hence the quality measurements need to

fulfill the overall goals of the organization, and define objectives for each release or product, and for major suppliers


Tracking, Measurement, Analysis

• These are key to making quality management work

• Software development must become quantitatively predictable to be regarded as a science

• Large volume of data not needed; instead, focused, accurate, useful data is better


Customer Problems

• Problems include defects, user errors, usability problems, unclear instructions, duplicates of known problems, etc.

• Usually express as Problems per User Month (PUM)– PUM = (Problems reported) /

(total license-months)


Customer Satisfaction

• Often measured with 5-point Likert scale (with optional Net Satisfaction Index (NSI)):– Very Satisfied (NSI = 100%)– Satisfied (NSI = 75%)– Neutral (NSI = 50%) – Dissatisfied (NSI = 25%)– Very Dissatisfied (NSI = 0%)


Customer Satisfaction• Note that average NSI is a weak metric, since it

hides the type of distribution• The average NSI for half Very Satisfied and half

Neutral customers looks the same as for all Satisfied– Like measuring the mean of a distribution, without

measuring the standard deviation (width of the bell curve)


In-Process Quality Metrics• Defect density during automated (machine) system

testing– Good for comparing different releases of

one product, or very similar products• Defect arrival rate

– # of defects found vs. time (e.g. week, month)– Cumulative defects found vs. time– Look for stability to decide done with testing


In-Process Quality Metrics• Clarify actual defects, vs. reported

problems, vs. backlog of defects• Actual defects measures how many known

defects have been found in the code

• Reported problems includes code defects, user errors, etc.

• Backlog shows how fast defects fixed, as well as found


In-Process Quality Metrics• “Phase-based defect removal pattern”

tracks defects during the entire life cycle• Ways to identify defects include

requirements and design reviews, code inspections, formal verification, and all phases of testing

• Want most defects found early in project (key theme)


Software Maintenance Metrics

• Focus on fixing existing problems, not changing features or requirements– Fix backlog = number of outstanding known

problems– Backlog Management Index = (# closed

problems) / (# new problems) * 100• BMI > 100 means backlog of problems

is being reduced


Software Maintenance Metrics• New, closed, and the status of problems can also

be tracked • Fix response time, measured by the

“mean time problems are open”• Percent delinquent fixes = % of problems which

exceeded desired response time– Note that this implies an active goal, since the

expected fix time has been defined


Fix Quality

• Fixes are, in general, not all perfect - they may create new problems!– Fix quality = % of fixes which are defective

(i.e. result in new problems)– Fix propagation rate = average number of

new problems resulting from a fix


What is Earned Value?

• Earned Value is a method for planning and tracking the cost, schedule, and technical accomplishments of a project

• Earned Value focuses on comparing the ACTUAL cost, schedule, and technical accomplishments to their BUDGETED (or planned) values


EarnedValue

Cost

Schedule Technical

Earned Value balances measurement of cost, schedule, and technical performance.

Earned Value


Why do we care?

• Some organizations are contractually required to use earned value to report their progress

• Payments are then based on the accomplishments claimed under earned value; hence earned value reporting claims can be legally binding!


Foundation

• Earned value requires having a written plan which describes each task, before that task is performed.

• The plan must define, for each task:– Schedule (start, stop, and calendar duration)– Cost (or earned value) it is worth, and – Earned value measurement method (see

later)


Budget Basics• The total amount of money available on a

contract is the Contract Budget Base (CBB)

• Wise managers leave a Management Reserve (MR) of about 10-15%, and budget the rest to the project’s Performance Measurement Baseline (PMB)– Hence CBB = MR + PMB


Budget Basics• Most mortals (project staff) are only aware

of the PMB, and do not know the extent of the MR– The PMB may be broken into the undistributed

budget (a more visible reserve than MR), plus distributed budget (which is the sum of all tasks’ cost accounts)

– PMB = undistributed budget + distributed budget


Budget Basics

C ontract Budget B ase (C B B ) - a ll m oney ava ilab le to p ro ject

Perfo rm ance M easurem ent Base line (P M B)M anagem ent

R eserve(M R)

Perfo rm ance M easurem ent B ase line (P M B )

D istributed Budget (m aps to W BS andm aster schedule)

U ndis tributedBudget


Earned Value TerminologyData

Element Term Acronym

Scheduled Work Budgeted Cost for Work Scheduled BCWS

Earned Value Budgeted Cost for Work Performed BCWP

Actuals Actual Cost of Work Performed ACWPAuthorized Work Budget At Completion BACForecasted Cost Estimate At Completion EACWork Variance Schedule Variance SVCost Variance Cost Variance CVCompletion

Variance Variance At Completion VAC


The Plan• The project plan produces a curve for the

BCWS (budgeted cost of work scheduled) through its planned completion. This curve is typically S-shaped, and is plotted as cost versus time (e.g. $K versus months).

• The total expected cost of the task, located at the end of the BCWS curve, is the BAC (budget at completion)


BCWS

0

0.5

1

1.5

2

2.5

3

3.5

0 2 4 6 8 10 12 14

Month

BCW

S ($

K)

BAC

Budgeted Cost of Work Scheduled


Tracking Progress• As the task is being performed, measure

the amount of money actually being expended (ACWP, actual cost of work performed)

• Keep track of the tasks which have actually been worked on, and look up how much earned value you get credit for that work (BCWP, the budgeted cost of work actually performed)


Tracking Progress• Plot the ACWP, BCWP, and BCWS over time • Notice that ACWP and BCWP can only be shown to

date, whereas the BCWS covers the entire project’s schedule– BCWS is the original project plan– ACWP is how much real money has been spent– BCWP is how much real work has been accomplished

with that money


Actual Expenses and Accomplishments

0

0.5

1

1.5

2

2.5

3

3.5

0 5 10 15

Month

$K

BCWS

ACWP

BCWP

Now

Earned Value Example


Analysis• At any time during the project:

– Cost Variance (CV, in $) = ACWP - BCWP– Schedule Variance (SV, $) = BCWS - BCWP– Schedule Slip (time) = (current date) minus (date

when current value of BCWP was found on the BCWS curve)

• Positive numbers are bad for all of these; negative is good


Projection• The ACWP curve is used to estimate when

the task will actually be completed• The projected ACWP curve is called

Estimate To Completion (ETC)• The end of the projected ETC curve is

the Estimate At Completion (EAC)– That’s the expected end of the project (in $

and schedule)


Projected Project CompletionProjected Completion

00.5

11.5

22.5

33.5

4

0 5 10 15

Month

$K

BCWS

ACWP

BCWP

ETC

BAC EAC

Notice the ACWP line becomes the ETC line after the present date


Project Completion• Compare BAC (the originally planned completion of

the project) and EAC (the current estimated completion of the project)– Projected Program Delay (schedule overrun) = Time

difference between BAC and EAC– Variance At Completion (VAC, cost overrun) = Cost

difference between BAC and EAC


Earned Value Summary

0

1000

2000

3000

4000

5000

6000

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun

T/N

Cost Variance Schedule Variance

ProjectedProgram

Delay

VAC

EAC

BAC(PMB)

CBBMR

ACWPBCWP

BCWS

ETC

Schedule Slip

T/N is Time Now

$


Earned Value Measures

PCUM

Term Formula

Percent Complete

Cost Performance Indexor Performance Factor

Checklist ActionsRatio of work accomplished in terms of the total amount of work to do.

Symbol% Done

CPI or PF

TCPI or VF

BCWPBAC

Average PerformanceBCWPcum

Duration (wks or mos) Since ACWP Began

BCWPcumDuration (wks or mos)

From Time Now to Manager's Stated Completion Date

SC or S/CSchedule Correlation

BCWPACWP

Ratio of work accomplished against money spent (an efficiency rating: Work Done for Resources Expended)

To CompletePerformance Index

or Verification Factor

BAC - BCWPEAC - ACWP

Ratio of work remaining against money remaining (Efficiency which must be achieved to complete the remaining work with the expected remaining money)

Schedule Performance Index SPI Ratio of work accomplished against what should have been done (Efficiency Rating: Work done as compared to what should have been done)

BCWPBCWS

SV

Ratio of Schedule Variance (SV) in terms of average amount of work accomplished (in weeks or months). It indicates a correlation to program true schedule condition

IEACIndependent EstimateAt Completion

BACPF

Calculation of a projected Estimate At Completion to compare with the CAM's Estimate At Completion:1) Ration of total work to be done against experienced cost efficiency2) Sunk costs added to a ratio of remaining work against weighted cost and schedule efficiencies

1)

2)BAC - BCWP.8CPI + .2SPI

ACWP +

Average Expected Performance To Finish

Average rate at which work has been accomplished since work began

Average rate at which work must be accomplished in the future to finish on the date the CAM has forecasted for completion of the work.P TO GO

PCUM

Key measures we’ll use are SPI and CPI


More Earned Value MeasuresTerm Formula

Risk Exposure Performance Index

Checklist ActionsSymbol

REPI ARWPREWP

Ratio of work accomplished against money spent (Efficiency Rating: Work Done for Resources Expended)

Risk Schedule Performance Index

RSPIRatio of work accomplished against what should have been done (Efficiency Rating: Work done as compared to what should have been done)

REWSREWP

ManagementReserve Ratio MRR

Ratio of Total Funds Available against total Estimated funds expended At Completion; or Total Funds Available against the Budgeted cost At Completionmultiplied by the Cost Performance Index

TFAEAC

= TFABAC * CPI

=


What are “good” values?PerformanceMetric Green Yellow Red

SPI 0.95 andgreater

0.90 andgreater

Less than0.90

CPI 0.95 andgreater

0.90 andgreater

Less than0.90

VAC 0.05 andlesser

0.10 andlesser

Greater than0.10

Per the “U.S. Marine Corps Acquisition Procedures Handbook,” June 1997


Projection

• Another method for projecting the end of the actual project is– Plot CPI and SPI over time– Based on the trends seen, estimate the CPI

and SPI at the end of the project• EAC(cost) = BAC(cost) / CPI• EAC(schedule) = BAC(schedule) / SPI


Example - Railroad Project• Given the following project: build 4 miles of railroad

track in 4 weeks for $4 million• Actual performance to date is: After 3 weeks, $2

million has been spent and one mile of track has been built

• Pure cost analysis would only show that the project has spent $1 million less than expected ($3M - $2M)– Is this good?


Work Accomplished and Projected

00.5

11.5

22.5

33.5

44.5

0 5 10 15

Weeks

Earn

ed V

alue

(tra

ck m

iles)

BCWS

BCWP

ETC

Example - Railroad Project


Earned Value Example - Cost

0

2

4

6

8

0 4 8 12

Weeks

Mill

ions

of D

olla

rs

BCWS

ACWP

BCWP

ETCBAC

EAC

Example - Railroad Project


Example - Status Analysis

• Today’s earned value status:– BCWS = $3M (expected expenditure & work)– ACWP = $2M (actual expenditure)– BCWP = $1M (actual work accomplished)

• Calculate– Cost Variance (CV) = ACWP - BCWP = $1M– Schedule Variance = BCWS - BCWP = $2M– Schedule Slip = (3 weeks) - (1 week) = 2

weeks


Example - Status Analysis• Earned value analysis:

– Planned $1M/week expenditure for 4 weeks– Hence planned $3M after 3 weeks (BCWS)– Earned value of only $1M (BCWP is 66% behind the

plan)– Actual expenditure of $2M to date (ACWP has 100%

overrun, given how much was accomplished)


Example - Projected Outcome

• Find EAC using SPI and CPI• Only are given one data point for status,

so use:– SPI = BCWP/BCWS = $1M / $3M = 0.333– CPI = BCWP/ACWP = $1M / $2M = 0.500


Example - Projected Outcome

• BAC was 4 weeks and $4 million• EAC is

– EAC(cost) = BAC(cost)/CPI = $4 million / 0.500 = $8 million

– EAC(schedule) = BAC(schedule)/SPI = 4 weeks / 0.333 = 12 weeks

• Hence projected cost overrun is $4M, and schedule overrun is 8 weeks


Measurement Methods

• The methods used to take credit for earned value are of key significance

• It is critical that earned value claims are accurate and consistent

• Credit, or Earned Value, for performing each task may be obtained in many different ways, depending on the type of task.


Measurement Methods• Milestone Events• Weighted Milestone• Fixed Formula• Percentage

Complete Estimates

• Percentage Complete and Milestone Gates

• Equivalent Completed Units

• Earned Standards• Apportioned

Relationship to Discrete Work

• Level of Effort


Milestone Events• Milestone events earn 100% credit when:

– The event has occurred, AND – All follow-up work is done (e.g. preparing and

distributing minutes after a formal review) AND – All action items are closed

• Until all of the foregoing activities are completed, no earned value may be taken


Weighted Milestone

• Useful for short-span tasks• Weighted budget amounts are applied to

milestones distributed across the duration of the task

• As each milestone is reached, the budget amount is earned


Fixed Formula• Useful for detailed short-span tasks• An amount ranging from 0 to 100 percent of the

task's budget is earned when the task begins• The remaining percentage is earned when the task

is complete, e.g. 0/100%, 25/75%, or 50/50%• These measures are typically used to track earned

value on nonrecurring tasks


Percentage Complete Estimates• Easiest method to administer• A "subjective" estimate of the percentage

of work completed is used for the earned value• Requires well-defined work packages and

guidelines to determine an accurate percent-complete value

• This is the biggest source of inconsistencies in earned value reporting


Percentage Complete

• For example, define the following allowable earned value for each phase of new feature development:– Analysis and design are ready for peer

review: up to 30% earned value– Successful completion of peer review: 35%– Preparation for customer review: 45%


Percentage Complete Estimates– Completion of customer review and approval

to proceed: 55%– Coding and unit testing completed: 75%– Integration testing completed: 85%– System testing completed: 95%– New feature released: 100%


Percentage Complete and Milestone Gates

• Popular method used by government organizations - uses a combination of weighted milestones and percent complete

• "Subjective" percent-complete estimates are allowed up to a specific ceiling associated with each milestone

• Advancement past the milestone is not allowed until tangible criteria have been met, hence the term "gate"


Equivalent Completed Units• Useful for extended duration and repetitive tasks• The overall project is divided into distinct units of

accomplishment• Earned value is computed by summing

the “units” completed


Earned Standards• Useful for production-type work• Perhaps the most sophisticated method to compute

earned value• Standards of performance based on historical cost

data, time and motion studies, etc., are used to compute the earned value on a given task...


Earned Standards• Often, several standards are used simultaneously,

and a management consensus determines which standard is ultimately used

• These measures are typically used to track earned value on either nonrecurring or recurring tasks


Apportioned Relationship to Discrete Work

• Useful for tasks in which their performance has a direct relationship to other tasks

• The earned value for apportioned tasks is a summary of the earned value measurements made on the work to which it is related

• Schedule variances in the apportioned work are usually identical to schedule variances in its related work


Apportioned Relationship to Discrete Work

• However, cost variances in the apportioned work are substantially different from the related work because of the dynamics of actual costs

• This measurement can employ any of the previous six methods


Level of Effort• Level of effort tasks are those earn value based

entirely on the actual amount of effort expended, hence whatever is set as the planned value is always the earned value, regardless of what work was done

• This method is generally not recommended to track earned value – no accountability!

• Used mainly for pure research or overhead tasks (e.g. clerical support)


References• Earned Value Management

– Sean Alexander, “Earned Value Management Systems Basic Concepts”

– Wayne Abba, “The New Management Revolution: Integrating Contractor Performance with Agency Budgets”

• See Balanced Scorecard for another management measurement system

http://www.acq.osd.mil/pm/

http://www.balancedscorecard.org/

Date post:	25-Feb-2016
Category:	Documents
Upload:	drake
View:	36 times
Download:	2 times

INFO 630 Evaluation of Information Systems Prof. Glenn Booker

Documents