INFO 636 Software Engineering Process I Prof. Glenn Booker

www.ischool.drexel.edu

INFO 636 Software Engineering Process I

Prof. Glenn Booker

Week 7 – Measurement

1INFO636 Week 7

www.ischool.drexel.eduINFO636 Week 7 2

Measurement

• Measurements are critical, because the act of measuring something focuses people’s attention on it

• We have already looked at several basic measures in the PSP– Now expand to a more general process

for choosing measurements and how to present them


Measurement

• The approach presented here is a variation on the GQM approach cited in the text, called GQ(I)M– The (I) adds Indicators, which are the means

used to present the measurements graphically

– This approach is also used in INFO 630


Why Care About Measurement?

• To quote Albert Einstein, "Not everything that counts can be counted and not everything that can be counted counts."

• We are seeking to identify things – 1) which can be counted, and – 2) which count toward achieving our goals


Reasons for Measurement

• Measurements are required by all major process models (CMM, ISO 9000, etc.)– To Characterize or understand the current

status of activities or products– To compare that understanding to our

objectives, and Evaluate whether the current status is “good” or “bad”


Reasons for Measurement

– To Predict future performance, based on past trends

– To form the basis for measuring Improvement• You won’t know if you improved, if you don’t know

where you started!


Where do we get Information?• Metrics are often used to support decision making (the

Evaluate step on a previous slide)• Decisions should be based on quantified information• To get that information, we calculate measures from raw

data called data elements• The data elements each come from a data source

DataSource

DataElements Measures Information


How do we Choose what to Measure?

• A commonly used method for selecting measurements is called GQ(I)M for Goal, Question, Indicator, and Measurement– It is based on GQM work by Victor Basili (first

reported circa 1988-89)– GQ(I)M was published by Robert Park et al

http://www.sei.cmu.edu/publications/documents/96.reports/96.hb.002.html


How do we Choose what to Measure?

• The GQ(I)M method uses ten steps to describe measurements systematically

• The steps don’t have to be followed in the order presented; their main purpose is to help ensure that measurements have been fully thought out– You can start at the top, or the bottom, or in the

middle


1. Identify Business Goals

• Business goals are the big, vague, lofty desired accomplishments for an organization

• Think of what you’d find bragged about in a company’s annual report– Reduce cycle time– Improve customer satisfaction


1. Identify Business Goals

– Develop detailed process history – Respond to changing business environment– Reduce overhead – Improve competitive position– Increase market share– Improve product quality


2. Identify Desired Knowledge

• Break down each goal into products, resources, and activities needed to meet those goals

• Think of questions like:– What activities do I manage or execute?– What do I want to achieve or improve?– To do this, I will need to …


3. Identify Subgoals

• Set subgoals (objectives) for each of the areas you manage

• What do you want to know about the results of step 2? What kind of information is important to you?– How big/fast/expensive/complex/much time

will a process/product/tool/resource take?


4. Identify Entities and Attributes

• Formulate questions to identify entities (documents, work products) created by your process, and the attributes of them you are interested in (size, quality, duration, cost)

• Don’t get too detailed at this point - just identify the type of information desired (what do you want to understand), and the subject of that information (what about it do you want to know)


4. Identify Entities and Attributes

• Entities that you want to measure can come from any of:– A process’ inputs– The activities within a process– The process’ outputs

• Then decide what characteristics of each entity are of interest


5. Define Measurement Goals

• First choose the type of measurement goals– Active goals: reduce or improve something– Passive goals: identify, assess, understand

• Lower maturity organizations start with passive goals, then work on active goals after some history is known



• Form structured statements of the measurement goals for each attribute– This step defines a metric in the form of a

sentence



• The general format or syntax is:– <verb> <measure> <qualifier(s)> [objective]

• where – <verb> is active or passive, as chosen earlier– <measure> is the actual measure to be collected– <qualifiers> indicate the scope or time frame of the

measurements (a.k.a. independent variables), – [objective] is only given for active measurement goals

– what value is desired?


5. Define Measurement Goals• Passive measurement goal examples:

– “Measure the number of requirements which changed each month.”

– “Identify the voluntary turnover rate per month for programmers and software engineers.”

• Active measurement goal examples:– “Reduce the defect rate of developed code over time

to under 20 defects/KLOC”– “Improve the percent of satisfied customers after 30

days of product use to 95% or more”



• DO NOT report traits of individual people (for fear of judgment)– Unless that’s part of their job description (e.g. sales

people)• Also need to balance how often measurements

are made– More frequent measurement gives finer control, but

excessive measurement wastes time and slows the process being measured


6. Quantify Questions and Indicators

• Pose questions to address your measurement goals (quantifiable ones, if addressing active goals)– Active: “Can we resolve customer

emergencies, on average, in under 24 hours?”

– Passive: “How many requirements do we have at the end of the Requirements Definition phase?”


6. Quantify Questions and Indicators

• Sometimes the search for a meaningful metric starts with a question, and that leads to filling out the GQ(I)M from the middle

• Identify indicators to show the results effectively (see later this lecture), such as:– Pie charts– Bar graphs– Scatter plots, etc.


7. Identify Data Elements

• Identify the data elements needed to prepare (calculate) each indicator – e.g. “defect rate by module each month”

needs a list of defects found, with the module each came from, for the last month, and the size of each module in kSLOC


7. Identify Data Elements

• Determine the source for each data element – from where do you get it?– E.g. defect data will come from our change

request database– Size data will be generated by the

development environment


8. Define Measures

• Define exactly what you mean by each measure – Use a checklist if needed to show what is and

isn’t included in its definition (such as we used for definition of LOC)

• Cite the source if an unusual measure is used; or if you made it up, explain why– New measures are allowed!


8. Define Measures

• Include any rules, assumptions, constraints, and environment which are part of the measure’s definition– E.g. Defect rate = (# of defects whose origin

was in each module)/(# of kSLOC in that module)


9. Identify Measurement Implementation

• Deciding how to implement a measurement program is generally done in three steps– Analyze what measures are currently

collected (if anything) and how they’re used – Diagnose how well the current measurements

meet your goals – find what’s missing?


9. Identify Measurement Implementation

– Act on implementing new measurements, possibly in a phased approach based on project or organizational priorities

• Implementing measurements is a cultural change, so like any other change, it’s best done in small steps– Add a few measurements now, wait six

months, add a few more, etc.– Let people see benefits from the new

measurements!


10. Prepare Measurement Plan

• Take all of the aforementioned information and create a complete plan to identify and implement measurement for your organization or project

• This is generally called a Metrics Plan or a Measurement Plan


Summary of GQ(I)M Steps

• Goal• Subgoal (objective; why collect this metric)• Question (answered by this metric)• Indicator (how display metric)• Measurement (the actual metric and

its definition)• Data Elements (used to calculate metrics)• Source (of each data element)


Indicators

• Indicators are the means used to present measures, such as charts, graphs, etc.

• Consider how your data will be presented - in color or B/W, live or printed– Will your audience see pristine originals, or

will it be copied a zillion times?


Indicators

• To choose the right indicator, consider: 1.The Amount of Data to be presented for each

interval (e.g. one measure at a time or five different survey responses at once)

2.The Number of Intervals to be shown, such as time units, modules of code, etc.


Indicators

• Different indicators are better at different Amount or Number characteristics


Indicators

• For most graphs:– The X-axis of a graph (the horizontal line) is

the independent variable; often a <qualifier>, such as time, severity, etc.

• If the X axis is Time, it should show how often measurements are made (weekly, monthly, etc).


Indicators

– The Y-axis of a graph (the vertical line) is the dependent variable; the thing you are measuring (the Measure).

YDependent

variable(measure)

X axisIndependent variable

(qualifier)

x

x

x

x

x

xx x


Pie Chart

• The lowly pie chart is good for presenting a small amount of data– % of customers satisfied and not satisfied– % of defects by severity at this moment

• Amount of Data: Shows a few data points (2-10)

• Number of Intervals: One - it generally shows only one moment in time


Sample Pie ChartSales ($M)

1.2

2.4

1.1

1.7

4.6

West

Northeast

South

Midwest

East


Ishikawa’s Basic Tools

• Developed circa 1989 for manufacturing production– Used widely in quality control

• Focuses on project level concerns; is this batch good enough to accept?

• Not very useful for research; has little theoretical basis


1. Check sheet

• Used to gather data easily, consistently, and in a standard format

• A check sheet used to help the quality of a process or product is a “checklist”

• Helps to define key parts of a process• Examples include code inspection

checklist, detailed test procedures


2. Pareto Diagram

• Used to identify problem areas - where to fix first; what are the biggest fires to put out– Defects tend to cluster in buggy portions of code

• Use Pareto to plot a column graph of the defect rate by the module it came from – Could add a line graph for the cumulative % of

defects above it (not shown)– Pareto diagram must list X axis categories in

descending order of value


2. Sample Pareto DiagramPareto Diagram of Component Defect Rate

22.3

15.613.3

11.18.9

5.6

0

5

10

15

20

25

Interface Query Core Import Export Reports

Component

Defe

ct R

ate

(def

ects

/KLO

C)


3. Histogram

• A bar chart is used to break down data by an ordered category (e.g. defect severity, satisfaction rating)– Can choose to put bars next to each other

(clustered), or stack them. • Stack when they add to a constant (e.g. 100%),

or when the total is also a useful measure (total number of defects)


3. Histogram

• Can show limited time spans, e.g. a few time intervals

No. of Problem Reports by Status

23

234

15 834

50

0

50

100

150

200

250

New Closed Pending Withdrawn Analysis Testing

Status

No. o

f Pro

blem

Rep

orts

Simple bar graph


3. HistogramProblem Reports in each Status over Time

0

100

200

300

400

500

Jan-02 Feb-02 Mar-02 Apr-02 May-02

Time

Num

ber o

f Pro

blem

Re

ports

Testing

AnalysisWithdrawn

PendingClosed

New

Problem Reports in each Status over Time

0

50

100

150

200

250

300


Time

Num

ber o

f Pro

blem

Re

ports

New

ClosedPending

Withdrawn

AnalysisTesting

Stacked

Clustered


3. Histogram

• A true histogram can group ranges of values in the X axis, and show counts or average Y values for each group– E.g. average salary (Y axis) for employees

aged 18-29, 30-39, 40-49, etc. (X axis groups)– Good for hiding individual values, and looking

at larger trends in the data


4. Run Charts

• Plots some measure versus time using a line graph– Often compare values to a desired or

target value, especially at higher process maturity levels (CMM Level 3 and up)

– Special case: The “S” curve plots (cumulative % completion of something) versus time


4. Sample Run Chart

Number of Defects Found by Severity

05

1015202530354045


Time

Num

ber o

f Def

ects

Fou

nd

Minor

Major

Total


5. Scatter Diagram

• Plot two measures against each other to see if there’s a correlation between them– E.g. Defect rate per module versus module

complexity, or productivity versus experience• When we say plot ‘Blah versus Ick,’ usually Blah is

the Y axis, and Ick is the X axis


5. Scatter DiagramCurrent Salary vs. Educational Level

EDUCATIONAL LEVEL

2220181614121086

CU

RR

ENT

SALA

RY

60000

50000

40000

30000

20000

10000

0


5. Scatter Diagram

• Once there’s enough data, can add curve fitted lines, and +/- variances

• The scatter diagram is the basis for regression analysis– Curve fitting to the data, to help define a

statistically significant connection between the two measures


6. Control Chart

• Statistical Process Control tool• Rare in software development since

– Specifications poorly defined– Each project takes a long time– Too many uncontrolled process variances– Process-quality relationship not well defined


6. Control Chart

– Too many processes used– Rapidly changing technology

Control Chart: Percent Passed

Sigma level: 3

121110987654321

93.000

89.000

85.000

81.000

77.000

Percent Passed

UCL = 92.3343

Average = 85.0833

LCL = 77.8324


6. Control Chart

• Pseudo-control charts include:– The ‘u’ chart for defect rates by component,

BMI, etc. versus time– The ‘p’ chart for percentages, such as

inspection effectiveness or customer satisfaction rating

• There are many other varieties of control charts


6. Control Chart

• Usual control limits are +/- 3 sigma from the mean, which define the upper and lower control limits (UCL and LCL)– Can add a warning limit at +/- 2 sigma

• Control Chart tend to be used in very mature (CMM level 4 or 5), long term projects


Fishbone Diagram

• Technically, the fishbone diagram is Ishikawa’s seventh tool, but I doubt anyone will use it in conjunction with the PSP


Measurement Basics

• Given all that on GQ(I)M and indicators, the most essential types of measurements are still pretty basic– Process – how long does it take?, how much

does it cost?, are we starting it on time?, etc.– Tools – (rare) how well are our development

tools being used?


Measurement Basics

– Resources – how much does it cost to keep our people trained? How much does it cost to find new personnel?

– Product – how big is our product (LOC)? What is its quality? How many defects have been found? How happy is the customer with it?


Gathering Data

• Critical from the PSP perspective is defining our processes to gather data consistently and efficiently– How do you record time spent?– How do you collect measurements from

various people and merge them? (A critical concern for INFO 637)


Gathering Data

• Our forms have been designed to help collect key aspects of the processes– Product size– Time spent in each part of the process– Defect injection and removal phases– Defect type and fix time– Bad fixes (the infamous Fix Error field)– Productivity (LOC/hour)


Gathering Data

• The result of this is a large body of data that can be collected across many projects– While the data gathering and analysis takes

time, it also produces real answers about how quickly and well you perform various tasks

– This is your personal process baseline

Date post:	21-Feb-2016
Category:	Documents
Upload:	darren
View:	50 times
Download:	0 times

INFO 636 Software Engineering Process I Prof. Glenn Booker

Documents