Measuring and Managing Performance in Organizations
EECS 811: Software Project ManagementPresenter: Molly Coplen
March 3, 2009
2
Contents
• Introduction– terminology– why we measure activity, why we care
• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study• Conclusion - lessons learned
3
Management and Measurement
• Why do we care? – each organization has a purpose or a goal– we measure <parameter> in hopes of
improving <goal>.
4
Terminology
A measure is a way to appraise or determine by comparing to a standard or unit of measurement. The act or process of measuring is referred to as a measurement.
A metric is a quantitative measure of the degree to which a component, system or process posses a given characteristic or attribute.
5
Why We Measure Activity • Focus workers’ attention on the task for a
desired outcome (motivate workers).
• Provide an objective basis for decision making – “You can’t control what you can’t measure.” Tom DeMarco
– resource allocation– hold workers accountable for
performance.
6
Contents
• Introduction• Types of measures
– motivational– informational
• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Conclusion
7
Types of Measures
Motivational measurements - intended to affect the people being measured, to provoke greater expenditure of effort in pursuit of organizational goals.
Informational measurements – valued for the logistical, statistical and research information they convey, which allows better short-term management and long-term improvement of organizational processes.
8
Informational Measurement
Process refinement measurement – provides information that reveals the detailed understanding of a process are useful in redesigning the process.
Coordination measure – provides information that allows short-term (real-time) management of organizational flows and schedules.
9
Informational Measures
• More accurate information, advanced warning of upcoming problems.
• Dysfunction can arise when rewards are tied to performance.
• Aggregating measures makes motivational measurement impossible.
• Self-assessment tool, not diagnostic/ analysis tool.
• Works well when workers are internally motivated.
10
Informational MeasuresE. Nichols and G. Peterson (2007)
• Design-time metrics – identify eaknesses early in the application’s life cycle
• Deployment-time metrics –identify the amount of change, uncover patterns over time, and establish baselines for anomaly detection
• Run-time metrics – determine the application’s behavior in production environment
11
Contents• Introduction• Types of measures• Basic problems with measuring
activity– economic costs– difficult to measure some things– measurement dysfunction
• Economic models and motivation theory• Software metrics• Case-study• Conclusion – lessons learned
12
Problems with Measurement Programs
• Costs– establishing/maintaining measurement
systems– workers take short-cuts which impact the
quality, require re-work
• Some activities are difficult to measure (i.e., are we using the correct measure?)
• Potential for dysfunction
13
Metrics cost a ton of money. It costs a lot to collect them badly and a lot more to collect them well. Sure, measurement costs money, but it does have the potential to help us work more effectively. At its best, the use of software metrics can inform and guide developers, and help organizations to improve. At its worst, it can do actual harm. And there is an entire range between the two extremes, varying all the way from function to dysfunction. --- Tom DeMarco
14
Measurement Costs - Factors
• Repetitiveness of task₋ offer more opportunities for observing all phases of
agent’s work₋ permit use of statistical sampling techniques₋ inexpensive measurement enabling full supervision
• Complexity of task₋ overtaxes principal’s cognitive capabilities
• Newness of task– principal more likely to be familiar with older tasks– older tasks provide more measurement opportunity over
time
15
Measurement Costs - Factors
• Specialized knowledge required by the task
₋ difficult/expensive to find someone qualified to do measurement & interpret results
₋ typically include tasks that are pushing technology beyond boundary
• Interdependence and separability of effort₋ tasks of numerous workers tightly coupled₋ all have ability to hinder others tasks₋ impossible to distinguish how much effort each
member is contributing
16
Measurement Dysfunction“Consequences of organizational actions that interfere with the spirit of the stated intentions of the organization.”
TimeTrue performance
Measurement Indicators
Level of performance
17
Characteristics of Measurement Dysfunction
• We measure too few things or the wrong things - we reward “A” for hoping for “B”.
– key measures do not provide clear account of performance
– measured standards interfere with achievement of intended effect
– employee rewards not associated with performance of organization
18
Problems with Measurement Deffers,C and McQuaid, P . (2002)
• Basic misunderstanding what is being measured
• Incorrect use of measurement data –leads to additional costs (morale, quality, lost revenues)
• Disregard for the human factor – how the cultural change of taking measurements will affect people – and the knowledge intensity of software development
19
Problems with Measurement Ordonez, M. and Haddad, H . (2008)
• Upfront commitment to gather necessary data for useful metrics, including time, cost, and resources factors
• Developers reluctant to collect and archive data that could be (mis)used against the developer and project stakeholders
20
Contents• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation
theory– internal and external motivation– theory X and theory Y– R-H and R-M models
• Software metrics• Case-study • Conclusion – lessons learned
21
Motivation
External motivation – a tendency to react in response to promises or rewards for performance according to mutually known criteria set in advance
Internal motivation – a tendency to act as compelled by private expectations of performance based on own personal criteria
22
Theory X
Managers believe people:• Dislike work and attempt to avoid it• Must be coerced, controlled, directed,
threatened• Inherently unambitious and want to avoid
responsibility• External motivation is a stronger and more
reliable way of motivating employees•http://www.youtube.com/watch?v=nXN5s8fQvHY
23
Theory Y
Managers believe people:• Expenditure of effort as natural as rest or
play• Exercise self-direction and self-control• Seek responsibility, exercise responsibility
in solving problems
Managers concerned with inspiring internal motivation and clearly communicating direction to employees
24
Internal and External Motivation
Internal motivation – Theory Y
External motivation – Theory X
Expected payback - feelings of fulfillment, gratification organizational success, identify with the organization (family)
Expected payback – promise (expectation) of reward
Considers effort on work as natural as play and rest; seek responsibility and exercise creativity in solving problems
Workers dislike work and are basically lazy
Self direction, self control to achieve objectives
Coerced, directed, threatened to achieve organizational objectives
Managers use communication, inspiration to motivate
Managers concerned with design & implementation of external motivation schemes
25
Ross – Holmstrom (R-H) Model
• Principal – manager/supervisor – motivated by profits
• Agent– worker– motivated by fondness for money, dislikes
work– possible internal motivation– effort gets more expensive as its level of
activity (output) increases
26
Ross – Holmstrom (R-H) Model
• Principal cannot determine the value of the agents production directly– Use measurements as a proxy of value
• Agent & principal have opposite interests– principle wants maximum value (effort) for
least cost– agent wants maximum value (money) for
minimum effort
27
Ross – Holmstrom (R-H) Model
• The optimal payment schedule –payment to the agent increases as revenue increases– justification for merit-pay, pay-for-performance
• Assumptions valid?– employees care about more than making
money and avoiding work– managers care about more than profit– effort/mix problem
28
Holmstrom and Milgroom
• Effort must be devoted to each task
• Internal motivation – finds pleasure to provide value to the customer– might agents be persuaded to enjoy doing
more work that is beneficial to the principal without monetary reward?
– work towards goals of the company by evoking feelings of identification, belonging
29
Holmstrom and Milgroom
• Contributions– abandoned the assumption that what is
measured is a perfect indicator of the true value produced by the agent
– true output depends jointly on effort devoted to all tasks
– value is provided in a non-additive manner as agents devote effort to tasks
– use multiple measures – have all key areas been measured
30
Designing Measurement Systems
• Comprehensiveness is desirable – must measure all critical dimensions– essential to prevent dysfunction– difficult to verify performance of knowledge workers– quality is notoriously difficult to measure
• Motivational uses of measurement– agent can not be prevented from reacting to measure– results in competitive dynamics that distorts measure– causes warning signs to be concealed– extremely difficult to achieve informational only use
31
Designing Measurement Systems
• Use measurements for information only – convince agents that measure is informational
only• promote people longer time• no sudden rising stars• reduces competition within employees
– aggregation of measurement• individual see measures at their or higher level • need procedural methods to enforce aggregation• gives manager a self assessment• managerial job shifts from evaluation to inspiration
and assistance to subordinates
32
How to Make Jobs More Measurable• Standardization
₋ almost all processes repetitive at some level₋ identify phases and establish standard
methods of execution
• Specification₋ construct detailed model of the process₋ standardize each step in a process₋ identify variances from specification
33
How to Make Jobs More Measurable• Subdivison
– decompose jobs into subtasks and group similar tasks
– grouping makes repetition & self similarity visible making measurement easier
– people doing similar activities can be assigned overseers that have specialized knowledge as workers
34
Delegatory Management
• Promote organizational intimacy– make hierarchy flatter & work groups smaller– reconfigure work spaces to promote casual
contacts between managers and workers in same team
• Promote trust– facilitate mutual trust - employer and
employees– investments that suggest permanence of
employee base such as investing heavily in training
35
Delegatory Management
• Provide better leadership– skilful expression of vision– management styles that emphasize inspiring
and trusting employees rather than analyzing and coercing
36
Internal Motivation vs External Reward
• Offer of an external reward for that which would be provided through internal motivation may be insulting and lower internal motivation
• “It’s demeaning... People are motivated by intrinsics and extrinsics and the intrinsics are much more powerful – the pride, the workmanship, the enjoyment of doing things with their colleagues....But the extrinsics shove the intrinsics aside. Say you give me an extrinsic reason to do this. I lose track of whether I’m having a good time working with my colleagues on this goal.” – DeMarco on performance measurement
37
Contents• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics
– common software metrics– factors to consider when choosing a metric– characteristics of successful programs– human factor – the programmers
• Case-study • Conclusion – lessons learned
38
What is a Software Metric?
A software metric is defined as a method of quantitatively determining the extent to which software processes, product, or project possess a certain attribute. (Daskalantonakis, 1992)
Rationale - a metrics program should lead the software organization to more disciplined processes through an efficient feedback mechanism. (Gopal, Krishnan, Mukhopadhyay, and Goldenson, 2002)
39
Common Software Metrics (1/2)
Activity Description
Cost and Effort Estimation models help plan and execute projects. Mathematical models such as Boehm’s COCOMO, Putnam’s SLIM, and Albrecht’s Function Points
Productivity Measures
Human side - team structure, tools, skills, backgrounds, environment.
Data Collection Leads to perception of being “under pressure” and “at risk” as negative data can be used against the developer.
Quality Assessment Efficiency, reliability, flexibility, portability, usability, correctness, etc.
Reliability Models Related to software failures
40
Common Software Metrics (2/2)
Metric Description
Process Focus on software development and maintenance and are used to assess people’s productivity (private metrics) or the productivity of the entire organization (public metrics).
Project Tactical and related to project characteristics and execution. Used by project managers and software developers to adjust product workflow and technical activities.
Product Measure key characteristics of the software product.
41
Common Software MetricsList of Project Metrics (1/2)
• Order of growth• Lines of code• Cyclomatic
complexity• Function points• Code coverage• Coupling• Cohesion
requirements size
• Application size• Cost• Schedule• Productivity• Number of software
developers
42
Common Software MetricsList of Product Metrics (2/2)
• Specification quality metrics• System size metrics• Architectural metrics• Length metrics• Complexity metrics• Testing effectiveness
43
Cem Kaner’s Factors to Consider When Choosing a
Metric (1/2)1. What is the purpose of the metric? 2. What is the scope of the measure?3. What attribute are you trying to are you
trying to measure?4. What is the attribute’s natural scale?5. What it the attribute’s natural variability?6. What instrument are you using to
measure the attribute, and what reading do you take from the instrument?
44
Cem Kaner’s Factors to Consider When Choosing a
Metric (2/2)7. What is the instrument’s natural scale?8. What is the reading’s natural variability
(measurement error)?9. What is the attribute’s relationship to the
instrument? 10. What are the natural and foreseeable side
effects of using this instrument?
- Cem Kaner (2002) as cited by Dekkers, C.A. and McQuaid, P.A. (2002)
45
Basic Guideline – Selecting a Metric
Dekkers and McQuaid• Research and analyze your organization’s
measurement goals• Decide what questions will tell you
whether you are progressing towards the goals.
• Select the appropriate measures to answer the questions.
• Recognize intentional and unintentional side effects of the chosen measures and metrics.
46
Goal Questioning Metric (GQM)Victor Basili and David Weiss
1. Goals – define the objectives (scope and purpose) to be addressed by measurement
2. Question – defines what quantifiable information is required to determine whether there is progress toward the goal(s)
3. Metric – defines the particular attributes, definitions, observation (collection) frequency for measurement data, measurement theory, statistical design, and applicability of measurement data.
47
Characteristics of Highly Successful Measurement
Programs (1/4)
1. Set solid measurement objectives and plans
– aim to achieve firm objectives– implement as a development project
2. Make measurement part of the process– measure key success factors– integrate data collection
48
Characteristics of Highly Successful Measurement
Programs (2/4)3. Gain a thorough understanding of
measurement – use metrics to:– implement corrective actions– measure requirements, not individual
performance
4. Focus on cultural issues– involve developers in the measurement
program’s development– recognize cultural changes affects how people
view their work
49
Characteristics of Highly Successful Measurement
Programs (3/4)
5. Create a safe environment to collect and report true data.
₋ “..understand the data that your people take pride in reporting. Don’t ever use it against them. Don’t even hint that you might.” Robert Grady – Hewlett Packard
₋ give people access to the measurement data₋ collect data using consistent measurement
definitions
50
Characteristics of Highly Successful Measurement
Programs (4/4)6. Cultivate a predisposition to change
₋ respond to the measurement data7. Develop a complementary suite
(dashboard) of measures
-- Dekkers, C.A. and McQuaid, P.A. (2002)
51
Hewlett-Packard (HP) (1/3)
• Goal - to improve software project management, team productivity, and software quality– achieved in the short term for individual
development projects
• Metrics categorized– primitive – directly observed (e.g., total
development time for a project, lines of code)– computed – mathematical aggregations of two
or more primitive metrics
52
Hewlett-Packard (HP) (2/3)
• Metrics for project scheduling, cost of defects, work load, and project control– average fixed defects/working day– average engineering hours/fixed defect– average reported defects/working day– defects/testing time– percent overtime: average overtime per week– phase: engineering months/total engineering
months
53
Hewlett-Packard (HP) (3/3)
• End product quality metrics– defects/KNCSS (thousand non-comment source
statements)– defects/lines of documentation not included in
the program source code
• Testing effectiveness metrics– defects/testing time
• Testing coverage metrics– branches covered/total branches
54
Motorola (1/2)
Goal Metric Program
Improve project planning
Schedule and Effort Estimation ActivitySEA = Actual project duration/estimated project durationEEA = Actual project effort/estimated project effort
Increase defect containment
Total Defect Containment EffectivenessTDCE = # of prerelease defects /( # of prerelease defects + number of post release defects)Phase i Containment EffectivenessPCEi = # of phase i errors/# of phase I errors + # pf phase i defects
Increased reliability
Failure Rate (FR)FR = # of failures /execution time
55
Motorola (2/2)
Goal Metric Program
Decrease software defect density
In Process FaultsIPF = in process faults caused by incremental software and development/assembly – equivalent delta source size
Total Released Defects TRD total = # of released defects/assembly – equivalent total source size
Customer Found Defects (CFD) totalCFD Total = # of Customer found defects / Assembly – equivalent source size
Improve customer service
NOP = Total new post release problems opened during the month
TOP = Total post release problems that remain open at the end of the month.
56
Why Programmers Avoid Metrics
• Problems with data collection– suppress partial information– scripting– entering false data– illusion of compliance
• Reasons for push-back– maintaining image– resistance to standardization– begrudging compliance– metrics are not representative of effort– product and process compliance
57
Why Programmers Avoid Metrics
• Metrics Acceptance Model – social influences – organizational usefulness– personal usefulness– available resources– self-efficacy– compatibility– ease of use– attitude
58
Contents
• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study
– Financial Software Solutions• Conclusion - lessons learned
59
Case: Financial Software Solution
Financial Group
Finance Bank Financial Software Solutions
60
Case: Financial Software Solution
• Primary business of FSS is the development of IT systems and services for Financial Group– also sells IT systems to other financial
institutions in Europe– expertise in developing banking, insurance,
mortgage and financing applications• 850 employees • Projects – small and short-term, 3-5 people
of the project team, 6-12 months
61
Case: Financial Software Solution• Projects
– small and short-term, 3-5 people, duration of 6-12 months
– major projects, 10-20 people, duration of years
• 4 development divisions– headed by a Senior Vice President– divisions composed of departments headed by
Vice Presidents, staffed by 20-25 people divided into 5 projects
• Project managers manage regular projects, VP manage high-profile projects
62
Case: Financial Software Solution
• FSS an independent subsidiary delivering IT services for Financial Group
• FSS wanted to stay competitive with others – Software Process Improvement (SPI) strategy for staying competitive– project given high profile – VP as a PM and
other VPs were team members– idea was each division have a representative –
few were actually active
63
Case: Financial Software Solution
• Key stakeholders– CEO – sponsor - goal – improve efficiency 10%– Vice Presidents– Project managers – provide data on their
project to the metrics program– SPI project manager – 4 during the life of the
project– Finley – VP and member of SPI improvement
group – key to defining the metrics program– Linda – VP and member of SPI group
• Who is not included?
64
Case: Financial Software Solution
• CMM level 1 - Initial (chaotic, ad hoc, heroic) the starting point for use of a new process
• Approached this project as a project• Goal – improve efficiency 10%
– how measured ? undefined– how is data to be analyzed? unanswered
65
Case: Financial Software Solution
Indicators of the Metrics Program
Factor Definitions
Project productivity Resources used to develop the system relative to its size in function points
Quality Number of error reports both absolute and relative to size in function points
Adherence to schedule
Variation from agreed time of delivery both absolute and relative to the size in function points
Adherence to budget Variation in estimated use of resources
Customer satisfaction Satisfaction with the development process and the implemented solution (questionnaire)
Employee satisfaction Satisfaction with the development process (questionnaire)
66
Case: Financial Software Solution
• Principles the metrics program should adhere to:– Measurements should be relevant in relation to
process improvement and quality– Measurements should be made automatic
whenever possible– Minimal cost of performing the measurement– Use of questionnaires limited– Data collected on finished projects – Results published quarterly
67
Case: Financial Software Solution
• Timeline– March 1997 – decision to implement– September 1997 – first report
• data on 3 of the 6 factors • 13 of 56 projects completed in September• report withheld from wide distribution due to
‘adherence to budget” measure• parts disseminated to developers via a “road show”
to raise awareness of the project
68
Case: Financial Software Solution
– March 1998 – second report • reported on 65% of completed projects• October – December 1997 (time lag)• data considered to be sufficiently unreliable for wide
distribution• concern with how the data will be used
• Public disclosure• One member of the team had promised
development staff that information would not be disclosed
• concerns about the customer satisfaction survey• resulted in a project to improve the metrics program
69
Case: Financial Software Solution
– Project established – August 1998 • 18 months after kick-off• signed contract to improve the metrics • goal: improve the quality of the measurement report
so much that management would publish the report• success: published report April 1999 with all 6
indicators, all projects completed 1Q 1999• function points dismissed due to problems of
inaccurate count• decided to not include a size metric and rely on the
rest of the measure to provide information on the effect
70
Case: Financial Software Solution
– Third report – April 1999• 10 projects reported complete data and data was not
reliable• misunderstanding of the definition of some fields
and other things entered• data not collected• data not maintained if changes need to be made
• no measurement report, produce a memo to management describing the problems
• bonus system introduced for divisions collecting accurate data
71
Case: Financial Software Solution
– Decision to publish – September 1999• 2 ½ years after project initiated• SVP made corrections they wanted made• released to VPs• 64% projects reported on• “commitment from management is required to reach
a better result”
72
Contents
• Introduction• Types of measures• Basic problems with measuring activity• Economic models and motivation theory• Software metrics• Case-Study• Conclusion - lessons learned
73
Lessons Learned
• Adequate resources available• Set solid objectives and plans• Approach as a software engineering
project– apply Goal – Question – Metric model
• characterize the environment• identify management goals and develop
measurement plans• define data collection procedures• collect, analyze, and interpret data• perform postmortem analysis and interpret data• package experience
74
Lessons Learned
• Adequate resources available• Set solid objectives and plans• Approach as a software engineering
project– apply Goal – Question – Metric model
• characterize the environment• identify management goals and develop
measurement plans• define data collection procedures• collect, analyze, and interpret data• perform postmortem analysis and interpret data• package experience
75
Lessons Learned
• Start with a few indicators – stepwise approach
• Use a complementary suite of metrics• Establish incentives for compliance• Create a safe-environment for collecting
and withholding information• performance measures of an individual should be kept to
the individual• don’t withhold information
76
Lessons Learned
• Recognize the culture shift decisions are based on relevant and accurate information, not intuition
• Discuss metrics and what makes sense for the organization
• Take corrective action based on the measurement
77
Lessons Learned
• Learning cycle of a metrics program
Interpretation
Data Collection
Intervention
Inte
rven
tion
Results Knowledge
Object of Study Object
78
References• Austin, R.D. (1996), Measuring and Managing Performance
in Organizations,” Dorset House Publishing.• Basili, V. and Weiss, D. (1984), “A Methodology for
Collecting Valid Software Engineering Data,” IEEE Transactions on Software Engineering, 10 (11), pp 728-738.
• Daskalantonakis, M. (1992), “A Practical View of Software Measurement and Implementation Experiences with Motorola,” IEEE Transactions on Software Engineering, 18 (11), pp. 998-1010.
• Dekkers, C.A. and McQuaid, P.A. (2002), “The Dangers of Using Software Metrics to (Mis)Manage, IT Pro, 4(2), pp. 24-30.
79
References• Gopal, A., Krishnan, M.S., Mukhopadhyay, T., Goldenson, D.
(2002), “Measurement Programs in Software Development: Determinants of Success,” IEEE Transactions on Software Engineering, 28 (9), pp. 863-875.
• Iversen, J. and Mathiassen, L. (2000) “Lessons from Implementing a Software Metrics Program,” Proceedings of the 33rd Hawaii International Conference on System Sciences.
• Nichols, E. and Peterson, G. (2007), “A Metrics Framework to Drive Application Security Improvement,” IEEE Security and Privacy, March/April 2007, pp. 88-91.
• Ordonez, M. and Haddadm, H.M. (2008), ‘The State of Metrics in Software Industry,” Fifth International Conference on Information Technology: New Generations, April 2008, pp 453-458.
• Umarji, M and Seaman, C. (2008), Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement , Kaiserslautern, Germany , pp. 129-138.