Metrics for Managing

transcript

CMM Based

Metric Workshop

• Metrics for Management– Metric program should be simple– Metric program should be vertically and

horizontally consistent– Metrics should be factual– Metrics serve as the catalyst for effective

management decisions at all organization levels

CMM Review– Good Business Model– L2/Tracking to plan (compliance to spec)– L3 Tailoring to client needs (efficiency)– L4 Performance and Goals at project level– L5 Performance and Goals at organization level

Measurement improves Visibility-Need for timeliness

-Need for accuracy

-Need for consistency

-Need for relevance

-Need for completeness

Measurement data sources are varied-quality reviews and audits; peer and technical reviews;

phased, management, and project reviews; testing

CMM ReviewStatus; used in each KPA (Why? What is

implication?)

Quality; part of most KPAs (Cost of Quality?)

Management Reviews; part of all KPAs

All KPAs are made up of key practicesExecution of practice requires resources. Resources

need to be controlled and managed.

• Simplicity of Measures:– Measures for managing activity– Measures for managing process quality– Measures for managing product quality– Measures for managing performance

•Vertical and Horizontal Consistency

Activity Activity

Developer Loop

Project Management Loop

Quality Management Loop

Organization Management Loop

Developer Loop

• Factual (accurate, complete, consistent)– Metrics are derived from factual data

• Test data

• Review and Audit data

• Accounting and Timekeeping data

• Project Status Data

• Fact-based metrics facilitate timely and management decisions.

• Metrics Act as a Catalyst– Provides management visibility– Eliminates guessing, removes subjectivity

from decision making process– Meetings are focused on facts– Consensus on status leads to consensus to

the actions to be taken

• Who needs metrics– Decision makers

– Analytical support activities like SEPG, Quality, Tools Group

– All levels of management

• Task Manager

• Project Leader

• Program Manager

• Group (Business Unit) Manager

• Organization Manager

• Basic Premise

“ If the metric does not support a management decision process, then it is not necessary and

should not be collected.”

• What Metrics should be collected?– Metrics that show the status of work activities in

the organization (Time, Effort, Current State, number of occurrances)

– Metrics that detail performance of the activities that are performed (actual vs plan)

– Metrics that detail the quality attributes of the products (error density, error rate, MTTR, MTBF, Reliability)

– Metrics that detail the quality attributes of the process (error density, error rate)

• Metrics that show status allow manage-ment to take effective actions to meet project requirements– Schedule status (effort of work items completed

less the effort of work scheduled to date) shows the effort that will need to be added to complete on schedule

– Effort status (effort of work items completed less the effort expended to date) shows whether there is overage or underage on budget

• Other status metrics– Number of changes (opened, received, closed, etc)

– Elements of system (requirements, design objects, units of software code, tests, etc) work/time

– Comparison of planned activities vs actual activities

• Value of these metrics is that they assist management in allocation of resources to achieve the project objectives

• Quality of product metrics– Number and type of defects found

– Distribution and value of defects in product

– Value of actions to remove and repair defects

• Value of product quality metrics is they:– Identify problems so that they can be fixed

– Provide insight into the product being developed

– Assist in identifying product quality issues

– Enable project to enhance overall product quality

• Quality of process metrics– Number and type of defects found

– Distribution and value of defects in process

– Value of actions to remove and repair defects

• Value of process quality metrics is, they:– Identify problems so that they can be fixed

– Provide insight into the process being utilized

– Assist in identifying process quality issues

– Enable organization to enhance overall process quality

• Executive Organization Metrics

Process Lifecycle Phases

Work Scheduled

Work Completed

Work Accomplished

Errors (value)

Error ratio

Moving Averages (3 Month Window)

New Projects

Active Projects

Project man/mth

• Executive Organization Metrics (continued)

Total Staff

Managers

Project Pers

Support

Turnover

Training Ratio

Systems Ratio

HR Ratio

• Executive Organization Metrics (continued)

Moving Averages (4 Week Window, Weekly)

Productivity

Delivery

Defect Rate

Removal Eff.

Defect Ratio

Performance

Technology

• Project metrics ( all stated in effort)

DEFECT DATA (In effort lost)

Schedule Effort REQ HLD DD C&T INT Other Total

Norm/Plan

Task 1

Task 2

Task 3

Task 4

Task 5

Project Total

Business Unit Head TotalBusiness Unit Head Total

Program Manager TotalProgram Manager Total

Project TotalProject Total

Project Total

Program Manager Total

Business Unit Head Total

Organization Total

• Project level metrics– Status: schedule and effort, planned vs

actual– Quality: product and process, allocation of

defects/nonconformance to elements of the product being built and to process by lifecycle phase

– Performance: relationship between accomplishments, time, and quality

• Clarification of Schedule and Effort Metrics (Schedule only as example)– Schedule: Determine variance to schedule in

precise quantitative terms(Effort of items completed-Effort of scheduled items)= Variance

– Example: All effort allocated to tasks of a project scheduled to be complete after three weeks total 200 man hours (10 tasks of 20 hours each). The actual effort of tasks completed is 160 (only 8 tasks are complete)…value is -40 hours (variance)

• Defect Data Characteristics– Project, task, WBS element, product or process

element

– Activity Type, evaluator, sample size (if audit)

– Defect Data• Location in product (or facility)• Description• Defect Code• Priority• Effort (if manually collected)• Disposition • Status

• At this point, will all of the data and metrics, you are still a level 3 company– You have consistency of process and

consistency of metrics used across and vertically in organization

– Information helps projects plan better and assists in understanding processes and ensuring product quality

• Transition from L1 to L2/3 to L4/5– L1 Metrics are known at the end of a

project…no value to management– L2 Metrics used on the project plan vs

actual– L3 Metrics consistent across organization

and used by others– L4 Metrics used to Manage Projects– L5 Metrics used to Manage Organization

Earned Value Metrics

Budgeted Cost of work Performed (BCWP)

Actual Cost of work Performed (ACWP)

Budgeted Cost of work Scheduled (BCWS)

BCWP:Original or Updated Estimate

% of Work or units of effort

(cost orEffort)

Cost Variance

Schedule Variance

Types of metrics

• Management– Performance, timeliness, status, efficiency,

profitability, return on investment, staff turnover, training days, growth rate, productivity, % of rework, etc.

• Quality– Error Rates, Cost of Quality, Number and value of

errors by type, seepage rate, distribution of errors by projects, by life-cycle phases, etc.

Management

Quality

Organization Sub-organization Project Activity

Each element has a unique set metrics that are important to

that organization element

Activity

ExecuteProcess

ExecutePlan

DefinePlan

Adjust Plan

MeasurePlan/Actual

DefineProcess

ControlProcess

MeasureProcess

Requirements Product

ImproveProcess

ProcessManagement

PerformanceManagement

Developing Metrics…a 4-Step Process

• Establish goals for measurements

• Define the attributes of the goals

• Describe metric measures

• Identify data required

• Goals Examples:– “We need to measure our productivity”– “We need to measure defect frequency”– “Philips needs to measure project

accomplishments”– “We need to measure the accuracy and

completeness of testing”

• Verification:

Establish how metric will be used to improve or manage performance or quality. Example: Productivity will be used to determine how projects are performing to their plan so that actions can be taken when significant deviations are noted.

• Attributes of the Goals: Productivity– What is existing productivity?– What is the expected productivity?– What is the improvement desired?– What was the productivity at the end of

each of the last 3 quarters?– What is our productivity trend?

• Metrics of Productivity

Effort Expended (Man-hours)

Product Units

Estimated Hours per Phase

Actual Hours per Phase

• Data needed for measures: Productivity– Units of work– Actual effort expended on work– Estimated effort of work– History of units of work produced and effort

consumed in production– Distribution of work by phase– Actual work distribution by phase

• Exercise One:– Groups of 5 persons grouped by function:

– Each group is assigned a level of management for metric definition

– Each group will identify at least 10 goals relative to the level of management of the group

– At least 6 goals should be “Management” goals and at least 4 should be “quality” goals

– Time is 1 hour: Then each group will present its goals to the class, 10 Minutes for each presentation

– Discussion by Class

• Exercise Two:– Each Group will verify each of its goals by

defining how the goal will be helpful in improving performance

– 20 Minutes for verification, then 5 minutes per group to explain verification to class

• Exercise Three:– Each Group will identify the attributes needed for so

that the Measurement Goals can be met. Each group will also define at least one metric method (Calculation) for each Measurement Goal.

– Each Group will take 45 minutes to complete; then they 10 minutes to report.

• Exercise Four– Each Group will identify the data elements

needed for each goal/attributes/measurement and determine if the data is currently available.

– 30 minutes. Then present need and availability.

Metrics for Managing

Documents