Date post: | 21-Jan-2016 |
Category: |
Documents |
Upload: | marcus-carr |
View: | 214 times |
Download: | 0 times |
Paul Goodman
Introducing Software Metrics via Benchmarking
Introducing Software Metrics via Benchmarking
2© 2004 META Group Consulting 01252 819494 metagroup.com
Topics for Discussion
Metrics Programme risks and reasons for failure
Benchmarking as a means to mitigating these risks
Moving beyond Benchmarking
3© 2004 META Group Consulting 01252 819494 metagroup.com
“80% of software measurement initiatives fail” (Howard Rubin – META Group)
Failure is defined as: (within 2 years);
Initiative is formally cancelled!
Resources are quietly reassigned
The work continues in some way but the results are ignored within the organisation
Introducing Metrics – “A Series of Unfortunate Events” (Lemony Snickett)
Introducing a metrics programme to an organisation is a risky business
4© 2004 META Group Consulting 01252 819494 metagroup.com
Key Reasons for Metrics Programme Failure
Expectations are not managed! Data becomes confused with information – of
which there is a serious lack! We build an ivory tower – we don’t involve the
senior or project managers (the customers of the measurement initiative)!
The devil is in the detail – or how many Function Points can we get on the head of a pin!
Senior management commitment is neither gained nor maintained!
5© 2004 META Group Consulting 01252 819494 metagroup.com
Some Good News
Over the last four years, I have seen more successful measurement initiatives than at any other period (and I have been in this game for a long time)
These have included measurement initiatives in: Financial organisations Local and Central Government sector Telecoms Manufacturing sector
6© 2004 META Group Consulting 01252 819494 metagroup.com
Why the Change?
My observation is that there are two main factors contributing to this greater success: More narrow focus – Don’t try to do too much,
too quickly The client side “pull” for benchmarking in its
various forms, primarily:– Traditional comparative benchmarking– Value for money and viability assessments of
proposals– Outsource contract management
As an industry we could be approaching a 50/50 measurement initiative success rate (so it is
still risky!)
7© 2004 META Group Consulting 01252 819494 metagroup.com
Measurement via Benchmarking
What follows is not just a sales pitch for Meta Group Benchmarking – honest!
Benchmarking brings its own issues and pains but also some significant benefits
First point to note: Benchmarking is based on the provision of
metrics data I want to consider the move to the point where
that data collection becomes inherently beneficial to the organisations providing the data
Can Benchmarking Address the Causes of Failure? (Reverse order)
Can Benchmarking Address the Causes of Failure? (Reverse order)
9© 2004 META Group Consulting 01252 819494 metagroup.com
Management Commitment
Facts of Life: Benchmarking costs money!
(ISBSG is a much cheaper alternative but limited in scope)
Spending money involves management signoff – often at a very senior level
Involving a third party (the “Benchmarker”) has political implications (Non-disclosure agreements at the very least)
First delivery of results is always mandated to be within months – not years! I.e. within the “management window” of 2 years
10© 2004 META Group Consulting 01252 819494 metagroup.com
Management Commitment
All of this generates: Visibility of the benchmarking initiative within
the organisation (Senior) management involvement during the
benchmark procurement Focus on results rather than process (not
good for the long term but we start from where we are)
Which in turn leads to management commitment to the Benchmark
11© 2004 META Group Consulting 01252 819494 metagroup.com
Devils and Detail – Short circuiting the “metrics debates”
Benchmarkers rely on comparability of data across organisations
Benchmarkers utilise standards wherever possible (IFPUG V4.2, Mark II, CMM-I etc)
Base metrics definitions are provided to the client as part of the deal – these are not debatable as comparability would be lost (although support for mapping client definitions to the Benchmark definitions should be provided)
Benchmarkers should be able to support (argue for) their definitions – i.e. they should be workable!
12© 2004 META Group Consulting 01252 819494 metagroup.com
Devils and Detail – Short circuiting the “metrics debates” - Downsides
Benchmarkers cannot be too innovative within their benchmarks E.g. there is currently little benchmark data
around based on COSMIC Some of the hiding places are suddenly not
there (you mean you don’t know how many defects were reported during warranty?)
This can lead to overt or covert resistance to the benchmark which needs to be managed
13© 2004 META Group Consulting 01252 819494 metagroup.com
Avoiding the Ivory Tower
The benchmark depends on a benchmarking model and two data sources:
The model: This may be defined within the benchmark
offering as non-negotiable – management will have been involved in accepting this during the sales negotiation
Definition of the model may be part of the benchmarking process (Meta ADM Benchmark) – involving client representatives
14© 2004 META Group Consulting 01252 819494 metagroup.com
Avoiding the Ivory Tower
Data: The benchmark database from the benchmark
supplier Project and Application Support data from the
client Typically the latter is less readily available than
the client may have believed This often leads to a recognition that basic
management information is lacking It usually needs the involvement of the client
Project Managers to get the data into the benchmark
15© 2004 META Group Consulting 01252 819494 metagroup.com
Avoiding the Ivory Tower
It is common practice to establish a Benchmarking Steering Group
This meets regularly under a defined Terms of Reference
Acts as a Measurement Co-ordination Group for the Benchmark
Will become a driving force for measurement within the client organisation
The Benchmarkers process should be well proven with other client reference sites
16© 2004 META Group Consulting 01252 819494 metagroup.com
Avoiding the Ivory Tower
The benchmarking process should incorporate the collection of contextual project information
Avoidance of ivory towers is achieved through: Involvement of the client in the process The use of a tried and tested process The Benchmarkers experience of applied
measurement in the real world
17© 2004 META Group Consulting 01252 819494 metagroup.com
From Data to Information
The primary focus of the Benchmarking initiative is on the resulting report to management
This report communicates not simply the raw data but meaningful management information, e.g. Comparative costs Productivity Time to Market Delivered Quality Process Maturity
18© 2004 META Group Consulting 01252 819494 metagroup.com
Managed Expectations
It is a fundamental rule that the Benchmarker should manage the expectations of the client with respect to: The scope and limitations of the benchmarking
model The use of a transparent process Involvement of the relevant stakeholders
19© 2004 META Group Consulting 01252 819494 metagroup.com
All is not perfect
During the first benchmarking cycle issues will arise These are likely to include:
Peer Group identification making the client think about what they do:– How is work categorised?– How is this work managed?
The client realising that some basic data is not as readily available as initially assumed
The growing recognition on the part of the client that much of the required data would add value if available on a more regular basis
The latter point can lead to a more general measurement programme requirement!
20© 2004 META Group Consulting 01252 819494 metagroup.com
Programme Set Up
The client can mitigate some of these issues by: Working to identify an agreed (by internal
stakeholders or the client and supplier for outsourced services) Application Portfolio
Establishing the set of “live” projects Ensuring that data (including plan data as well
as actuals) for recently closed projects is retained and is readily available
Establishing the concept of a warranty period for completed projects
21© 2004 META Group Consulting 01252 819494 metagroup.com
Moving Beyond Benchmarking
When the client and the Benchmarker work together to achieve a successful benchmark this often generates the recognition of the added value of metrics data
The Benchmarker will often provide advice on widening the benchmark initiative even if they are not directly involved
The Benchmark Steering Group can be a key catalyst in moving things forward
Client recognition that the measurement function needs resources
22© 2004 META Group Consulting 01252 819494 metagroup.com
Moving Beyond Benchmarking
That software metrics becomes a natural component of software development, support
and enhancement activities within the organisation
Benchmarking truly succeeds when it evolves to become the client’s measurement programme
The Goal