Date post: | 16-Dec-2015 |
Category: |
Documents |
Upload: | orion-rasch |
View: | 222 times |
Download: | 1 times |
1
Performance Metrics
Performance Metrics – What Are They And How Do I Write Good Ones?
Chris HammOperations Director
General Services Administration
GSA Training Conference
and Expo 2010
2
Performance Metrics
This session is sponsored by the
Federal Acquisition Institute
The primary organization providing knowledge and support to the federal civilian acquisition
workforce.
For more information about FAI, please visit our website at www.fai.gov
3
Performance Metrics
Topics
• Introduction to Performance Metrics• Step by Step Process for Creating Metrics• Post Award Management Process• Pitfalls
5
Performance Metrics
Stickk.com & The Biggest Loser
Stickk
Dean Karlan - Economics professor at Yale University Ian Ayres - a Law professor at Yale University Jordan Goldberg - a student from Yale School of Management
The mere act of writing down your goals increases your chances of success. Tying a financial incentive (e.g., make a bet or Commitment Contracts) increases your odds even more!
Proof: Years of rigorous academic research using two of the hardest goals to achieve: Losing weight and quitting smoking.
The Biggest Loser Incredible results are possible with regular measurement, quality assurance
surveillance
6
Performance Metrics
What are Performance Metrics?
They are the short hand for telling where we are.
• Number on your speedometer• Grades your children bring home• Performance results on your service contract
• Customer satisfaction level• Uptime• Usage
A means of measuring that indicates how well our performance objectives are being met
An integral part of your requirements development and performance management strategy
Also called SLAs, KPIs
7
Performance Metrics
Why are we talking about Metrics?
FY08 – 50% of all eligible services acquisitions should be PBA
FAR Subpart 37.6 (b) Performance-based contracts for services shall include—
(1) A Performance Work Statement (PWS); (2) Measurable performance standards (i.e., in terms of
quality, timeliness, quantity, etc.) and the method of assessing contractor performance against performance standards; and
(3) Performance incentives where appropriate. When used, the performance incentives shall correspond to the performance standards set forth in the contract.
8
Performance Metrics
Cost Reimbursable to Fixed Price Continuum
Research Development
Government assumes
more cost riskContractor assumes
more cost risk
Production / Sustainment
As you move to the right, the more detailed and meaningful the performance measures become. . . .
Higher risk, less-defined requirements
CPFF CPFF / CPAF CPIF/CPAF
R&D Development
FPAF/FPIF/ FFP FPIF /FFPEnhancements / Production Operations &
Maintenance
Lower Risk, well-defined requirements
9
Performance Metrics
Performance Metrics are. . . . • A communications tool - The value of an agreement is not just in the
final product; the very process of establishing metrics helps to open up communications
• A conflict-prevention tool - An agreement helps to avoid or alleviate disputes by providing a shared understanding of needs and priorities. And if conflicts do occur, they tend to be resolved more readily and with less gnashing of teeth
• A living document - This is one of its most important benefits. The agreement isn't a dead-end document consigned to the Forget Forever file. On a predetermined frequency, the parties get to review the metric to assess service adequacy and negotiate adjustments
• An objective basis for gauging service effectiveness - A metric ensures that all parties use the same criteria to evaluate service quality
10
Performance Metrics
http://www.acquisition.gov/comp/seven_steps/index.html
12Performance-Based Acquisition©Acquisition Solutions, Inc.
Seven Steps to PBSASeven Steps to PBSA
11
Performance Metrics
Create an Integrated Project Team! The IPT should have your technical expert, stakeholders, financial person, and acquisition. Be sure to include contracting so that you start off on the right strategy and discuss the requirement.
Invite the technical staff that currently have access to any performance data. There is data somewhere. Find it. Get copies of it. Ask the current contractor for it.
Establish the Team
12
Performance Metrics
Interview all internal and external parties to determine the needs, desired outcome, potential solutions, obstacles. Get BUY IN!
Take notes during the meeting, send them out to everybody. Documentation takes time, but eliminates complaints and issues that may arise. You cannot over-communicate. . .
Finalize definitions on the terms you will be using for measuring performance (e.g., Service Level Agreement [SLA], Key Performance Indicator [KPI])
Describe the Problem that Needs Solving
13
Performance Metrics
Conduct Market Research
Research the public sector to see what has worked for your counterparts in other agencies
Locate sources for performance measures. Type what you are seeking into any search engine and you can find many examples. There are dozens of companies that track performance and create standards
Meet with vendors one-on-one to learn about your industry. Ask them what metrics they use
14
Performance Metrics
Due Diligence / Industry Days / RFIs What if you don’t know what you want to measure?
Industry Day: This is our current environment and requirement, we give a presentation to interested vendors.
Due Diligence: The Government releases a draft description of the requirement and opens up the doors to interested parties, helping each side clarify the requirement.
Request for Information: Can include draft requirements, but is geared most to soliciting ideas and options from the vendors.
Industry Day is good, Due Diligence and RFI’s are better, because you can ask for specific information on industry standard measures, metrics, and approaches. Both will increase the quality of your RFP and the vendor proposals.
15
Performance Metrics
How to Include Metrics in the Solicitation
• Draft Incentive/ Award Fee Plan and SLA Format included in Section J
• Option 1: Specify Metrics & have all vendors bid the same• Option 2: Let vendors chose Metrics & targets
• Review requirements with the client before release and see if tasks are sufficiently defined to create notional metrics to include as recommendations
• Remember that the Government’s strongest negotiation position is pre-award.
• Fee Plans are unilateral before the period begins, bi-lateral during the period (if the CO allows any changes at all)
Option # 2 is harder to evaluate but better post award: Better measures, more leverage after award.
16
Performance Metrics
PBA Matrix Template
OutcomesRequiredServices
PerformanceStandards
MonitoringMethods Incentives
17
Performance Metrics
Quality Assurance Surveillance Plan
Evaluation Factor weightings for Section M of RFP
Contract Line Item structure for RFP
PBA MatrixPBA Matrix
Objective Standards AQL Inspection Incentive
Performance Work Statement or SOO
Map the Matrix to the Solicitation
18
Performance Metrics
Decide how to Measure & Manage Performance
• People get really wrapped up in performance measures and SLAs, but this can be made simple if you want it to be.
• Think about the outcomes rather than the processes or org charts
• Use the systems that you have in place
• Only measure things that are meaningful, not things because you can
Bad Example: Cost Savings over IGCE
19
Performance Metrics
19
Contract Types for Measuring
Good Contract Types:
• “Incentive” types (CPIF / FPIP)
• Firm Fixed Price (FFP) and Fixed Unit Price (FUP)
• Award Fee (CPAF / FPAP)
Less Good Contract Types:
• Time & Materials and Labor Hour can have metrics, but cannot have metrics with financial incentives
Regardless of contract type, the Government must still perform quality assurance. The level of effort on surveillance is higher on the good contract types
20
Performance Metrics
Source Selection Process
• Include the vendor’s proposed metrics in the technical evaluation
• Review the feasibility of the measure, whether it is appropriate, meaningful, and end-to-end
• Is the measure entirely within the control of the vendor?
• Most vendor proposed metrics will require some changes after award
21
Performance Metrics
21
Manage Performance (immediately after post award)
Ensure Contract Award includes any contractor proposed metric and/or changes from Discussions and Negotiations
Set up separate meeting from Kickoff Meeting with key stakeholders to review metrics
- Ensure that all stakeholders agree to the metrics
- Set the “weight” of each metric
- Make sure the metric is entirely within the control of the contractor (no mixed responsibility)
- Add further definition of the metric and all exclusions
22
Performance Metrics
Manage Performance (immediately after post award)
Develop the methodology of how to measure performance
Periods (monthly, quarterly, bi-annually) Set Levels (Stretch, primary, secondary, tertiary) Set Measurement Scale with Incentive / Disincentive
– For Example: Stretch: 100% + rollover Primary: 100% of available fee Secondary: 90% Tertiary: 75% Failure: 0 % Fee
Not all levels are required. Some only have one target. It depends on the requirements.
23
Performance Metrics
Manage Performance - Measurement Scale
Award Fee Plan Scale:
Outstanding (95-100) Excellent (90-95) Good (80-90) Average (70-80) Poor (0-70)
SLA Measurement Scale:
Outstanding 99.99+ Excellent 99.9 – 99.99 Good 99 – 99.9 Average 98 – 99 Poor >98
24
Performance Metrics
Manage Performance - Examples of Measurements
Percentages See Chart
Hours Indicate normal business
hours for response!
Days Always clarify business vs.
calendar days!
Yes / No
Measure Downtime in one year
99% 3.7 days
99.90% 8.8 hours
99.99% 53 minutes
99.999% 5.3 minutes
Most projects do not need Five 9s and could not afford it if they wanted it. The cost is prohibitive!NEW EXCEPTION - CLOUD availability
25
Performance Metrics
Manage Performance – a few months post award
•Hold a regular review of the metrics to ensure that the levels are set correctly
•Set up a quality assurance function to check the contractor’s self reporting metrics
•Insist on access to the raw data from the system tools used to generate the metrics reports
•Review all trends and focus on failures
•Root cause analysis can identify process issues
26
Performance Metrics
Manage Performance – Build Dashboard
Summary Scorecard CLIN 0001 Contractor # 2
Dinning
Facility
Supplier
A&AS
Supplier
Cost
Customer Satisfaction
Performance
Small Business Goals
27
Performance Metrics
Manage Performance – a few months post award
•Set a structure for gradual improvement of metrics over time
•For particularly difficult metrics to negotiate, consider benchmarking them for a period without making them SLAs to obtain buy-in and set the appropriate target
•Gradually increase the overall weight of metrics in the fee plan overtime, with the goal of eliminating subjectivity as much as possible
28
Performance Metrics
Manage Performance - Exceptions
• Scheduled Downtime
• Government Intervention in Contractor Response
• Other Contractor Intervention
• Acts of God / Force MajureMake sure someone from the Government is aware of and approves all scheduled downtime or deviations from SLAs. No fox in the hen house!
Another benefit of outsourcing your infrastructure to the “cloud” . . . No more scheduled downtime!
29
Performance Metrics
Manage Performance – Rollover
Good metrics are designed to challenge the vendor to deliver good performance. When the vendor does not achieve the performance goal, the Government has the option of “Rollover” *, where the financial incentive is not lost, but moved to the next period.
If mitigating circumstances exist (e.g., Government redirected performance), Rollover to the next period may be appropriate. If the vendor simply failed to meet the goal, Rollover is not appropriate.
* Interim Rule Section 814 – Reporting on Rollover
30
Performance Metrics
Manage Performance – Pitfalls Metrics should align with the client and project objective AND
motivate good contractor behavior Do not design a metric that, when missed, motivates the
contractor to give up on that measure for the rest of the period
– Example: Doing an “average” over a six month period. If the contractor fails in the 1st month, there is no incentive to maintain service levels or improve over time
Just because something can be measured doesn’t mean it should be a metric Example: It is easy to measure how fast someone picks up a
phone, but a better measure is how long it takes to actually solve the problem
31
Performance Metrics
Manage Performance – Pitfalls
Having too many measures Example: Too many metrics, some of which were redundant
and worth less than 1% of the pool Example: Aligning metrics to Government staff. At least one
metric (and sometimes more) for each of 25 tasks to appease stakeholders
Targets Just like performance reviews, if you set a target, the
contractor will plan and staff to meet that target. If the target is an average, make sure that they plan to make the average, not to always beat the target
32
Performance Metrics
Manage Performance – Pitfalls
Funding
If the metric has an incentive, that amount must be fully funded to prevent any anti-deficiency situation
– This is what makes really good incentives difficult level of oversight
Oversight
COR and Contractor may initially balk at the level of effort required to stand up a metric management process
33
Performance Metrics
Manage Performance – Pitfalls
Over reliance on Industry Standards Benchmarks are great, but that does not mean they are right for your
client’s infrastructure. Just because Gartner says five 9s is normal for a call center doesn’t mean that your requirement needs that
Customer Satisfaction This one is difficult. It will take time to reach agreement on what
scale to use (1-5, Y/N, etc), what survey to use, how to count the no-response pool.
At the end of the day, customer satisfaction may be the most important measure but the hardest to get right
34
Performance Metrics
Learn more about the Federal Acquisition Institute and the resources it offers at www.fai.gov