#atlassian
JOHN CUSTY • MANAGING CONSULTANT • JPC GROUP • @ITSMNINJA
Understanding Metrics:What to Measure and Why
Service Management Practitioner, Consultant And EducatorAbout John Custy
• Distinguished Professional in IT Service Management
• ITIL Expert & ITIL Service Manager
• ITIL Intermediate – SS, SD, ST, SO, CSI, OSA, SOA, PPO, RCV
• KT Certified Instructor• ITIL Accredited Trainer• KCS Verified Consultant• ISO/IEC 20000 Consultant• ISFS, ISMAS based on
ISO/IEC 27002• HDI Faculty & Certified
Instructor
A story about metrics…
Source: Joëlle Flumet’s Broken Bed Installation
Source: betweennapsontheporch.net
The customer rating is 5/5, but the customer
WON’T return!
What are you measuring? Why are you measuring it?
T Y P E S O F M E T R I C S
P U R P O S E O F M E T R I C S
What to Measure and Why
C O M M O N S U P P O RT M E T R I C S
M E T H O D S O F R E P O RT I N G
Q U E S T I O N S
P U R P O S E O F M E T R I C S
How to use metrics
P U R P O S E O F M E T R I C S
How to use metrics• Inform your stakeholders• Report measurements so that stakeholders can understand activities and results• Promote the value of the organization• Determine the best way to communicate the information to the stakeholders• Perform better stakeholder analysis to facilitate stakeholder buy-in• Improve performance - people do what is measured
P U R P O S E O F M E T R I C S
What are we trying to accomplish?
• Account for IT Processes and Deliverables
• Inform stakeholders
• Understand IT Performance
E N S U R E A L I G N M E N T
• Achieve certifications; ISO/IEC 20000, COBIT
• Measure progress to goals/objectives
C O M P L I A N C E
• Measure IT Performance
• Control IT Processes
• Maximize IT Productivity (people)
• Report Costs
• Demonstrate value of IT Organization
O P E R AT I O N A L E X C E L L E N C E
P U R P O S E O F M E T R I C S
What are we trying to accomplish?
The Relationship of Metrics to Goals
The Relationship of Metrics to Goals
V I S I O N
M I S S I O N
G O A L S
O B J E C T I V E S
C S F
K P I
M E T R I C S
M E A S U R E M E N T
P U R P O S E O F M E T R I C S
Sharing Accomplishments
What should you report?
P U R P O S E O F M E T R I C S
Sharing Accomplishments
Key performance indicatorsCritical success factorsVariances to baseline
Progress towards targetsAnnotate milestones and abnormalities
Service improvement projects
What should you report?
Understanding the Different Types of Metrics
D I F F E R E N T T Y P E S O F M E T R I C S
Metrics & Characteristics
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
D I F F E R E N T T Y P E S O F M E T R I C S
Metrics & Characteristics
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
M E T R I C S !
!
•Performance indicators (PI) !
•Key performance indicators (KPI)!
•Key results indicators (KRI)
C H A R A C T E R I S T I C S !!
•Efficiency vs. effectiveness!
•Leading vs. lagging
D I F F E R E N T T Y P E S O F M E T R I C S
Efficiency vs. Effectiveness
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
D I F F E R E N T T Y P E S O F M E T R I C S
Efficiency vs. Effectiveness
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
E F F I C I E N C Y !
!
•How fast?!
•How many?!
•Transactional Cost!
•Incident/Request/Access Management!
•Departmental Goals
E F F E C T I V E N E S S !!
•Accuracy!
•Customer Satisfaction!
•Total Organizational Cost!
•Problem Management!
•Enterprise Objectives
D I F F E R E N T T Y P E S O F M E T R I C S
Quantitative vs. Qualitative
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
D I F F E R E N T T Y P E S O F M E T R I C S
Quantitative vs. Qualitative
Quantitative
• How much or how many
• Ex. The number of times customers contact the service desk
% :)
Q U A N T I TAT I V E !
•How much or how many!
•Ex. The number of times customers contact the service desk
Q U A L I TAT I V E !
•How well something or someone is performing.!
•Ex. Customer Satisfaction, Employee Satisfaction, stock price.
Service Metrics
• Time to process an order • Time to check inventory item • Time to send/receive an e-mail • Time to …
@ITSMNinja
End-to-End Performance
• Cost Per Transaction • Cost Per User
@ITSMNinja
End-to-End Performance
Uptime – compared to …• Downtime per Service
• Frequency and total amount of time• Number of incidents (type/category)• Number of recurring incidents• Time per incident
@ITSMNinja
Service Availability
• Problems identified per Service • # incidents per problem• Lost time per problem
• Changes Per Service • # successful changes (time, budget)• Lost time due to changes – incidents and requests• # Service Requests due to changes• # problems due to changes – IT and business lost time• % improvements due to changes
@ITSMNinja
Service Availability
B A L A N C E N E E D E D
Operational metrics allow you to understand where to improve
Service metrics report on the overall performance
of the service
What Type of Metrics Are Reported?
What Type of Metrics Are Reported?
I T I N T E R N A L M E T R I C S
S E N I O R M A N A G E M E N T
S E RV I C E M A N A G E M E N T
B U S I N E S S U N I T M E T R I C S
R E G U L AT O RY /C O M P L I A N C E
Four Types of Process Metrics
P R O G R E S S E F F I C I E N C Y E F F E C T I V E N E S S C O M P L I A N C E
!I N P R O C E S S
M AT U R I T Y
!U S E O F
R E S O U R C E S
!C O R R E C T A N D
C O M P L E T E T H E F I R S T T I M E
!T O P R O C E S S A N D
R E G U L AT O RY R E Q U I R E M E N T S
Identifying Common Service
Desk Metrics
• Response • % connected immediately (Real-Time)• Abandon Rate• Wait (hold/queue) Time• Average Speed to Answer (ASA)• Response Time service level XX% in YY seconds• Call-Back Time• Desktop (PC)
@ITSMNinjaTypical Operational Measurements
• Resolution • Resolved First Contact• Resolved X hours, Y hours, Z hours• Cases re-opened, Repeats• Requests resolved without assistance (self-help)• Calls/Cases avoided due to self-help
@ITSMNinjaTypical Operational Measurements
• Cost • Call/Contact• Customer/User• Total Cost (TCO) of Support/Service
@ITSMNinjaTypical Operational Measurements
• Service Desk • Volumes, trends• Performance to goals
• Incident Management • Volumes, trends, repeat incidents• Reduction in restoration time• Performance to goals• Customer Satisfaction
@ITSMNinjaTypical Operational Measurements
• Request Fulfillment • Volumes, trends• Time to complete requests• Performance to goals• Customer satisfaction
@ITSMNinjaTypical Operational Measurements
• Customer Satisfaction • Employee Satisfaction
@ITSMNinjaTypical Operational Measurements
@ITSMNinjaTypical Operational Measurements• Knowledge Base Usage • Accesses/Searches per contact• # solutions per search• # solutions searched/opened/viewed• Time spent reviewing solutions• Ease of finding solutions• Quality of solutions (ability to use solutions)
• Service Asset and Configuration Management • Errors in CMDB• Resources improvement utilizing CMDB
• Change Management • Number of incidents/requests due to the change• Additional (reduction) workload due to changes
• Release Management • Number of incidents/requests due to the release• Additional (reduction) in workload due to releases
@ITSMNinjaTypical Operational Measurements
S E RV I C E D E S K M E T R I C S
S E RV I C E D E S K M E T R I C S
W O R K L O A D
•
• Vo l u m e s • C a l l s / C a s e s p e r
c u s t o m e r p e r m o n t h
• N u m b e r o f re g i s t e re d u s e r s /To t a l n u m b e r o f u s e r s
• T i m e s p e n t c o n t a c t i n g u s e r s
• T i m e s p e n t o n c h a n g e re l a t e d i n c i d e n t s / re q u e s t s
S E RV I C E D E S K M E T R I C S
W O R K L O A D
•
• Vo l u m e s • C a l l s / C a s e s p e r
c u s t o m e r p e r m o n t h
• N u m b e r o f re g i s t e re d u s e r s /To t a l n u m b e r o f u s e r s
• T i m e s p e n t c o n t a c t i n g u s e r s
• T i m e s p e n t o n c h a n g e re l a t e d i n c i d e n t s / re q u e s t s
I N D I V I D U A L S
• N u m b e r o f c a l l s t a k e n
• Av e r a g e H a n d l e T i m e ( A H T )
• Av a i l a b i l i t y • O c c u p a n c y • N u m b e r o f
i n c i d e n t s / re q u e s t s c l o s e d o n f i r s t c o n t a c t
• C u s t o m e r S a t i s f a c t i o n
• C o n t r i b u t i o n t o K n o w l e d g e b a s e
S E RV I C E D E S K M E T R I C S
W O R K L O A D
•
• Vo l u m e s • C a l l s / C a s e s p e r
c u s t o m e r p e r m o n t h
• N u m b e r o f re g i s t e re d u s e r s /To t a l n u m b e r o f u s e r s
• T i m e s p e n t c o n t a c t i n g u s e r s
• T i m e s p e n t o n c h a n g e re l a t e d i n c i d e n t s / re q u e s t s
I N D I V I D U A L S
• N u m b e r o f c a l l s t a k e n
• Av e r a g e H a n d l e T i m e ( A H T )
• Av a i l a b i l i t y • O c c u p a n c y • N u m b e r o f
i n c i d e n t s / re q u e s t s c l o s e d o n f i r s t c o n t a c t
• C u s t o m e r S a t i s f a c t i o n
• C o n t r i b u t i o n t o K n o w l e d g e b a s e
C U S T O M E R S
• C u s t o m e r S a t i s f a c t i o n
• F re q u e n c y o f s u r v e y i n g , N u m b e r n o t re s p o n d i n g
• Vo l u m e s • C a l l s / C a s e
S E RV I C E D E S K M E T R I C S
W O R K L O A D
•
• Vo l u m e s • C a l l s / C a s e s p e r
c u s t o m e r p e r m o n t h
• N u m b e r o f re g i s t e re d u s e r s /To t a l n u m b e r o f u s e r s
• T i m e s p e n t c o n t a c t i n g u s e r s
• T i m e s p e n t o n c h a n g e re l a t e d i n c i d e n t s / re q u e s t s
I N D I V I D U A L S
• N u m b e r o f c a l l s t a k e n
• Av e r a g e H a n d l e T i m e ( A H T )
• Av a i l a b i l i t y • O c c u p a n c y • N u m b e r o f
i n c i d e n t s / re q u e s t s c l o s e d o n f i r s t c o n t a c t
• C u s t o m e r S a t i s f a c t i o n
• C o n t r i b u t i o n t o K n o w l e d g e b a s e
C U S T O M E R S
• C u s t o m e r S a t i s f a c t i o n
• F re q u e n c y o f s u r v e y i n g , N u m b e r n o t re s p o n d i n g
• Vo l u m e s • C a l l s / C a s e
R E S P O N S E
• Av e r a g e S p e e d t o A n s w e r ( A S A )
• % c a l l s a n s w e re d l i v e v s . . q u e u e d
• C a l l b a c k t i m e • A b a n d o n R a t e
( A B A ) • R e s p o n s e s w i t h i n
S e r v i c e L e v e l & O u t s i d e s e r v i c e l e v e l
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
R E S O L U T I O N
•
• I n c i d e n t c l o s u re ( f ro m t i m e o f s u b m i s s i o n )
• M e a n T i m e f o r S e r v i c e R e s t o r a t i o n ( M T S R ) f o r L e v e l s 1 , 2 , & 3
• I n c i d e n t s m a t c h e d ( K E )
• I n c i d e n t s R e -O p e n e d
• C l o s e d F i r s t C o n t a c t
• E s c a l a t i o n s f o r re s o l u t i o n
• R e m o t e t o o l u t i l i z a t i o n
• D e s k - s i d e v i s i t s • I n c i d e n t s c l o s e d
v i a s e l f - h e l p
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
R E S O L U T I O N
•
• I n c i d e n t c l o s u re ( f ro m t i m e o f s u b m i s s i o n )
• M e a n T i m e f o r S e r v i c e R e s t o r a t i o n ( M T S R ) f o r L e v e l s 1 , 2 , & 3
• I n c i d e n t s m a t c h e d ( K E )
• I n c i d e n t s R e -O p e n e d
• C l o s e d F i r s t C o n t a c t
• E s c a l a t i o n s f o r re s o l u t i o n
• R e m o t e t o o l u t i l i z a t i o n
• D e s k - s i d e v i s i t s • I n c i d e n t s c l o s e d
v i a s e l f - h e l p
V O L U M E
• To t a l n u m b e r o f i n c i d e n t s / re q u e s t s ( b y p r i o r i t y & c a t e g o r y )
!• S e c u r i t y re l a t e d
i n c i d e n t s
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
R E S O L U T I O N
•
• I n c i d e n t c l o s u re ( f ro m t i m e o f s u b m i s s i o n )
• M e a n T i m e f o r S e r v i c e R e s t o r a t i o n ( M T S R ) f o r L e v e l s 1 , 2 , & 3
• I n c i d e n t s m a t c h e d ( K E )
• I n c i d e n t s R e -O p e n e d
• C l o s e d F i r s t C o n t a c t
• E s c a l a t i o n s f o r re s o l u t i o n
• R e m o t e t o o l u t i l i z a t i o n
• D e s k - s i d e v i s i t s • I n c i d e n t s c l o s e d
v i a s e l f - h e l p
V O L U M E
• To t a l n u m b e r o f i n c i d e n t s / re q u e s t s ( b y p r i o r i t y & c a t e g o r y )
!• S e c u r i t y re l a t e d
i n c i d e n t s
R E S P O N S E T I M E
• S e r v i c e D e s k p e r f o r m a n c e
!• L e v e l 2 / 3 – s a m e
a s S D m e t r i c s
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
R E S O L U T I O N
•
• I n c i d e n t c l o s u re ( f ro m t i m e o f s u b m i s s i o n )
• M e a n T i m e f o r S e r v i c e R e s t o r a t i o n ( M T S R ) f o r L e v e l s 1 , 2 , & 3
• I n c i d e n t s m a t c h e d ( K E )
• I n c i d e n t s R e -O p e n e d
• C l o s e d F i r s t C o n t a c t
• E s c a l a t i o n s f o r re s o l u t i o n
• R e m o t e t o o l u t i l i z a t i o n
• D e s k - s i d e v i s i t s • I n c i d e n t s c l o s e d
v i a s e l f - h e l p
V O L U M E
• To t a l n u m b e r o f i n c i d e n t s / re q u e s t s ( b y p r i o r i t y & c a t e g o r y )
!• S e c u r i t y re l a t e d
i n c i d e n t s
R E S P O N S E T I M E
• S e r v i c e D e s k p e r f o r m a n c e
!• L e v e l 2 / 3 – s a m e
a s S D m e t r i c s
E S C A L AT I O N
• T i m e t o e s c a l a t e !
• % E s c a l a t e d t o c o r re c t g ro u p !
• Te c h n i c a l & H i e r a rc h i c a l
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T
C U S T O M E R S AT I S FA C T I O N
• I n c i d e n t , R e q u e s t & A c c e s s M a n a g e m e n t p ro c e s s e s
•
• N u m b e r o f u n i q u e u s e r s !
• Av e r a g e t i m e p e r u s e r !
• # p a g e s v i e w e d
S E L F - S E RV I C E
Methods to Report Performance
P E R F O R M A N C E R E P O RT S
Methods to Report Performance
B A L A N C E D S C O R E C A R D
S U P P O RT S C O R E C A R D
Factors to Consider When Reporting
Factors to Consider When Reporting
• Who are the stakeholders? • How does what you are reporting impact the stakeholders? • Reports must be easy to read and understood, thus they need to be developed with
the stakeholder in mind.• Reports need to show how the support center is contributing to the goals of each
stakeholder and the business.• Reports must identify the appropriate channels to communicate with each of the
stakeholders.
Daily Report Examples
Trending Report Examples
Trending Report Examples
Activity Report Examples
Present your achievements with
scorecards
Scorecards Drive Transformation
High (Business Transformation)Low (Process Improvement) Moderate
I m p r o v e m e n t G o a l
Rel
evan
t to
IT
Ho
li
st
ic
Gen
eral
Spe
cific
T C OISO/IEC17799
ITIL/ISO/IEC20000 C M M
COBIT
Six Sigma
ISO/IEC 9000
Malcolm Baldrige Award
Scorecards
Standards help us to ensure that IT is aligned to meet business objectives
• Simple indicator • Reference base • Measures the key issues • Reports on progress to goals
@ITSMNinjaSupport Scorecard
Scorecard Criteria: Operational Performance
Overall performance
Response time
Resolution time
Closed first call
Abandon time
Wait time
Status time
Backlog aging
C O L U M N T I T L E C O L U M N T I T L E C O L U M N T I T L E
Scorecard Criteria: Operational Performance
Overall performance
Response time
Resolution time
Closed first call
Abandon time
Wait time
Status time
Backlog aging
C O L U M N T I T L E C O L U M N T I T L E C O L U M N T I T L E
Scorecard Criteria: Operational PerformanceSERVICE SCORECARD The Acme Support Center Scorecard provides a weekly report of performance on our Service Level commitments.
CS Overall Support Center Product
Key Service Area Goal Actual
Response Time Front-line Measurements:• Call Pick-Up Time - All incoming calls are answered by a support consultant• Call Waiting Time - Average is less that 3 minutes• Back-line Measurements:• Non-Accepted Call Back Time - All customers not responded to on the initial
call by a support consultant will be called back within 30 minutes.
80%3 minutes
90%
Resolve Time Resolved on First Contact - 30% resolved first call (4 month goal is 50%)Resolved Same Day - 40% resolved within 1 business day (4 month goal is 60%)Resolved Same Week - 85% resolved within 5 business days
30%40%85%
Status Priority 1 Issues - Customer provided status every 4 hours until resolvedPriority 2 Issues - Customer provided status every 24 hours until resolved or workaround providedCall Aging - Manage backlog so that no more than 20% over 2 weeks and 5% over 30 days
80%80%80%
Backlog (Average age of open items) 3 days10 days
Event Survey Overall satisfaction rating on a 1-5 scale 4.1
C O L U M N T I T L E
Service Desk Scorecard
Overall performance
Response time
Resolution time
Closed first call
Abandon time
Wait time
Status time
Backlog aging
C O L U M N T I T L E
Customer sat/quality
Overall satisfaction
Response satisfaction
Resolution satisfaction
Status satisfaction
Improvement goals
Alignment to business
C O L U M N T I T L E
Acme Service Desk Scorecard
Balanced Scorecard
Business Goals
Financial PerspectiveProvide a good return on investment on IT-enabled business investments
Manage IT-related business risks
Improve corporate governance and transparency
Customer Perspective
Improve customer orientation and service
Offer competitive products and services
Establish service continuity and availability
Create agility in responding to changing business requirements
Achieve the cost optimization of service delivery
Obtain reliable and useful information for strategic decision making
Internal Perspective
Improve and maintain business process functionality
Lower process costs
Provide compliance with external laws, regulations and contracts
Provide compliance with internal policies
Manage business change
Improve and maintain operational and staff productivity
Learning and Growth PerspectiveManage product and business innovation
Acquire and maintain skilled and motivated personnel
C O L U M N T I T L E
IT Balances Scorecard @ITSMNinja
Source: ITIL Continuous Service Improvement
You cannot manage what you cannot CONTROL
You cannot control what you cannot MEASURE
You cannot measure what you cannot DEFINE
What do you need to measure? What should you be doing you do with the metrics you produce?
Thank you!
John Custy • Managing Consultant • JPC Group • @ITSMNinja
Join me tomorrow @2PM to learn
Knowledge Centered Support (KCS) – The Methodology that Really Works