Poll QuestionPoll QuestionBUSINESS 2.0 BUSINESS 2.0 Dream Team*Dream Team*
CEOCOOCFOCMOCIO
SVP – Industrial DesignSVP – R&DSVP – Online SalesSVP - Procurement
*BUSINESS 2.0 July 2004
Who’s missing?a. No one
b. SVP – Corporate Services
c. Chief Learning Officer
d. Michael Jordan
Sample of Sample of Dream Team Dream Team AccomplishmentsAccomplishments
• Saved $1.8 billion in 18 months and increased sales from $10 million to more than $85 million in five years
• Increased shareholder base from less than 15 percent institutional ownership to more than 90 percent
• Increased number of new patents• Reduced 4 payroll systems to 1• Reduced 11 billing systems to 2 in 24 months
Meet Donald HodgesMeet Donald Hodges•President, GlobalCom University
•15 years experience in adult learning
•Handpicked by previous CEO
•Wants to impress new CEO with program results
Donald’s results as reported to new CEO Developed 10 new programs
Offered 1,120 hours of training Had 1,500 employees attend training
Received on average 4.5 out of 5 on the program satisfaction rating
Donald’s results as reported to new CEO Developed 10 new programs
Offered 1,120 hours of training Had 1,500 employees attend training
Received on average 4.5 out of 5 on the program satisfaction rating
Poll QuestionPoll Question
How likely is Don to be recruited for a position on the Dream Team?
a. No way.
b. Possibly, with a little improvement.
c. Absolutely!
11 ObjectivObjectiv
ee
To help you beginToday we have
1. Effectively communicating results on a routine basis
2. Developing an ROI implementation strategy
3. Determining the ROI on the ROI
What do we mean by What do we mean by resultsresults??
Level
1. Reaction & Planned Action
2. Learning
3. Application
4. Impact
5. Return on Investment
Measurement Focus
Measures participant satisfaction with the program and captures planned actions
Measures changes in knowledge, skills, and attitudes
Measures changes in on-the-job behavior
Measures changes in business measures
Compares program benefits to the costs
Eco
nom
icS
yste
mIn
divi
dual
ROI MethodologyROI MethodologyAdapted from: Phillips, J. J., Stone, R. D., Phillips, P. P. (2001) Adapted from: Phillips, J. J., Stone, R. D., Phillips, P. P. (2001) The Human Resources Scorecard: Measuring the Return on InvestmentThe Human Resources Scorecard: Measuring the Return on Investment. Boston, MA: . Boston, MA: Butterworth-HeinemannButterworth-Heinemann
Develop EvaluationPlans and
Baseline Data
Develop EvaluationPlans and
Baseline Data
Stage 1Evaluation Planning
Stage 2 Data Collection
Collect DataDuring SolutionImplementation
Collect DataDuring SolutionImplementation
Collect DataAfter Solution
Implementation
Collect DataAfter Solution
Implementation
DevelopObjectivesOf Solution
DevelopObjectivesOf Solution
Level 1
Level 2
Level 3
Level 4
TabulateCosts
Of Solution
TabulateCosts
Of Solution
Isolate theEffects ofSolution
Isolate theEffects ofSolution
Convert Data to Monetary
Value
Convert Data to Monetary
Value
Calculatethe Return On
Investment
Calculatethe Return On
Investment
Implement Communication
Process
Implement Communication
Process
Stage 3 Data Analysis
Identify Intangibles
Identify Intangibles
Stage 4 Communicate
Results
Level 5
Intangible Benefits
Reaction & Planned Action
Learning
Application & Implementation
Impact
ROI
Isolate the Effects of the Program
Report the complete story of success.
Intangible Benefits
Developing a communication Developing a communication strategy requires us to answer strategy requires us to answer
three questions.three questions.
1. What do we want?
2. Who can give it to us?
3. How best can we ask?
Common Target AudiencesCommon Target Audiences
What do we want? Who can give it to us?Secure approval for program Client, top executives
Gain support for the program Immediate managers, team leaders
Build credibility for the training staff Top executives
Enhance reinforcement of the program Immediate managers
Enhance results of future programs Participants
Show complete results of the program Key client team
Stimulate interest in training programs Top executives
Demonstrate accountability for client
expenditures All employees
Market future training programs Prospective clients
How best can we ask?How best can we ask?
• Ongoing Summaries
• Complete ROI Impact Study Report
• Executive Summary
• One-page Report
• Training Scorecard
How? What? Who? When?
Ongoing Summary Brief report of program success
•Training staff
•Evaluation team
•Participants
One week after program (L1 & L2)
One week after data are collected (L3 & L4)
Complete Report Details of evaluation project
•Training staff
•Evaluation team
•Client
•Senior Managers*
30 days after data are collected
Executive Summary Brief overview of evaluation project
•Training staff
•Evaluation team
•Client
•Participants
•Senior Managers*
30 days after data are collected
One-Page Report Micro-level scorecard of program success
•Training staff
•Evaluation team
•Senior Managers*
•Supervisors
30 days after data are collected
(Use with caution!)
Complete ReportComplete Report
• General Information• Methodology for Impact Study• Data Analysis• Costs• Results• Barriers and Enablers• Conclusions and Recommendations• Exhibits
When communicating with senior managers consider
two questions:
Do they believe
you
Can they
take it
Communication ProgressionCommunication Progression
First 2 ROI
Studies
3-5 ROI Studies
6 Plus ROI
Studies
Detailed Study
Executive Summary
One Page Summary
Meeting
No Meeting
No Meeting
Training ScorecardTraining Scorecard
• Provides macro-level perspective of success
• Serves as a brief report versus detailed study
• Shows connection of training’s contribution to business objectives
• Integrates various types of data• Demonstrates alignment between
programs, strategic objectives, and operating goals
Training scorecards include Training scorecards include seven categories of data.seven categories of data.
• Indicators• Reaction and Planned Action• Learning• Application• Business Impact• ROI• Intangibles
Developing an ROI strategy Developing an ROI strategy requires us to answer four requires us to answer four
questions.questions.1. Do we need it?
a. Our assessmentb. Our clients’ assessment
2. What are our desired outcomes?a. What are the consequences of our implementing an ROI strategy?b. What are the negative consequences if we do not implement an
ROI strategy?
3. Why?a. What purpose will implementation success serve us?b. What purpose will implementation success serve our team?c. What purpose will implementation success serve the organization?
4. How can we get there?a. What are the perceived barriers?b. How can we overcome them?c. What is one action we can take today to move forward?
Are you a candidate for ROI? Are you a candidate for ROI? Sample Assessment QuestionsSample Assessment Questions
• Is your organization large with a variety of programs?
• Is your organization undergoing significant change?
• Is there pressure from senior management to measure results?
• Is the image of your training function less than desirable?
• Is there an increased focus on linking training to strategy?
For complete assessment go to Tools & Templates at www.roiinstitute.net.For complete assessment go to Tools & Templates at www.roiinstitute.net.
Potential Outcomes of ROI Potential Outcomes of ROI ImplementationImplementation
• Eliminate unnecessary programs• Increase (or maintain) training budget• Improve image with line supervisors• Reallocate resources to allow for staff
development• Reallocate resources to build evaluation
infrastructure• Expand implementation of successful
programs
If you implement an ROI strategy, what outcomes do you expect?
What are the consequences of your implementing this strategy?
If you do not implement this strategy, what are the negative consequences?
Exercise 1
Purpose drives the initiative to Purpose drives the initiative to achieve outcomes. achieve outcomes.
Implementation of our evaluation strategy gives us the information we need to help develop our people, improve our processes, and contribute to the bottom line – literally!
We ensure that taxpayer dollars are allocated to the programs that directly support the agency mission.
What purpose will achieving the ROI strategy outcomes
serve you, your team, and your organization?
Exercise 2
Poll QuestionPoll Question
Which of the following represents the biggest barrier to your implementing ROI?
1.Confidence to implement ROI2.Management support3.Cost4.Time5.Resources (people, technology)
Cost-Saving Approaches to ROI
• Plan for evaluation early in the process
• Build evaluation into the training process
• Share the responsibilities for evaluation
• Require participants to conduct major steps
• Use short-cut methods for major steps
• Use sampling to select the most appropriate
programs for ROI analysis
• Use estimates in the collection and analysis of
data
• Develop internal capability to implement the
ROI process
• Utilize web-based software to reduce time
• Streamline the reporting process
Cost-Saving Approaches to ROI
Actions to make ROI work Actions to make ROI work includeinclude
• Planning and discipline– Establish goals and
targets– Assign responsibilities
• Improve needs analysis process
• Implement cost-savings approaches
• Communicate progress• Engage Management
– In Programs– In Evaluation
• Develop staff skills– ROI Networks– Workshops– ROI Certification– Competency groups– Read books/case
studies/articles– Develop a case study– Teach the process to others
What is one action you can take today to move forward with ROI implementation?
Exercise 3
So what is the ROI on ROI?So what is the ROI on ROI?
Maximum Use
$ High ROI $
Success with ROI implementation is related to the extent to which results are used for process improvement!
Success with ROI implementation is related to the extent to which results are used for process improvement!
Individual Perspective (Levels 1 and 2)
•Are facilitation scores improving?
•Are programs perceived as more relevant and important?
•Do content and program design better reflect adult learning principles?
•Is there evidence that learning is taking place at a higher rate?
System Perspective (Level 3)
•Do supervisors better support training?
•Are systems in place to better support the learning transfer?
•Is frequency of use of knowledge and skills increasing?
•Are there fewer barriers to learning transfer?
When developing the ROI on When developing the ROI on the ROI consider results from the ROI consider results from
all perspectives!all perspectives!
When developing the ROI on When developing the ROI on the ROI consider results from the ROI consider results from
all perspectives!all perspectives!Economic Perspective (Level 4 and 5)
• How many unnecessary programs have we eliminated?
• Have we decreased the cost of training while improving the quality of our programs?
• How much has our budget increased?
• When comparing cost savings and budget increases to the cost of ROI implementation, do we see a positive economic return?
Join the Join the Dream TeamDream Team
CEOCOOCFOCMOCIO
SVP – Industrial DesignSVP – R&DSVP – Online SalesSVP - Procurement