Beyond Measuring Usage - Tools & Techniques for Demonstrating Program Value SkillSoft and Knowledge...

Beyond Measuring Usage - Tools Beyond Measuring Usage - Tools & Techniques for Demonstrating & Techniques for Demonstrating Program ValueProgram ValueSkillSoft and Knowledge AdvisorsSkillSoft and Knowledge Advisors

Learning: Re-Imagined

Workshop AgendaWorkshop AgendaWorkshop AgendaWorkshop Agenda

• Welcome and Introduction

• Metrics in Times of Economic Uncertainty Jeffrey Berk - Knowledge Advisors

• Metrics That Matter ™ Case Study –CNATom Simon and Mike Castillo - CNA

• SkillSoft Tools & Techniques Heather McCain, Laura Rexford, Lori Turley - SkillSoft

• Wrap up

Economic Uncertainty Emphasizes Human Capital Measurement

Jeffrey Berk

KnowledgeAdvisors

© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 4Apr 19, 2023

Measurement in Economic UncertaintyMeasurement in Economic Uncertainty

Agenda

Measurement for Performance Management


Business Case for MeasurementBusiness Case for Measurement

© 2008, KnowledgeAdvisors

Bumpy Economic Times

S&P 500 Down Double Digits $700 Billion Bailout DJIA Record Daily Losses Unemployment on the Rise Well Known Brands in

Trouble

© 2008, KnowledgeAdvisors 5Apr 19, 2023


What are Business Leaders Saying?


Source: UK Commission for Employment and Skills, November 10, 2008

• Firms that don’t train are 2.5 times more likely to fail than those that do train.

•“Investing now in building new skills will put us in the strongest position as the economy recovers.”

•“The skills of our people are our best guarantee of future prosperity - - and the best investment a business can make in challenging times.”


The Economy Changed in 16 Years19920.6:1

20085:1

Tangible Assets

Intangible Assets

MarketValue

MarketValue

Intangible Assets

Tangible Assets

Intangible/Tangible = HCI(Human Capital Index)

66% Productivity Gain in 16 Years

Gary Becker Wins Nobel Prize in Human Capital

S&P 500 Grows at Record Levels

Organizations Invest More in L&D, Leadership, HRIS, LMS, Analytics, Performance Management,

Engagement & Recruiting


Measuring Talent Maximizes Economic Value

Shareholder Returns of companies investing in L&D measurement have 15.18% higher returns than the S&P 500

Source: Human Capital Measurement and Its Impact on Stock Performance, KnowledgeAdvisors, 2008


Identify Gaps in Human Capital Processes

PE

RF

OR

MA

NC

E

RE

CR

UIT

ING L

&D

TA

LE

NT

LE

AD

ER

SH

IP

EN

GA

GE

ME

NT

PE

RF

OR

MA

NC

E

Forecast Workforce RequirementsRecruit, Select, and Hire International AssignmentMobile WorkforceEmployee Turnover

On-boardingL&DCoachingKnowledge Mgmt

Competencies Assessment

Succession PlanningLeadership Development

Compensation and BenefitsEmployee SatisfactionEmployee EngagementWork/Life BalanceWorkforce Diversity

Performance Appraisal


Case: Talent Gap

04/19/23 © 2008, KnowledgeAdvisors 10

Background: Printing and publishing company

Business Issue: Gaps identified in skilled resources for changing business strategy

Measurement Solution: Provided general skills competency set, collected and reported data w/analysis

Results: Data used for L&D strategy and for recruiting job descriptions.


Case: Leadership Evaluation


Background: Major health benefits provider

Business Issue: Turnover in first 90 days of employment was 25 to 30%

Measurement Solution: Built a leadership program for the Blue Chip hires and used smarter evaluations to measure it and improve it

Results: 93% of Blue Chip new hires were retained and no longer flight risks. 73% reduction in admin burden in the measurement process itself

“Metrics that Matter has provided objective data that supports “gut feelings” about where improvements can be made and where things are going well.”

-Director of Leadership & Associate Development


Case: Template Links L&D to Sales


Background: Consumer Packaged Goods firm

Business Issue: Needed reasonable evidence of sales impact from training

Measurement Solution: Used actual results and reasonable assumptions via impact template

Results: Proved that $74,256/person of sales dollars were tied to a $5,000/person training investment.

Example of template used to showcase ROI from Sales trainingSource: KnowledgeAdvisors


Case: ERP Technology Company

Apr 19, 2023 © 2008, KnowledgeAdvisors 13

Background: ERP company performing customer education

Business Issue: Show value of training to customers in terms of increased productivity and quality

Measurement Solution: Collected 12,770 end of program evaluations within 6 months asking participants to forecast impact

Analytics Results: 18.2% estimated increase in productivity from the training (adjusted down for self-reported bias)

Business Results: ProductivityPost

EventTotal % improvement in productivity including training 51.0%Training's contribution to improved productivity 57.0%Percent improvement due to training 29.1%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 18.9%

Business Results: QualityPost

EventTotal % improvement in quality, including training 54.0%Training's contribution to improved quality 57.0%Percent improvement due to training 30.8%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 20.0%

Business Results: Customer SatisfactionPost

EventTotal % improvement in customer satisfaction including training 52.0%Training's contribution to improved customer satisfaction 57.0%Percent improvement due to training 29.6%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 19.3%

Analyze & monetize this result





Agenda





Fact

1. Most organizations invest 1% to 4% of revenue in talent development source ASTD stated L&D is 2 to 3% of revenue

2. Most organizations waste 50% or more of their talent development expenses source Metrics that Matter 600,000 follow up data points is 55%, IDC sites

50% waste, Brinkerhoff cites 80% or more waste

3. Most organizations waste .5% to 2% of revenue on ineffective talent development calculation from item 1 x 2 above

4. Effective investments in predictive analytics increases performance by 2.4% source IDC

Talent Measurement Results: Increases Performance * Reduces Waste * Slashes Admin Burden


Fact

According to IDC, programs that use predictive analytics solutions, such as Metrics that Matter, generate 60% better returns than those that don’t.

Example: If one invests $1,000,000 in an L&D program that generates a $1,000,000 profit, without using a predictive measurement solution, effective use of one, could have generated an additional $600,000 in profit.


Measurement Impact on Increased Performance

1. Major services firm realized $200,000 increased annual profit by using Metrics that Matter predictive analytics capabilities.

2. Through measurement a leading federal agency significantly increased output and business impact without increasing budget over a 3 year period.

3. Major telecommunications firm using Metrics that Matter was able to increase sales by $1.25 million dollars through better supply chain management.

4. 15,000 hours in improved productivity by a floral company in identifying appropriate training investments.


Measurement Impact on Reduced Waste1. 25 retail stores avoided $100,000 of wasted training. Retail store training

measured after first of six weeks saw 1 set of ineffective trainers and discontinued using them.

2. Instructor compared herself to other instructors and changed approach. 6 months of additional sessions performed better to avoid at least $100,000 in scrap learning.

3. $500,000 in funding gained by measuring scrap learning. Services firm measured training results and found high quality but low impact. The program was retained and additional funding rewarded to consult with managers on increasing impact.

4. 88% of learners not applying training is NOT due to bad training. Waste is due to unsupportive employee environment. A billion dollar company can save at least $1,000,000 if it can reduce waste by just 20% by performance managing the employee environment.


Measurement Impact on Reduced Admin

1. 75% reduction in administration costs from automating measurement process vs. manual measurement.

2. 15 seconds to retrieve actionable data for decision-making vs. 15 hours prior to automating the measurement process. This automation of 1 report alone saved over $36,000 per year.

3. $400,000 savings by major accounting firm that used smarter measurement in a practical way to yield similar results compared to complex and burdensome alternative measurement techniques


Calculate Performance Gain

$1,000,000,000

Revenue

25% Labor Percentage

$250,000,000 Labor Cost

1%Increased Productivity from Measurement%

$2,500,000 Increased Performance from Measurement


Calculate Waste Reduction

Description

$1,000,000,000

Revenue

1% L&D as % of Revenue

$10,000,000 L&D Investment

50% Waste or Unrealized Value %

$5,000,000 Waste or Unrealized Value

5%Waste Reduction from Measurement%

$250,000Waste Reduction from Measurement


Calculate Admin Savings

Description

$55,000 Salary of 1 FTE admin for metrics

$50,000Cost avoided from unnecessary analysis

$50,000Cost to replication fraction of MTM benchmarks

75%Admin savings from MTM measurement and benchmarks

$116,250Administration Savings from Measurement


Total Value

Value Summary

$2,500,000 Performance Gain

$250,000 Waste or Scrap Avoided

$116,250 Administration Savings

$2,866,250

Total Quantified Value



Agenda





Talent and L&D Measurement Tips

Identify and Audit Gaps in Human Capital Processes

Build a Robust Evaluation Process

Track and Link to Actual Business Results

Reduce Errors and Scrap Leverage Technology Use Metrics for

Performance Management



Benchmarks for Performance Management

Benchmarks: Internal or External

Provide a point of referenceMotivate by exampleAvailable thru Metrics that Matter


Goals for Performance Management

Goals: Actual to Goal

Set in Metrics that MatterMove in right directionChallenging yet attainable


Trends for Performance ManagementTrends: Time based analysis

Available in Metrics that MatterShows improvement over time


Dashboards for Performance Management

Dashboards: •Concise Data•Summary Data•Visual View•Available in Metrics that Matter for course managers, instructors, impact and operations


A template for the learning team to conduct more comprehensive business impact and ROI analysis. Templates bring stewardship and collaborative guidance to the process of linking training to results.

Track & Link to Actual Business Results


Build a Robust Evaluation Process



Agenda





Experts in metrics that

matter

Experts in metrics that

matter KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions.

KnowledgeAdvisors


KnowledgeAdvisors Clients


© 2008, KnowledgeAdvisorsApr 19, 2023 Page 35

www.knowledgeadvisors.com

For further information on the information contact:

Jeffrey Berk

Chief Operating Officer

+1 312 676 4411 (phone)

+1 312 676 4401 (fax)

[email protected]

Measurement at CNA

Measurement at CNATom Simon & Michael Castillo Area of Presenter

Measurement

CNA At A Glance

•1 million business and professionals insured•7th largest commercial lines insurer•U.S. and international operations•9,000 employees•$7.8 billion in revenues•Assets of $52 billion•“A” rated for financial strength by A.M. Best

Measurement

Measurement

Measurement

Measurement

Measurement

Why ROI?Why ROI?Why ROI?Why ROI?

Benefits of ROI:

• Business cases benefit strongly from ROI

• Measuring financial impact helps illustrate importance of learning

• Value-based measurements support Learning Growth Model progression

Challenges of ROI:

• Limited time and resources

• Hard to evaluate ROI for intangibles (learning)

• Securing internal data difficult

SkillSoft Measurement ToolsSkillSoft Measurement ToolsSkillSoft Measurement ToolsSkillSoft Measurement Tools

• EZ ROI Calculator - “soft measure”

• Business Impact Analysis – Validated by internal survey

• Value Measurement Toolkit (VMT) – ROI, Phillips Level 5

Business Business Impact Impact

AnalysisAnalysis

EZ ROI EZ ROI CalculatorCalculator

ValueValue Measurement Measurement

ToolkitToolkit

time, resources, data involvedtime, resources, data involved

estimate vs. rigorous measureestimate vs. rigorous measure

EZ ROI CalculatorEZ ROI CalculatorEZ ROI CalculatorEZ ROI Calculator


AnalysisAnalysis



ToolkitToolkit



Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and Attributes

Objectives:• Deliver more value-based metrics • Help strengthen business cases• Prospective use (Should we train?) and retrospective use

(How did we do?)

Attributes:• Easy to use• Requires little time and very little data • Defensible • Immediate analysis• Cost displacement and performance improvement measures

EZ ROI CALCULATOREZ ROI CALCULATOREZ ROI CALCULATOREZ ROI CALCULATOR

Components:

• Introduction and assumptions

• Cost displacement calculator

• Cost displacement sample

• Productivity impact calculator

• Productivity impact sample

Microsoft Excel Worksheet

Application Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI Calculator

• SkillSoft versus another learning option

• Monetary value of productivity gains

• Limited licenses? Select wisely

Business Impact AnalysisBusiness Impact AnalysisBusiness Impact AnalysisBusiness Impact Analysis


AnalysisAnalysis



ToolkitToolkit



Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and Attributes

Objectives:

• Measure satisfaction, determine motivation and determine job impact by the following:

• Determine individual skill requirements and needs• Evaluate program satisfaction• Gauge skill improvement (based on individual

responses/group roll-up) to determine productivity gains

Attributes:• Easy to use • Company-specific • Requires limited time and data• Longer timeline• Stronger measurement story• Potential testimonials

Business Impact AnalysisBusiness Impact AnalysisBusiness Impact AnalysisBusiness Impact Analysis

Components

• Custom survey

• Communications to launch the survey

• Summary of feedback collected

• Impact/Results Summary


Sample Survey QuestionsSample Survey QuestionsSample Survey QuestionsSample Survey Questions

1.Job Category Drop Down of Job Families

2.Salary Range Drop Down of Salary Ranges

3.Would you recommend the SkillSoft resources to a colleague?

Select One(Yes, Somewhat, No)

4.What was the main reason why you accessed SkillSoft training?

Drop down with list of likely options plus an “other” option that has a blank field

5.Primarily, what skill were you seeking to develop? Open short text field

6.What percent of your job requires you to use this skill?

Drop down with a range of percentages(0%, 1-10%, 11-20%, 21-30%, 31-40%, 41-50%,

51-60%, 61-70%, 71-80%, 81-90%, 91-100%)

7.As a result of the training, how much of a skill increase did you experience?

Drop down with a range of percentages(0%, 5-10%, 11-15%, 15-20%, 21-25%, 26-30%,

More than 30%)

8.If you did not have access to SkillSoft resources, what alternatives would you have considered?

Drop down of common training types including “no alternatives” and “other” options

9.In approximate terms, how much do you think the alternate option would have cost?

Open numerical field, includes the options “no cost” and “not sure”

10.When are you planning to use SkillSoft resources again?

Select One (Not sure, within 2 weeks, within 30 days, within 90

days, more than 90 days)

Output of SurveyOutput of SurveyOutput of SurveyOutput of Survey

• Survey Results document including:• Background• Survey Responses (graphical representation)• Comment Highlights• ROI Impact

Application Ideas for Business Impact Application Ideas for Business Impact AnalysisAnalysis

Application Ideas for Business Impact Application Ideas for Business Impact AnalysisAnalysis

• Translate productivity gains into monetary terms specific to your company

• Utilize survey results for ongoing marketing and increased usage

• Create targeted programs based on skill needs identified

Value Measurement ToolkitValue Measurement ToolkitValue Measurement ToolkitValue Measurement Toolkit


AnalysisAnalysis



ToolkitToolkit



Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and AttributesObjectives:

• Measure the value derived from the implementation and ongoing support of learning initiatives.

• A method offered to aid clients to establish a process for measurement.

• Deliver value in terms your stakeholders will believe in – cot savings, productivity impact, increased revenue, etc.

Attributes:• Longer term study based on a defined process• Requires internal, validated data• Defensible and credible due to stakeholder input• Return on investment

Notice that the 5 levels are not each a step in the process. The process can be used to any level.

The Value Measurement ProcessThe Value Measurement ProcessThe Value Measurement ProcessThe Value Measurement Process

• White Paper

• Readiness Worksheet

• Planner

• Calculator

• Case Studies

Value Measurement ToolsValue Measurement ToolsValue Measurement ToolsValue Measurement Tools


Microsoft Word Document

Application Ideas for Value Measurement Application Ideas for Value Measurement Application Ideas for Value Measurement Application Ideas for Value Measurement

• Validate the investment in technology enabled learning

• Meet stakeholder expectations/shape expectations

• Measuring key strategic programs with high visibility

Hands-On ExerciseHands-On ExerciseHands-On ExerciseHands-On Exercise

• Divide into teams

• Review your scenario

• Agree on appropriate tool

• Be prepared to present considerations for tool selection

Thank you for participating!Thank you for participating!Thank you for participating!Thank you for participating!

Date post:	30-Dec-2015
Category:	Documents
Upload:	lee-sherman
View:	215 times
Download:	0 times

Beyond Measuring Usage - Tools & Techniques for Demonstrating Program Value SkillSoft and Knowledge...

Documents