How to Design Skill Certifications Like a Pro
Liberty J. Munson, PhD.Principal PsychometricianMicrosoft Worldwide Learning
Brought to you by
2
Shreya Sarkar-Barney, PhD.
CEO & Founder
Liberty J. Munson, PhD.Principal Psychometrician
3
Leadership
Excellence
Talent
Management
Excellence
US India
We help organizations achieve better outcomes through talent using
science, analytics, and empathy.
Drive business impact
through timely talent
actions
Evidence-based solutions
for measurable and
sustained improvements
in leadership
The Zoom Web Conferencing Platform
Raise Hand to comment by
voice
Chat for Tech
support or to
comment
To test/adjust
computer
audio
4
If you require HCG
or HRCI
recertification
credits for your
participation, please
be sure to join with
your full name.
18.52
44.44
40.74
a. Yes, they are required to perform certain
roles
b. Certifications are preferred but not required
c. Certifications are not required for any of the
roles
33.33
25.93
7.41
37.04
a. Employees are encouraged to earn
industry recognized external certifications
b. We develop role/skill specific
certifications in-house
c. We use external and internally
developed certifications
d. We have no formal requirements
around certifications
29.63
25.93
11.11
37.04
a. Employees become certified after
demonstrating they can perform their role
b. Employees become certified after completing
required training
c. Employees must pass a test to become
certified
d. Employees must pass a test and demonstrate
their skills on the job to become certified
29.63
25.93
11.11
37.04
a. Employees become certified after demonstrating
they can perform their role
b. Employees become certified after completing
required training
c. Employees must pass a test to become certified
d. Employees must pass a test and demonstrate
their skills on the job to become certified
29.63
14.81
3.70
22.22
33.33
a. Technical roles (e.g. programmers, IT, cyber
security)
b. Management roles (e.g. Project management)
c. Finance (e.g. risk management)
d. Sales (e.g. Product knowledge, sales skills)
e. Functional roles (e.g. HR, Legal, marketing)
25.93
22.22
22.22
33.33
a. Options for developing a certification program
and the tools required
b. The strategic value of assessments and
certifications in a learning culture
c. The relationship between certifications and role
readiness
d. How to build and maintain an assessment
program to promote a learning culture
Data Driven: • Participate in hacks to better
understand learner and test taker behaviors
• Evaluate the effectiveness of initiatives to improve learning
• Work with teams to better surveys that will results in clear recommendations
Learner Obsessed: • Ensure validity, reliability,
psychometric soundness of all skills validation processes (exams, assessments, etc.)
• Ensure that we are creating quality content
• Maintaining and sustaining cloud-based exams
Driving Impact:• Reduced the costs of exam
development by more than 50%• Reduced development time by 30%• Defined process to keep cloud-
based exams up to date• Regularly monitor exam quality
(psychometrics) and passing rates to quickly identify issues before they affect candidates, partners, and trainers
• Identified approaches to drive closer alignment between exams and training
• Ensure exams are valid and reliable drives overall value of certification program
• Thought leader in certification industry—driving changes in the way industry thinks about certification
Invest in Innovation:• MOOC to cert pilot• Adaptive testing/learning pilot• Investigating how AI, ML, and bots
could fundamentally change exams and certification
• Innovating to develop content more quickly to support cloud-based exams and internal needs
Simplify: Developing repeatable and scalable processes to minimize internal resources to develop and sustain exams
Overview of Microsoft’s Certification Program
Enabling the next generation of innovators, entrepreneurs, and developers
Enable Customers
& Partners at Scale
Learning Credits
Badges
Microsoft Professional Degree
Machine Learning
Microsoft Azure Fundamentals
Programming in C#
MOC On Demand
Creative Coding through Games and Apps
Harvard/MSFT CS50.AP
C# for Absolute Beginners
Hour of Code
Minecraft
Drive Usage, Deployment& Consumption
Empower Students& Educators
Activate Cloud &
Productivity Partners
2
5
Adoption
Drive Deep Engagement
Advocacy
Enable Capability Mastery
Affinity
Spark Excitement
Total reach
New/Competitive Developers reached
Skilled technologists building on Microsoft platforms and services
Preference for Microsoft platforms & services
1 3 4
Drive adoption of Microsoft technologies through training and certification
Similar paths for Office:
• Microsoft Office Specialist
• Microsoft Office Expert
• Microsoft Office Master
Microsoft Certification Structure
CLOUD PLATFORM & INFRASTRUCTUREMCSECloud Platform & InfrastructureEarned: 2017
MCSA Windows
Server 2012
MCSA Linux on
Azure
MCSA Windows
Server 2016
MCSA Cloud
Platform
Elective
Choose two from:
532: Developing Microsoft Azure Solutions
533: Managing Microsoft Azure Infrastructure Solutions
534: Architecting Microsoft Azure Solutions
473: Designing and Implementing Cloud Data Platform Solutions
475: Designing and Implementing Big Data Analytics Solutions
LFCS:
Linux Foundation
Certified System Administrator
533: Managing
Microsoft Azure
Infrastructure Solutions
741:
Networking with
Windows Server 2016
740:
Installation, Storage, and
Compute with
Windows Server 2016
411:
Administering
Windows Server 2012
410:
Installing and Configuring
Windows Server 2012
412:
Configuring Advanced
Windows Server 2012
Services
742:
Identity with
Windows Server 2016
Exam 1
Exam 2
MCSA
MCSAElective Exam
MCSE/D
MCSE/D
Different Elective or
Retake Exam
MCSE/D + Year
Renewed
• Certifications include achievement date
• Signifies investment in staying up to date on the technology
• Every year, they can re-earn their MCSE certification by passing an additional exam from the list of electives
• Option to retake exams where technology is rapidly changing either to renew their MCSE or to simply show currency in the technology
Renewing Certifications & Retaking Exams
An Overview of MS Certification Exams
Computer administered
Test center deliveries or online proctoring
Global distribution
Ongoing delivery
Variety of item types (e.g., multiple choice, drag
and drop, active screen, hot area, case studies,
labs, text entry, code analysis, etc.)
22
Development Process
Objective
DomainBlueprint
Item Development
Research
Candidate
Manager
Alpha Review &Item Revision
Field Testing (a.k.a. Beta Exam)
Forms Assembly
LocalizedExamStandard
Setting
Item Selection
Publish Exam
Ongoing Psychometrics
Focus on “application” level questions rather than “remembering” or
“understanding” level questions
Keys to Successful Evaluation
• Application of knowledge, skills, and abilities, not recall, understanding, etc.
• Present a scenario describing a business problem or situation that requires application of skills in ways that are not directly covered in the training
• Ask about a real-world application of a concept covered in the training
• Require the learner to:
• Analyze and provide a solution for a complex problem
• Combine several concepts, ideas, or skills to solve the problem
• Identify and analyze patterns based on a limited amount of information or data
• Design a solution by drawing on multiple skills learned during the training, and/or
• Evaluate outcomes that may result from actions taken
• Assess ability to apply knowledge to real-world problems, and/or
• Ask about concepts, actions, and outcomes that require extrapolating beyond what is specifically covered in the training or linking information from multiple sources
Key Attributes of Well Designed Measurement
Continental Congress
A. John Hanson
B. John Adams
C. Thomas Jefferson
D. George Washington
Stem = Question
Options = Key +
Distracters
Distracters =
Wrong Answers
Key = Correct
Answer
Stem
Format
• Open
• The year that John
Adams was elected
President:
• Closed (Preferred)
• In what year was John
Adams elected
President?
Best Practices
• Succinct –remove unwanted
language
• Relevant and important
• Non-trivial
• Stem is NOT teaching
• Avoid using “Not” and “Except”
• Avoid using definitions in stem
• Avoid two questions at once
Quality Check
• Can you cover the options and
answer the question?
• What is the capital of France?
• A. Lyon
• B. Paris
• C. Normandy
• D. Orlean
Key: Best Practices
• Don’t be different from
the other options such as
the longest and technical
• Don’t use words that are
in the stem and repeated
only in the key = cueing
Distractors: Best
Practices
• Incorrect
• Plausible
• Common
misconceptions
• No overlap
• All options begin with the same part of speech (e.g., gerund, infinitive)
• Example - A) Jumping B)Ran C)To flee D) marathons
Formatting
• Alpha-numerical
• Example - A. 5.0% *B. 25.0% C. 6.5% D. 3.0%
Ordering
• A) 10-30lbs B) 20-60lbs C) 55-95lbs
Overlap
Proper spelling, grammar and punctuation
• Don’t use you
• Example – If you want to distribute shares, the statue you would be most concerned with is:
Use third person
• Do NOT use: “always,” “never,” “only,” “appropriate,” “proper,” “typically,”
“usually,” and “might”
Avoid Specific Determiners
• Unless you are asking “best” answer questions with clear documentation
supporting what is “best”, do not use “best,” “most,” and “least” because
they are subjective
Avoid Specific Modifiers
Do NOT use All or None of the Above
• Spell out acronyms first, unless they are understood by the professional
community
Acronyms
• Write using the fewest number of words to answer the question
• Example – The best part of junk food is how good it tastes. According to the FDA, how
much sugar should be consumed daily?
Be Succinct
• Place all common wording in the stem that appears in all the options
• A. Health Care Food Act
• B. Dietary Act
• C. American Health Eating Act
• D. Anti Junk Food Act
Eliminate Repetition
• Avoid specific cultural, gender, ethnic, religious, or personal pronoun references
• Example – A Japanese female decides to calculate how many calories a Christian eats. She
needs to determine the square feet immediately after the 9/11 tragedy for the entire Word
Trade Center including the chapel and mosque? What is the square feet of the Word Trade
Center that she needs to calculate if the dimensions are 10 x 10 feet?
Ensure Test Fairness
How Can This Apply in Your Organization?
Microsoft’s Technical Credentialing Program
Creating a Learning Culture
What is it?
•Set of organizational values,
processes, and practices that
encourage employees to increase
knowledge & competence
Built on a growth mindset
•Belief that potential is not fixed, that failure is on the
path to mastery, willing to take risks
•See gaps in their knowledge as opportunities to
learn
•See learning as a journey
How?
•Establish clear links between
learning and performance
•Reward appropriate risk taking
•Ensure that learning is applied to
the job
•Make employees more
accountable for their own learning
path
Overview Ensuring these employees
build and validate their
technical knowledge and
skills on an ongoing basis
and in conjunction with
broad readiness goals
1
Implementing technical
credentialing requirements
on a consistent,
predictable, annual rhythm
2
Presenting role-based
requirements to the field
and their managers that
are clear and actionable
3
1.00
1.20
1.40
1.60
1.80
2.00
Very Effective Effective Somewhat
effective
Not at all
Effective
0
500
1000
1500
2000
2500
3000
3500
4000
Azure certification Dynamics
certification
Office and
productivity
certification
Windows and
mobility
certification
All MSFT FY17 target audience
• 5,200+ technical sales professionals required to meet technical credentialing requirement to demonstrate readiness
• 76% find certification useful to job effectiveness
Since launch of first Azure initiative in FY15 over 20,000 certifications achieved
Interesting: Those who selected exams are more likely to believe that the
tech requirement has improved job effectiveness
An ExampleTi
mel
ine
Q1 Q2 Q3 Q4 Q1
FY18 FY19May 31Sept 1
Key Changes
• Exam requirement replaced with a combination of
exams and courses
• Pass 1 exam or 3 courses to fulfill your requirement
• Additional options (new exams, courses, and activities)
may be added as they are made available
Same as FY17
• Where: Requirement appears in My Learning
• Who: Applicable to those in role by September 1
• Recommended for those who enter role after that date
• When: Complete by May 31
Role Business Intelligence TSP Collaboration TSP
Course
Options
Complete any three of relevant Open EdX courses
• 6 available to date - Excel, Power BI, O365
• Additional courses/activities TBA throughout FY18
Complete any three of relevant Open EdX courses
• 23 available to date - Exchange, SharePoint, O365
• Additional courses/activities TBA throughout FY18
Exam Options • Pass any one of the following exams:• 70-346: Managing Office 365 Identities and
Requirements*
• TBA Power BI exam (late FY18)• NOTE: Practice exams and prep resources will likely not
be available for some time after release!
• Pass any one of the following exams:• 70-339: Managing Microsoft SharePoint 2016
• 70-345: Designing and Deploying Microsoft
Exchange Server 2016
• 70-346: Managing Office 365 Identities and
Requirements*
• 70-347: Enabling Office 365 Services*
OR OR
Characteristic Assessment Certification
Primary purpose Provide instruction/training to aid in
the acquisition of
knowledge/skills/competencies
(learning through instruction)
Assess knowledge/skills/
competencies that have already
been acquired (validation through
testing)
Purpose and scope of evaluation
process
Evaluate accomplishment of
intended learning outcomes of a
specific education/training program
Confirm mastery of the
knowledge/skills/competencies
required to effectively perform a job
function or occupational/
professional role
Duration of program Ends when certificate is awarded Ongoing; requirements must be met
on a routine basis to maintain
credential (recertification)
Recognition Appears on transcript; may be
related to badging but no official
acronym follows learner’s name
Recipient uses an acronym or letters
after his/her name to highlight
certified status
Formative vs. Summative Assessment: Driving a Learning Culture
Formative assessment: Allows
the learner to attempt problems
multiple times, often providing
hints, so s/he can identify areas
of weakness for ongoing
learning
Summative assessment:
Evaluation if test taker has the
skills, learned the course
objectives, etc.
Combining Learning with Evaluation: Moving Beyond Certificate to Certification
Formative assessments built
in the course through
knowledge checks,
homework, etc. that allow for
multiple attempts
Summative assessment is end
of course exam requires
application of skills attained
during the course; it is MORE
than an evaluation of learning
objective attainment
Skill set 1 validated
Course & Mini Exam #1
Course & Mini Exam #2
Work Project #1
Skill set 2 validated
Course & Mini Exam #3
Course & Mini Exam #4
Work Project #2
The Certification Journey in Your Organization Could Be…
Wednesday, April 25th, 2018
48
Thank You!For more information contact
Shreya Sarkar-Barney, Ph.D.
49
www.linkedin.com/company/human-capital-growth
@hcgtm
707.317. 7644
www.Humancapitalgrowth.com
Follow us
@sarkarbarney
https://www.linkedin.com/in/sarkarbarney/