Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A s s e s s f o r S u c c e s s
Questionmark Conference 2018 Savannah
Beyond Recall: Taking Competency Assessments to the Next Level
John Kleeman
Questionmark Executive Director and Founder
January 13, 2018
38 minutes later…
Jan 23, 2018
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Is it possible to assess someone’s abilities to make judgements and decisions
when faced with a dilemma?
Slide 9
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Different kinds of cognitive skills we can
assess
Observational assessments
Introduction to Situational
Judgment Assessments
(SJAs)
How to author SJAs
A practical exercise
How Questionmark
can be used for SJAs
Agenda
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Does anyone currently use Situational Judgment Questions/Tests?
Or use other ways to assess judgment?
Question to the room
Slide 11
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What we can learn from assessments
Slide 12
Did they like it?
Did they learn it?
Can they do it?
Can you prove it?
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
13
Knowledge (a cognitive Skill)
• Memory is a cognitive skills that provides ability to recall knowledge:
• “Knowledge How”
• “Knowledge To”
Above Knowledge Skills
• Cognitive skills above recalling knowledge(Comprehension , Application, Analysis, Synthesis, Evaluation and Judgment)
Observable Skills
- Motor Skills i.e. physical movement/coordination
- Inter-personal skills
Behaviors & Attitudes
• Motivation
• Ability to achieve goals
• Team fit
Ability to Do/Perform“Competency”
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Recall questions check knowledge and slow the Forgetting Curve
Slide 14
Time
Kn
ow
led
ge/S
kills
Lear
nin
g Ex
per
ien
ce
Lear
nin
g Ex
per
ien
ce
Lear
nin
g Ex
per
ien
ce
Qu
esti
on
s
Qu
esti
on
s
Qu
esti
on
s
Qu
esti
on
s
Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve
Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Blooms taxonomy describes cognitive skills in levels (revised version shown here)
Most real world jobs require many levels of taxonomy
If your assessments focus only on recall, may well not test job competence
create
evaluate
analyze
apply
understand
remember / recall
Recall only small part of job requirements
Slide 15
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What does a yellow traffic light mean?a) Stop
b) Go
c) Caution
d) Look behind you
“Recall” example
Slide 16
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
If you are driving toward an intersection and the light turns from yellow to red, what should you do?
a) Speed up and cross the intersection
b) Continue at the same speed and cross the intersection
c) Stop suddenly
d) Stop gradually
“Understand” example
Slide 17
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
You are giving a close friend a lift to work and because of traffic you are running 15 minutes late for an important appointment. You are driving toward an intersection with traffic lights on it. You can see the intersecting road to some extent and there seem to be no cars on it. The light turns to yellow as you approach. The car behind you sounds their horn and seems to be accelerating in the expectation that you are going to cross the intersection. Your friend tells you to put your foot down as you are running late and there are no cars visible on the intersection. What should you do?
a) The road appears to be clear so speed up and cross the intersection to exit the intersection as soon as possible
b) Stay at the same speed and continue to cross the intersection while keeping an eye out for crossing cars
c) The light is about to turn red so you should stop immediately and ignore the car behind you as they will need to stop as well
d) Put your hazard lights on to signal to the car behind you and stop at the intersection if the lights facing you turn red
e) Ignore your friend’s irritation and move across to the side of the road to let the other car pass
“Evaluate” or “Judgment” example
Slide 18
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
William Coscarelli and Sharon Shrock: ▪ Knowledge (recall) level items are by far the
easiest to write.
▪ Developing tests that truly reflect on-the-job performance requires … writing items at the higher cognitive levels
▪ In general, the single most useful improvement you can make in writing test items is to write them above the memorization level.
Advice from the experts
create
evaluate
analyze
apply
understand
remember / recall
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Beyond Multiple Choice…
Slide 20
Also reduces chance to guess. In a 4 question multiple choice
question, 25% chance of guessing. In these questions 3%
chance of guessing
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Observational Assessment
Slide 21
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Observational Assessment
Observer “rates”the participant
Example Applications
Sales Training
Certification
Medical and Dental (OSCE)
Equipment operation
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Monitor/observer logs in
Selects assessment
Selects participant to be observed
Answers/submits responses to questions
Results/feedback reported to stakeholders
Typical workflow for “observer”
Slide 23
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Items typically solicit ratings for:
• Performance of task(s)
• Demonstration of appropriate knowledge, skills, abilities, behaviors
Goal is to ensure ratings are as objective as possible
• Keys to ensuring reliability of observational assessment results:
• Establish consistent rating scales & scoring rubrics
• Educate raters to ensure consistent understanding and application of rating scales
Getting objective ratings from Observational Assessments
Slide 24
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Testing Judgment
Slide 25
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
An unannounced drill was initiated Employees provided with conflicting information:
▪ “properly includes the drill language ‘EXERCISE, EXERCISE, EXERCISE’”-- but --
▪ “erroneously contains the text of an EAS message for a live ballistic missile alert, including the language, ‘THIS IS NOT A DRILL.’”--”
U.S. Sen. Brian Schatz: “The reports from the FCC and the state of Hawaii demonstrate systems and judgment failures on multiple levels”
FCC Report suggests this was dilemma requiring judgment
Slide 26
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Rules immensely important
• in defining the parameters within which teams and individuals should operate
Rules alone are not enough
• Whether and how they are adopted by people when making decisions that matters
“Judgement is at the heart of nearly every business scandal that ever occurred”
Why judgment matters (Ernst & Young)
Slide 27
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Almost all job roles require judgment Situational Judgement Assessments present work-relevant dilemmas
with questions that require judgments about possible responses Assessing how well participants can:
▪ Recognize critical aspects of a situation and how they should be prioritized▪ Identify which actions/options are more vs. less likely to achieve objective▪ Determine which responses are more vs. less aligned with organizational expectations▪ Display the soft skills required work collaboratively with colleagues to overcome obstacles and
achieve desired outcomes▪ Deal with dilemmas in which there are no clear ‘positive outcomes’ -- or in which available
information may be ambiguous
Assessing for Situational Judgment
Slide 28
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
A score or evaluation is
made
The participant
chooses from options
Dilemma that needs judgment
Context is defined
Structure of an Situational Judgment Assessment (SJA)
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
You work in the back office in the team approving new customers ensuring that the organization’s procedures have been followed (such as credit rating and know your customer). Your manager is away on holiday this week. A senior manager in the company (three levels above you) comes into your office and says that there is an important new customer who needs to be approved today as they want to place a big order, and that he can vouch that the customer is good. You review the customer details, and one piece of information required by your procedures is not present. You tell the senior manager and he says not to worry, he is vouching for the customer. You know this senior manager by reputation and have heard that he got a colleague fired a few months ago when she didn’t do what he asked.
Select the most and least effective of the following options:▪ Take the senior manager’s word and approve the customer▪ Call your manager’s cellphone and interrupt her holiday to get advice▪ Tell the manager you cannot approve the customer without the information needed▪ Ask the manager for signed written instructions to override standard procedures to allow you to approve the
customer
Example SJA question - compliance
Slide 31
Make a judgment
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Pre-hire
• Realistic job preview –help applicant and employer work out job fit
Post-hire workforce development
• Diagnosing training needs
• As training with feedback
• Post training assessment
• Measurement of soft skills
• Evaluating job readiness
Post-hire compliance
• Assess judgment of workforce
• Survey what people in organization would do in dilemmas
• Spot possible problems and mitigate risk
Certification
• Many high-tech certification programs criticized for being too focused on basic knowledge, not enough on important job skills
• SJAs provide a practical and inexpensive way of testing judgment
Applications of Situational Judgment
Slide 32
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Genuinely measures judgment in a way seen by all as fair
Lower cost compared to many other approaches
Scalable – can delivered widely
Benefits
Slide 33
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
How to create Situational Judgment Assessments (SJAs)
Slide 34
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Attributes / Competencies
Situations Dilemmas Item stemResponse options
You need SMEs (Subject Matter Experts) to author and review SJAs as only they will know what judgments are needed in the job role and how to set up realistic and relevant dilemmas
Authoring Process for SJAs
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Commitment to Professionalism
• Dealing with issues of confidentiality such as hearing a colleague talking about a patient outside of work
• Challenging inappropriate behavior such as a consultant speaking to a colleague/patient in an inappropriate way
Coping with Pressure
• Knowing how to respond when you make a mistake such as providing wrong medication to patient
• Seeking help when not sure of the correct procedure/best way of doing things
Effective Communication
• Gathering information and communicating your intentions to nursing staff or other colleagues
• Listening and effectively communicating such as with an unhappy patient or relative
Patient Focus
• Identifying that a patient’s views and concerns are important and they should have input into their care
• Spending time trying to understand a patient’s concerns and empathising with them
Working Effectively as Part of a Team
• Recognising and valuing the skills and knowledge of nursing staff, when faced with a disagreement about a patient’s care
• Offering assistance and support to colleagues when they are not able to handle their workload
Example Attributes for SJAs (UK medical doctors)
Slide 36
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Ask Subject Matter Experts▪ Do you do task or supervise it
▪ How Difficult is it to do task
▪ How Important to do task
▪ How Frequently to do task
Identify what to assess
Job Task Analysis Survey can help determine Job Needs
Slide 37
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Item writing workshop / Survey SMEs Brainstorm situations that discriminate
between effective/appropriate behavior and less effective
Include real life examples Analyze, sort and frame
dimensions/topics Good for ensuring that real life
instances of judgment identified
Start with a job analysis and competencies required
Item writing workshop / Survey SMEs
For each competency, identify situations for each competency where demonstrated/not demonstrated
Good for ensuring competencies covered
Two ways for SMEs to identify Situations for SJAs
Slide 38
Critical Incident Analysis Competency-based Approaches
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Choice needs to be made
Using judgment for the choice is clearly linked to a needed domain of knowledge, skill or competency in the job role
Requires professional judgment to answer well. (Question discriminates between more and less capable people)
Key Aspects of a Dilemma
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
What the participant thinks is best to do:▪ Choose the most effective response▪ Choose the most and least effective
response
Best for higher stakes and competency testing
What the participant thinks he/she would actually do:▪ Which response best typifies what you
would do in this situation▪ Choose the response that best reflects how
you would respond and least how you would respond
Best for understanding participant behavior, e.g. in recruiting or learner diagnostics or compliance surveys
Response Options for SJAs
Slide 40
“Should” “Would”
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Select most appropriate choice▪ Simplest cognitively▪ Easiest to guess
Select most and least appropriate choice▪ Checks knowledge of what is best and worst▪ A good solution
Select x best choices▪ For example, choose the 2 best options out of 5
Rank the choices▪ Hardest for participant▪ Challenge to design as experts have to agree the
best ordering of all choices
Rate each choice ▪ E.g. agree/disagree/neither agree nor disagree that
someone in your organization would do this▪ Good for surveys including in compliance
Five different kinds of response options
Slide 41
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
A dilemma is more than a situation/scenario – must be a choice to be made linked to required domain of knowledge/ skill/ competency/ compliance
SJAs can be mixed with other questions or presented on their own. You should allow 1-2 minutes time in the test to answer an SJA question
Best to use a response type that requires more than just “choose the best” to reduce chance of guessing▪ With 4 choices, there is a 25% chance of guessing a “choose the best” and an 8% chance of
guessing “choose the best and the worst”
Usual guidelines on item writing (avoid ambiguity, responses similar length etc. apply)
You need consensus from SMEs as to what the best choice is. A dilemma where experts disagree on the best answer is not useful
SJA Item Writing Advice
Slide 42
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Important to survey SMEs not involved in the item development▪ Present the item stem
▪ Present the response options
▪ Ask them to rate
Compare their ratings with the scoring you have developed and adjust choices as necessary
Validating with other SMEs or experts
Slide 43
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
For medium or high stakes use of SJAs, just like other questions, you should▪ Present SJAs to a pilot group
▪ Conduct item analysis
▪ Look carefully at each question’s statistics and remove/adjust questions which need it
Pilot and run Item Analysis
Slide 44
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Exercise
Please get in small groups to work together
Slide 45
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Suppose we were looking to write SJAs for assessment professionals, what kind of situations/dilemmas come up which need judgment when you are authoring, delivering and managing assessments?
Please get into groups and identify▪ A situation and context
▪ A dilemma where
There is a choice
Which needs judgment
And requires context
▪ You may want to think about a real issue that has happened to you
Critical Incident Analysis
Slide 46
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Let’s take an example dilemma and discuss possible response options together
Brainstorm an SJA question
Slide 47
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
How Questionmark can help
A reliable, secure, scalable assessment management system for SJAs
Slide 48
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Planning assessment▪ Use Job Task Analysis Surveys to
help set attributes / competencies
▪ Survey SMEs to identify or rate dilemmas
▪ Set up a topic structure to base your assessment on
Questionmark can be used throughout SJA lifecycle
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Capability to author and score all SJA item types including “Most and least”
Collaborative authoring environment where SMEs can easily review questions and see how they change
Manage item bank including SJAs Shuffle choices and/or question
selection
Authoring items and Assembling assessment
Slide 50
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Include SJAs as experimental questions within live tests to gain data to validate question
Use comment box to get feedback from participants or SMEs
Use the Item Analysis Report to identify and improve weak questions
Pilot and Review
Slide 51
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
SJAs easily callable from LMSs and other systems
Instant scoring and feedback Scalable and robust Secure against breaches and
cheating Effective UI and smartphone
support
Comprehensive reports to make professional decisions
Coaching report to share with participants
Test and item analysis reports to aid validity/reliability
Delivery and Reporting
Slide 52
Delivering SJAs Reporting on SJAs
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Example Questions: http://www.questionmark.com/go/example-sja
Slide 53
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Summary
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Assessments that test recall:▪ Check someone’s knowledge
▪ Give retrieval practice which reduces forgetting
Competence in most jobs requires testing a lot more than recall:▪ Testing higher level cognitive skills
▪ Testing practical skills (observational assessments)
▪ Testing judgment (SJAs)
create
evaluate
analyze
apply
understand
remember / recall
Beyond Recall
Slide 55
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
A score or evaluation is
made
The participant
chooses from options
Dilemma that needs
judgment
Context is defined
Dilemma
Requires choice to be made
Using judgment for choice clearly linked to needed
domain in job role
Requires professional judgment to answer well
SJAs an Effective means of Testing Judgment
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Measure and Enhance Learning
•Measure judgment
•Identify job readiness
•Diagnostic assessments
•Improve learning and performance
Qualification and Certification
•Deliver reliable, defensible tests to certify employees
•Include measurement of judgment
•Ensure sound personnel decisions
Regulatory Compliance
•Measure judgment in your personnel
•Survey to assess what personnel think others would do
•Get early warning of potential issues that could lead to a compliance failure
Sales Channel Verification
•Measure not just how sales people know the product, but also their softer skills and judgment
•Increase revenues and customer satisfaction
Technical Channel Verification
•Measure judgment of technical and service professionals
•Reduce costs and improve customer satisfaction
Credentialing
•Relatively inexpensive way of extending coverage of certification programs to higher order and softer skills for candidates
SJAs can be used in many Applications
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Can you see applications of SJAs within your organization?
If so, what actions will you take to start using?
Group discussion
Slide 58
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Assessing for Situational JudgmentDesigning, deploying and getting value from
Situational Judgment Assessments
By Eugene Burke & John Kleeman
To learn more, see our white paper
Assessing for Situational Judgment
Slide 59
40-page white paper provides:▪ Overview of Situational Judgment
Assessments (SJAs)
▪ Benefits and applications of SJAs
▪ Techniques for creating and validating SJAs
▪ Example SJA items to illustrate key concepts
www.questionmark.com/learningresources
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Questionmark Conference 2018 ❖ Savannah
Your questions