+ All Categories
Home > Documents > Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The...

Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The...

Date post: 07-Sep-2018
Category:
Upload: vanngoc
View: 218 times
Download: 0 times
Share this document with a friend
59
Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged. Assess for Success Questionmark Conference 2018 Savannah Beyond Recall: Taking Competency Assessments to the Next Level John Kleeman Questionmark Executive Director and Founder
Transcript
Page 1: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

A s s e s s f o r S u c c e s s

Questionmark Conference 2018 Savannah

Beyond Recall: Taking Competency Assessments to the Next Level

John Kleeman

Questionmark Executive Director and Founder

Page 2: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

January 13, 2018

Page 3: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A
Page 4: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A
Page 5: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

38 minutes later…

Page 6: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A
Page 7: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Jan 23, 2018

Page 8: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A
Page 9: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Is it possible to assess someone’s abilities to make judgements and decisions

when faced with a dilemma?

Slide 9

Page 10: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Different kinds of cognitive skills we can

assess

Observational assessments

Introduction to Situational

Judgment Assessments

(SJAs)

How to author SJAs

A practical exercise

How Questionmark

can be used for SJAs

Agenda

Page 11: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Does anyone currently use Situational Judgment Questions/Tests?

Or use other ways to assess judgment?

Question to the room

Slide 11

Page 12: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

What we can learn from assessments

Slide 12

Did they like it?

Did they learn it?

Can they do it?

Can you prove it?

Page 13: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

13

Knowledge (a cognitive Skill)

• Memory is a cognitive skills that provides ability to recall knowledge:

• “Knowledge How”

• “Knowledge To”

Above Knowledge Skills

• Cognitive skills above recalling knowledge(Comprehension , Application, Analysis, Synthesis, Evaluation and Judgment)

Observable Skills

- Motor Skills i.e. physical movement/coordination

- Inter-personal skills

Behaviors & Attitudes

• Motivation

• Ability to achieve goals

• Team fit

Ability to Do/Perform“Competency”

Page 14: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Recall questions check knowledge and slow the Forgetting Curve

Slide 14

Time

Kn

ow

led

ge/S

kills

Lear

nin

g Ex

per

ien

ce

Lear

nin

g Ex

per

ien

ce

Lear

nin

g Ex

per

ien

ce

Qu

esti

on

s

Qu

esti

on

s

Qu

esti

on

s

Qu

esti

on

s

Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve

Purpose:• Memory retrieval practice• Strengthens memory recall• Reduce Forgetting Curve

Page 15: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Blooms taxonomy describes cognitive skills in levels (revised version shown here)

Most real world jobs require many levels of taxonomy

If your assessments focus only on recall, may well not test job competence

create

evaluate

analyze

apply

understand

remember / recall

Recall only small part of job requirements

Slide 15

Page 16: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

What does a yellow traffic light mean?a) Stop

b) Go

c) Caution

d) Look behind you

“Recall” example

Slide 16

Page 17: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

If you are driving toward an intersection and the light turns from yellow to red, what should you do?

a) Speed up and cross the intersection

b) Continue at the same speed and cross the intersection

c) Stop suddenly

d) Stop gradually

“Understand” example

Slide 17

Page 18: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

You are giving a close friend a lift to work and because of traffic you are running 15 minutes late for an important appointment. You are driving toward an intersection with traffic lights on it. You can see the intersecting road to some extent and there seem to be no cars on it. The light turns to yellow as you approach. The car behind you sounds their horn and seems to be accelerating in the expectation that you are going to cross the intersection. Your friend tells you to put your foot down as you are running late and there are no cars visible on the intersection. What should you do?

a) The road appears to be clear so speed up and cross the intersection to exit the intersection as soon as possible

b) Stay at the same speed and continue to cross the intersection while keeping an eye out for crossing cars

c) The light is about to turn red so you should stop immediately and ignore the car behind you as they will need to stop as well

d) Put your hazard lights on to signal to the car behind you and stop at the intersection if the lights facing you turn red

e) Ignore your friend’s irritation and move across to the side of the road to let the other car pass

“Evaluate” or “Judgment” example

Slide 18

Page 19: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

William Coscarelli and Sharon Shrock: ▪ Knowledge (recall) level items are by far the

easiest to write.

▪ Developing tests that truly reflect on-the-job performance requires … writing items at the higher cognitive levels

▪ In general, the single most useful improvement you can make in writing test items is to write them above the memorization level.

Advice from the experts

create

evaluate

analyze

apply

understand

remember / recall

Page 20: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Beyond Multiple Choice…

Slide 20

Also reduces chance to guess. In a 4 question multiple choice

question, 25% chance of guessing. In these questions 3%

chance of guessing

Page 21: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Observational Assessment

Slide 21

Page 22: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Observational Assessment

Observer “rates”the participant

Example Applications

Sales Training

Certification

Medical and Dental (OSCE)

Equipment operation

Page 23: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Monitor/observer logs in

Selects assessment

Selects participant to be observed

Answers/submits responses to questions

Results/feedback reported to stakeholders

Typical workflow for “observer”

Slide 23

Page 24: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Items typically solicit ratings for:

• Performance of task(s)

• Demonstration of appropriate knowledge, skills, abilities, behaviors

Goal is to ensure ratings are as objective as possible

• Keys to ensuring reliability of observational assessment results:

• Establish consistent rating scales & scoring rubrics

• Educate raters to ensure consistent understanding and application of rating scales

Getting objective ratings from Observational Assessments

Slide 24

Page 25: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Testing Judgment

Slide 25

Page 26: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

An unannounced drill was initiated Employees provided with conflicting information:

▪ “properly includes the drill language ‘EXERCISE, EXERCISE, EXERCISE’”-- but --

▪ “erroneously contains the text of an EAS message for a live ballistic missile alert, including the language, ‘THIS IS NOT A DRILL.’”--”

U.S. Sen. Brian Schatz: “The reports from the FCC and the state of Hawaii demonstrate systems and judgment failures on multiple levels”

FCC Report suggests this was dilemma requiring judgment

Slide 26

Page 27: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Rules immensely important

• in defining the parameters within which teams and individuals should operate

Rules alone are not enough

• Whether and how they are adopted by people when making decisions that matters

“Judgement is at the heart of nearly every business scandal that ever occurred”

Why judgment matters (Ernst & Young)

Slide 27

Page 28: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Almost all job roles require judgment Situational Judgement Assessments present work-relevant dilemmas

with questions that require judgments about possible responses Assessing how well participants can:

▪ Recognize critical aspects of a situation and how they should be prioritized▪ Identify which actions/options are more vs. less likely to achieve objective▪ Determine which responses are more vs. less aligned with organizational expectations▪ Display the soft skills required work collaboratively with colleagues to overcome obstacles and

achieve desired outcomes▪ Deal with dilemmas in which there are no clear ‘positive outcomes’ -- or in which available

information may be ambiguous

Assessing for Situational Judgment

Slide 28

Page 29: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

A score or evaluation is

made

The participant

chooses from options

Dilemma that needs judgment

Context is defined

Structure of an Situational Judgment Assessment (SJA)

Page 30: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

You work in the back office in the team approving new customers ensuring that the organization’s procedures have been followed (such as credit rating and know your customer). Your manager is away on holiday this week. A senior manager in the company (three levels above you) comes into your office and says that there is an important new customer who needs to be approved today as they want to place a big order, and that he can vouch that the customer is good. You review the customer details, and one piece of information required by your procedures is not present. You tell the senior manager and he says not to worry, he is vouching for the customer. You know this senior manager by reputation and have heard that he got a colleague fired a few months ago when she didn’t do what he asked.

Select the most and least effective of the following options:▪ Take the senior manager’s word and approve the customer▪ Call your manager’s cellphone and interrupt her holiday to get advice▪ Tell the manager you cannot approve the customer without the information needed▪ Ask the manager for signed written instructions to override standard procedures to allow you to approve the

customer

Example SJA question - compliance

Slide 31

Make a judgment

Page 31: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Pre-hire

• Realistic job preview –help applicant and employer work out job fit

Post-hire workforce development

• Diagnosing training needs

• As training with feedback

• Post training assessment

• Measurement of soft skills

• Evaluating job readiness

Post-hire compliance

• Assess judgment of workforce

• Survey what people in organization would do in dilemmas

• Spot possible problems and mitigate risk

Certification

• Many high-tech certification programs criticized for being too focused on basic knowledge, not enough on important job skills

• SJAs provide a practical and inexpensive way of testing judgment

Applications of Situational Judgment

Slide 32

Page 32: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Genuinely measures judgment in a way seen by all as fair

Lower cost compared to many other approaches

Scalable – can delivered widely

Benefits

Slide 33

Page 33: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

How to create Situational Judgment Assessments (SJAs)

Slide 34

Page 34: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Attributes / Competencies

Situations Dilemmas Item stemResponse options

You need SMEs (Subject Matter Experts) to author and review SJAs as only they will know what judgments are needed in the job role and how to set up realistic and relevant dilemmas

Authoring Process for SJAs

Page 35: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Commitment to Professionalism

• Dealing with issues of confidentiality such as hearing a colleague talking about a patient outside of work

• Challenging inappropriate behavior such as a consultant speaking to a colleague/patient in an inappropriate way

Coping with Pressure

• Knowing how to respond when you make a mistake such as providing wrong medication to patient

• Seeking help when not sure of the correct procedure/best way of doing things

Effective Communication

• Gathering information and communicating your intentions to nursing staff or other colleagues

• Listening and effectively communicating such as with an unhappy patient or relative

Patient Focus

• Identifying that a patient’s views and concerns are important and they should have input into their care

• Spending time trying to understand a patient’s concerns and empathising with them

Working Effectively as Part of a Team

• Recognising and valuing the skills and knowledge of nursing staff, when faced with a disagreement about a patient’s care

• Offering assistance and support to colleagues when they are not able to handle their workload

Example Attributes for SJAs (UK medical doctors)

Slide 36

Page 36: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Ask Subject Matter Experts▪ Do you do task or supervise it

▪ How Difficult is it to do task

▪ How Important to do task

▪ How Frequently to do task

Identify what to assess

Job Task Analysis Survey can help determine Job Needs

Slide 37

Page 37: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Item writing workshop / Survey SMEs Brainstorm situations that discriminate

between effective/appropriate behavior and less effective

Include real life examples Analyze, sort and frame

dimensions/topics Good for ensuring that real life

instances of judgment identified

Start with a job analysis and competencies required

Item writing workshop / Survey SMEs

For each competency, identify situations for each competency where demonstrated/not demonstrated

Good for ensuring competencies covered

Two ways for SMEs to identify Situations for SJAs

Slide 38

Critical Incident Analysis Competency-based Approaches

Page 38: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Choice needs to be made

Using judgment for the choice is clearly linked to a needed domain of knowledge, skill or competency in the job role

Requires professional judgment to answer well. (Question discriminates between more and less capable people)

Key Aspects of a Dilemma

Page 39: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

What the participant thinks is best to do:▪ Choose the most effective response▪ Choose the most and least effective

response

Best for higher stakes and competency testing

What the participant thinks he/she would actually do:▪ Which response best typifies what you

would do in this situation▪ Choose the response that best reflects how

you would respond and least how you would respond

Best for understanding participant behavior, e.g. in recruiting or learner diagnostics or compliance surveys

Response Options for SJAs

Slide 40

“Should” “Would”

Page 40: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Select most appropriate choice▪ Simplest cognitively▪ Easiest to guess

Select most and least appropriate choice▪ Checks knowledge of what is best and worst▪ A good solution

Select x best choices▪ For example, choose the 2 best options out of 5

Rank the choices▪ Hardest for participant▪ Challenge to design as experts have to agree the

best ordering of all choices

Rate each choice ▪ E.g. agree/disagree/neither agree nor disagree that

someone in your organization would do this▪ Good for surveys including in compliance

Five different kinds of response options

Slide 41

Page 41: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

A dilemma is more than a situation/scenario – must be a choice to be made linked to required domain of knowledge/ skill/ competency/ compliance

SJAs can be mixed with other questions or presented on their own. You should allow 1-2 minutes time in the test to answer an SJA question

Best to use a response type that requires more than just “choose the best” to reduce chance of guessing▪ With 4 choices, there is a 25% chance of guessing a “choose the best” and an 8% chance of

guessing “choose the best and the worst”

Usual guidelines on item writing (avoid ambiguity, responses similar length etc. apply)

You need consensus from SMEs as to what the best choice is. A dilemma where experts disagree on the best answer is not useful

SJA Item Writing Advice

Slide 42

Page 42: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Important to survey SMEs not involved in the item development▪ Present the item stem

▪ Present the response options

▪ Ask them to rate

Compare their ratings with the scoring you have developed and adjust choices as necessary

Validating with other SMEs or experts

Slide 43

Page 43: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

For medium or high stakes use of SJAs, just like other questions, you should▪ Present SJAs to a pilot group

▪ Conduct item analysis

▪ Look carefully at each question’s statistics and remove/adjust questions which need it

Pilot and run Item Analysis

Slide 44

Page 44: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Exercise

Please get in small groups to work together

Slide 45

Page 45: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Suppose we were looking to write SJAs for assessment professionals, what kind of situations/dilemmas come up which need judgment when you are authoring, delivering and managing assessments?

Please get into groups and identify▪ A situation and context

▪ A dilemma where

There is a choice

Which needs judgment

And requires context

▪ You may want to think about a real issue that has happened to you

Critical Incident Analysis

Slide 46

Page 46: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Let’s take an example dilemma and discuss possible response options together

Brainstorm an SJA question

Slide 47

Page 47: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

How Questionmark can help

A reliable, secure, scalable assessment management system for SJAs

Slide 48

Page 48: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Planning assessment▪ Use Job Task Analysis Surveys to

help set attributes / competencies

▪ Survey SMEs to identify or rate dilemmas

▪ Set up a topic structure to base your assessment on

Questionmark can be used throughout SJA lifecycle

Page 49: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Capability to author and score all SJA item types including “Most and least”

Collaborative authoring environment where SMEs can easily review questions and see how they change

Manage item bank including SJAs Shuffle choices and/or question

selection

Authoring items and Assembling assessment

Slide 50

Page 50: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Include SJAs as experimental questions within live tests to gain data to validate question

Use comment box to get feedback from participants or SMEs

Use the Item Analysis Report to identify and improve weak questions

Pilot and Review

Slide 51

Page 51: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

SJAs easily callable from LMSs and other systems

Instant scoring and feedback Scalable and robust Secure against breaches and

cheating Effective UI and smartphone

support

Comprehensive reports to make professional decisions

Coaching report to share with participants

Test and item analysis reports to aid validity/reliability

Delivery and Reporting

Slide 52

Delivering SJAs Reporting on SJAs

Page 52: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Example Questions: http://www.questionmark.com/go/example-sja

Slide 53

Page 53: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Summary

Page 54: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Assessments that test recall:▪ Check someone’s knowledge

▪ Give retrieval practice which reduces forgetting

Competence in most jobs requires testing a lot more than recall:▪ Testing higher level cognitive skills

▪ Testing practical skills (observational assessments)

▪ Testing judgment (SJAs)

create

evaluate

analyze

apply

understand

remember / recall

Beyond Recall

Slide 55

Page 55: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

A score or evaluation is

made

The participant

chooses from options

Dilemma that needs

judgment

Context is defined

Dilemma

Requires choice to be made

Using judgment for choice clearly linked to needed

domain in job role

Requires professional judgment to answer well

SJAs an Effective means of Testing Judgment

Page 56: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Measure and Enhance Learning

•Measure judgment

•Identify job readiness

•Diagnostic assessments

•Improve learning and performance

Qualification and Certification

•Deliver reliable, defensible tests to certify employees

•Include measurement of judgment

•Ensure sound personnel decisions

Regulatory Compliance

•Measure judgment in your personnel

•Survey to assess what personnel think others would do

•Get early warning of potential issues that could lead to a compliance failure

Sales Channel Verification

•Measure not just how sales people know the product, but also their softer skills and judgment

•Increase revenues and customer satisfaction

Technical Channel Verification

•Measure judgment of technical and service professionals

•Reduce costs and improve customer satisfaction

Credentialing

•Relatively inexpensive way of extending coverage of certification programs to higher order and softer skills for candidates

SJAs can be used in many Applications

Page 57: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Can you see applications of SJAs within your organization?

If so, what actions will you take to start using?

Group discussion

Slide 58

Page 58: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Assessing for Situational JudgmentDesigning, deploying and getting value from

Situational Judgment Assessments

By Eugene Burke & John Kleeman

To learn more, see our white paper

Assessing for Situational Judgment

Slide 59

40-page white paper provides:▪ Overview of Situational Judgment

Assessments (SJAs)

▪ Benefits and applications of SJAs

▪ Techniques for creating and validating SJAs

▪ Example SJA items to illustrate key concepts

www.questionmark.com/learningresources

Page 59: Beyond Recall: Taking Competency Assessments to … · Most real world jobs require many ... The light is about to turn red so you should stop immediately and ignore the ... NOT A

Copyright © 1995-2018 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.

Questionmark Conference 2018 ❖ Savannah

Your questions


Recommended