Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Six Steps for Trustable Assessment Results: Ensuring Valid and Reliable Assessments with Questionmark
A look at how Questionmark’s tools enable valid, reliable and defensible testing and assessment
Brian McNamaraMarketing Director, Questionmark
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessments – Examples and Applications
What makes assessment results “trustable”
Six steps for trustable results
Examples of supporting technology
Today’s webinar…
Slide 2
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Founded in 1988
• Offices in US, UK, Canada; Distribution partners in 27 countries
• Assessment technologies and portal solutions that enable organizations to measure knowledge, skills, and attitudes securely for certification, regulatory compliance and improved learning outcomes.
Background
• 1,000 current enterprise customers worldwide
• Corporate and Government (Learning/HR function); Higher Education; Awarding Bodies and Test Publishers
Customers
About Questionmark
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Assessment Management System
• Questionmark OnDemand (SaaS/cloud)
• Questionmark Perception (installable software)
• Multilingual
• Localized for dozens of different languages and cultures
• Global 24/7 Support
Products and Services
About Questionmark
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
White Paper:
Slide 6
www.questionmark.com/go/trustable-results-white-paper
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Quizzes, Tests, Exams
•Questions and scoring algorithms
•Scores and feedback at question, topic and assessment level
Surveys
•Questions relate to a person’s opinions on a topic
•Conclusions drawn based on analyzing frequency of responses
•Examples include course evaluations, employee attitude surveys
Observational Assessments
•Someone assesses or rates a person’s performance, knowledge, skills, and abilities
Types of “assessments”
Slide 7
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What can we learn from assessments?
Slide 8
Did they like it?
Did they learn it?
Can they do it?
Can we prove it?
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Regulatory Compliance
• Ensure knowledge of laws, regulatory requirements, health & safety
• Demonstrate compliance; reduce risk
Qualification & Certification
• Reliable, valid tests to certify knowledge, skills
• Make sound, defensible personnel decisions
Sales Channel Verification
• Ensure competence and product knowledge
• Increase revenues and customer satisfaction
Technical Channel Verification
• Ensure competence and expertise of technical and service professionals
• Reduce costs and improve customer satisfaction
Measure & Enhance Learning
• Measure knowledge, skills and attitudes –before, during and after learning
• Correct misconceptions and increase knowledge retention
How our customers use assessment
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Getting Trustable Results
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Trustable assessment results matter
Slide 13
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Recent Headlines
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Make informed, defensible personnel decisions
Mitigate regulatory compliance risk
Aligned workforce competencies with organizational objectives
Target learning investments accurately
Onboard new employees effectively
Ensuring competency of sales force impacts revenue, customer satisfaction
Ensuring competency of technicians impacts costs, customer satisfaction
Why trustable results matter
Slide 15
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
• Assessments need to be • Valid• Reliable• Secure
• Sound practices for developing, delivering and reporting on assessments
• Technologies to support sound assessment practices
What makes results “trustable” ?
Slide 16
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Valid and Reliable Assessments Reliable: dependable, repeatable, consistent
Valid: measures appropriate knowledge and skills
Figure 1:Reliable but not Valid
Figure 2:Not reliable, not Valid
Figure 3:Reliable and Valid
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Planning assessment
Authoring items
Assembling assessment
Pilot and review
DeliveryAnalyze results
Six Steps to Trustable Assessments
Security
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assessment Management System
Collaborative, secure authoring of surveys,
quizzes, tests, and exams
Evaluate results and inform stakeholders
with timely, meaningful reports
Blended delivery to support a broad range of devices and security
needs
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Planning assessment
Authoring items
Assembling assessment
Pilot and review
DeliveryAnalyze results
Examples of Supporting FeaturesIdentify, validate content with JTA;
security requirements
Involve SMEs in collaborative
authoring and review
Secure delivery; deliver to tablets and
smartphones
How are my tests doing? How are my employees doing?
Random, rules-based question selection
from item bank
Include “experimental” questions that don’t
impact scores
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Planning assessment
Authoring items
Assembling assessment
Pilot and review
Delivery
Analyze results
Sound Assessment Practices +Assessment Management System
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
A closer look…Examples of features and functions supporting sound assessment practices
Slide 22
Planning assessment
Authoring items
Assembling assessment
Pilot and review
DeliveryAnalyze results
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Organize items in an item bank with topic structure
Metatag items to associate with task knowledge, skill or ability
Use Job Task Analysis (JTA) surveys to help blueprint assessments
Authoring processes secure to protect against content theft
Manage translations to aid multilingual assessments
Planning assessment
Slide 23
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Validate assessment content using Job Task Analysis
Ask Subject Matter Experts:
• How difficult is it to do this task?
• How important is it to do this task?
• How frequently do this task?
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Multilingual Assessments
• Ability to manage translations within the application
• Import/export assessment content for localization by third party
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring tool Subject Matter Experts can use directly
20+ question types
Ability to include relevant stimulus including video, audio and equations
Item version history for legal defensibility
Easy collaboration for item reviewers to help make items more valid
Search questions to identify those that need update
Authoring items
Slide 26
Ability to include relevant stimulus including video, audio
and equations
Intuitive, browser-based application; SMEs can start using
with little or no orientation
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Authoring tool SMEs can use collaboratively, securely
Set permissions for authors and reviewers
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Item version history for defensibility
Slide 29
Audit trail of edits and option to “rollback”
Easily “track changes” made in collaborative
authoring or review
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Random selection of questions from item bank
Random ordering of questions and choices
Set a pass score including topic pre-requisites
Set assessment time limit (and override if required)
Topic scoring and feedback
Assembling the assessment
Slide 30
Corrective feedback points participant to appropriate learning resources
“Topics” of questions answered in assessment
Score on topic
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Assembling assessment:Random item selection
Slide 31
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Easily deliver assessments for trial purposes
Capture participants comments on items to improve validity
Run useful reports on the results of trial assessments
Use experimental questions without impacting scores
Pilot and review
Slide 32
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Consumer quality participant UI
Blended delivery, mobile support
Deliver observational assessments
Option to present different questions on a retake
Time limits; set time periods before a retake
Delto help measure performance
Launch and track assessments from other systems of record
Delivery
Slide 33
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Deliver:Responsive Design, Blended Delivery
One Assessment…
…many devicesand browsers
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Observational Assessment: Can they do it?
Observer “rates” the participant
Smartphones/tablets ideal for observational assessment
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Results saved at the server “as you go” so IT failures don’t lose data
Hope for the best; prepare for the worst
Slide 36
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Secure browser to reduce content theft and improve exam integrity
Delivery for high stakes
Slide 37
Windows Mac iPad
PrintingScreen captureTask switching
Control/system keysRight-click / context menu
optionsView source
Accidentally exiting exam
Prevents….
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Option to require monitor to confirm participant identity
Delivery can be integrated with online proctoring solutions
Require Proctoring
Slide 38
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Text sizing, contrast controls to enhance accessibility
Also support for…- Non-mouse navigation; alternate
pointing devices- Screen readers
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Accurate, robust and tamper-proof results storage
Differential security on results so only the right people see them
Item analysis report to weed out poor items
Test analysis report to calculate reliability
Wide range of accurate reports to help professionals make good decisions
Results can be anonymous to help make employee surveys valid
“Big Data” potential to make better business decisions
Analyze results
Slide 40
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
What it measures: % answered correctly
How it is expressed: Number from 0.0 to 1.0
How to interpret: Too high = too easy Too low could mean…
Too difficult Confusing/ambiguous Miscoded
What it measures: Correlation between
item score and test score
How it is expressed: Number between
-1.0 and +1.0.
How to interpret: Generally…
< 0.2 low > 0.3 good
Reliable results: Basics of Item Analysis
Difficulty index (aka P “value”) Item correlation discrimination:
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 42
Difficulty/P-Value
Correlation -Discrimination
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.Slide 43
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Built-in Reports Custom Reports and Dashboards
Microsoft Excel BI Tools
Big Data Applications: Results API
Slide 44
R e s u l t s W a r e h o u s e
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Summary of key functions
Collaborative Authoring
•Secure
•Browser-based
•Available globally
Responsive Design, Blended Delivery
•Latest browsers
•Tablets/smartphones
•Accessibility
•Secure delivery for high stakes exams
Extensive reporting and analytics
•How are my participants doing?
•How are my tests doing?
•Results API
Multilingual
•Translation Management
•Screens/Controls in 34 languages
•Right-to-left languages
Integration with Enterprise Systems
•Supports industry standards
•APIs
•Connectors
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
White Paper:
Slide 46
www.questionmark.com/go/trustable-results-white-paper
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark.All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Keep in Touch
Slide 47
blog.questionmark.com
plus.google.com/+QuestionmarkCorp/
twitter.com/questionmark
linkedin.com
youtube.com/questionmarkvideo
facebook.com
www.questionmark.com/newsletter
Copyright © 1995-2015 Questionmark Corporation and/or Questionmark Computing Limited, known collectively as Questionmark. All rights reserved. Questionmark is a registered trademark of Questionmark Computing Limited. All other trademarks are acknowledged.
Thank you for attending!
Brian McNamaraMarketing Director, Questionmark