Effective Communication of Exam Results:

What Should (or Shouldn't) be Included in the Candidate's

Score Report

Elizabeth A. Witt, Ph.D.American Board of Emergency Medicine

2006 Annual Conference

Alexandria, Virginia

Council on Licensure, Enforcement and Regulation

Expect the Unexpected: Are We Clearly Prepared?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Validity

• Degree to which evidence supports the interpretation and proposed use of test scores

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Certified Chefs of America

Examination Results

Exam: South American Cuisine

Date: August 15, 2006

Name: Jane Smith

Result: Pass

Your certificate will be mailed within 10 days.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

RESULTS

Summary

The following graph reveals how closely Sample Report's responses match up to those of top performers.

Overall Suitability

Overall, Sample Report exhibits satisfactory levels of key competencies, as compared to top performers. Sample's pattern of responses is similar to those of successful real estate agents, although there are areas of divergence. By targeting training and development to the areas of weakness, Sample could boost performance to a higher level. Please review the Detailed Results below to identify Sample's areas of strengths and weakness from the perspective of a career in real estate sales.

Detailed Assessment

The following graph illustrates Sample Report's areas of strength and weakness. The graph is laid out in terms of critical competencies that are required to successfully perform as a real estate agent. For each competency, Sample Report is evaluated in terms of Sample's current level of development. A high score indicates strength in that particular competency. A low score indicates that a competency may require further development. Please review the information below the graph for an interpretation of the results.

SALES ABILITY

Building Rapport And Acquiring Information

Listening And Comprehension Skills

Managing The Sales Process

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What information do examinees want?

Pass/Fail status

Actual score 78

Comparison to passing score

Areas of strength/weakness

Comparison with others

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

What else do examinees want?

Timely notification

Explanations – graphs, technical terminology

Graphics, visual

presentation

Simplicity

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Timely notificationTimely notification

Why do we not just give them what they want?

Instant notification possible with CBTBUT

Not with paper examsAND

Requires relinquishing controlINCLUDING

Elimination of poorly performing items

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

SimplicitySimplicity

Why do we not just give them what they want?

We lose sight of the purpose/audience.

This is not a technical report!

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

ExplanationsExplanations – graphs, technical terminology

Why do we not just give them what they want?

Not always necessary

Familiarity – lose sight of the audience

May require psychometric services

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Graphics, visual Graphics, visual

presentationpresentation

Why do we not just give them what they want?

Not always necessary

Small staff, small program

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Why do we not just give them what they want?Actual scoreActual score – may need explanation

Raw score: number correct, percent correct

Logits

Scaled score

Scores may not be precise away from the cut

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Why do we not just give them what they want?

Comparison to passing scoreComparison to passing score

– Scores may not be precise except at the cut

– May need explanation (SEM, for example)

– May be self-explanatory(if a simple raw is reported, for

example)

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Why do we not just give them what they want?

Areas of strength/weaknessAreas of strength/weakness

- Need a subscore for each area

- Scores based on few items are not reliable

- Subscores may not sum to total test score- confusing to examinees

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Why do we not just give them what they want?

Comparison with othersComparison with others

- Scores are normally criterion-referenced.

- The group testing may differ by examadministration.

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Recommendations

DO include:DO include:

Name, exam title, exam date

Pass/Fail Status

An interpretable score

Score required to pass

Same results by section, if appropriate

Simple, clean graphics

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Recommendations

DO include:DO include:

Simple, clear explanations for any graphs

or technical terminology

Brief, simple explanation of scoring

Diagnostic information for those who fail

How to request score verification, appeal result

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Recommendations

Do NOT include:Do NOT include:

DecimalsMean score = 275.358

Mean score = 275 Excessive detail Jargon

SEM Coefficient Alpha Logit Personal information (SSN, for example)

Teeny, tiny font sizes

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

RecommendationsCandidate score reports should be:Candidate score reports should be:

Timely

Simple

Clear

Brief

Easy to read and understand

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Bottom LineWho is the audience and

what do they need to know?

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

References/ResourcesAmerican Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association

Ryan, J. M. (2006). Practices, issues, and trends in student test score reporting. In S. M. Downing & T. M. Haladyna (Eds.), Handbook of Test Development (pp. 677-710). Mahwah, NJ: Lawrence Erlbaum.

Schroeder, L. L. & Castle, R. Technology and testing: Reporting scores. (Winter 2006). CLEAR Exam Review, 27(1), 8-10.

http://www.ets.org/Media/Products/IDMS/pdf/51588_idmsRpt.pdf

http://www.realestatesimulator.com/pages/sample_reports/Sample3.htm

http://www.visualcertexam.com/screenshots.html

Presented at the 2006 CLEAR Annual ConferenceSeptember 14-16 Alexandria, Virginia

Speaker Contact Information

517-332-4800, ext. 328ewitt@ABEM.orgwww.ABEM.org

Elizabeth A. Witt, Ph.D.American Board of Emergency

Medicine3000 Coolidge Rd.East Lansing, MI 48823