CLEAR 2003 Annual Conference Toronto, Ontario

Technology, Service, and Delivery

Stephen Greenfield

Product Services Manager

CLEAR 2003 Annual Conference Toronto, Ontario

Using the Internet to Enhance Your Testing Program

Internet-based tools for licensure programs

• Scheduling

• Communication

• Data management

• Reporting

Scheduling

Reporting

• Access for sponsors to data about the program– Registration and scheduling– Testing locations– Testing volumes– Exam results

Data Management

• Enhanced options for program management– Information on candidates– Custom reports– Incident tracking– Managing candidate eligibility, special

accommodations etc.

Internet provides tools for

• Scheduling

• Communication

• Reporting

• Program management

CLEAR 2003 Annual Conference Toronto, Ontario

Innovative Uses of Technology in Testing

“... advances in Internet security...”

• Network-layer security (SSL, firewall, etc.) • Message-level security (encryption, etc.)• Application-level security (logins, etc.).

Test security

• Technological protection of test items is just one part of test security– Protection of materials during test development– Confirmed identity of test takers– Confirmed identity of testing staff– Proper procedure by proctors / invigilators– Prevention of sharing of test items

Technology support for test security

– Controlled access to test materials– Testing center procedures– Testing center facilities– Biometric ID of

candidates & staff

Test security

• Computer-based testing allows security enhancements– Alternate forms, randomized or adaptive

presentation, to reduce item exposure– Analysis of test-taker performance (responses,

latency) to detect possible cheating– Analysis of result patterns– Monitoring of proctor actions to detect errors and

possible fraud

Innovative item types

• Benefits– Realistic stimuli (images, audio, motion)– Recall rather than recognition– More realistic tasks

• Performance-based • Simulation

– Face validity

Innovative item types

• Issues include– Time and cost to create items– Objective scoring– Statistical tools for analysis– File sizes

Automated scoring of narrative answers / essays

• Commercially-available programs show high correlations with human raters

• Methodologies include– Syntactic and rhetoric analysis– Semantic and content analysis– “Training” with scored sample essays

Automated scoring of narrative answers / essays

• Vast majority of essays still go to human raters

• Internet-based tools to support human raters– Rubrics– Analyze scoring patterns– Exchange cases

Internet-based technology and certification programs

• Scheduling • Delivery of tests• Collecting and reporting results• Managing the program• Security benefits as well as challenges• Advanced item types