Presented to:CCSSO 2012 National Conference on Student Assessment

June 27, 2012

Quality Assurance in a Chaotic WorldAn External Perspective

The External Perspective

HumRRO serves as – National Assessment of Educational Progress (NAEP) Quality

Assurance contractor

– Independent evaluator of California High School Exit Examination (CAHSEE)

– Psychometric second-checker for several states

In these roles, we stand outside operational assessments and provide feedback, warnings, guidance to federal and state clients

This presentation focuses on NAEP.

2

NAEP Quality Assurance

Several years ago, the National Center for Education Statistics (NCES) contracted with a single vendor for NAEP. This vendor, in turn, managed subcontracts.

NCES now contracts directly with several agencies for:– Item Development

– Web/Technology Development, Operations and Maintenance

– Materials Distribution, Processing and Scoring

– Sampling and Data Collection

– Design, Analysis, and Reporting

– NAEP State Service Center

– Alliance

– Quality Assurance

3

NAEP QA: Definitions

Quality Control (QC): integrated steps embedded within a process that check, verify, and document that data, analysis, and results are accurate.

  Quality Assurance (QA): an external set of steps that

verify the QC process is effective, complete, and efficient.

4

Categories of QA Activities

Participation in planning and monitoring meetings Observation of operational meetings (NAGB, DAC, NVS) Review of contractor QC plans Observation of operational processes Special studies

5

Participation in planning and monitoring meetings

Annual Design Summit Annual Quality Summit Periodic contractor meetings Inter-contractor quality teams

6

Observation of operational meetings

National Assessment Governing Board (NAGB) Design and Analysis Committee (DAC) NAEP Validity Studies Panel (NVS)

7

Review of contractor QC plans

Annual advance review of QC plans Critical components include:

– Completeness

– Ongoing self-monitoring

• Recording relevant data

• Monitoring of trends

– Continuous process improvement

• Feedback loops

Importance of closing the loop with operational contractors

8

Observation of operational processes

“Site visits” in four phases:– Item development

– Test administration

– Processing and scoring

– Analysis and reporting

Brief site visit reports after each event– Includes recommendations & commendations that are logged centrally

Annual summary report for each phase– Reflects upon overall patterns

9

Special studies

Quality Assurance Technical Panel (QATP) Small number of studies with a quality focus, such as:

– Use of trend papers in scoring

– Investigation of “interesting” analytic results in advance of public release

– Investigation of alternative approaches to full population estimates

– Evaluation of alternative approaches to NAEP-TIMSS linkage

– Evaluating improvements to processes fir development and review of math items

10

Examples of Specific Contributions

Improved contractor QC plans (e.g., specific decision criteria and follow-up actions)

Improved scoring process Improved training of field staff Increased self-monitoring by operational contractors Independent confirmation of a variety of findings before

public release Specific suggestions for following up on NAGB questions

and issues

11

Benefits of an External QA Contractor

Another set of eyes Unbiased perspective Free from some of the time and workload stresses of

operational contractors

12