1
Quest for Perfection
2
Florida Comprehensive Assessment Tests and End-of-Course Assessments
FCAT 2.0 Reading Grades 3 – 10FCAT 2.0 Mathematics Grades 3 – 8 FCAT 2.0 Science Grades 5 & 8FCAT Writing Grades 4, 8 &10Algebra 1 & GeometryBiology Under development: U.S. History, Civics
Pervasive Quality Assurance
Built into every process from planning and designing, to executing
Process and outcome documentation Process and outcome audits Transparency Continuous refinement and redesign Discipline and vigilance; trust but
verify 3
Pervasive Quality Assurance
Design and Development
Test Construction
Document Quality
Test Security and Administration
Scoring
Calibration/Equating
Reporting
4
5
Test Development Center Contracted center, staffed by educators Ensures educator involvement & content quality
Standing and Rotating Committees (e.g., Content Advisory, STEM Experts, College-ready Standards, …)Annual Report to District Superintendents District educators who served in past year District educators who are in database
(statewide ~4200) Request for nominations
6
Test and Item Content Quality
Content Advisory
Committees
Test and Item Specs
Item Review Committees
Review and select field-test items
Items are field tested
Appropriate?Yes
Meet statistical criteria?
Yes
HS
HS
HS Handscoring
Standards AssessedClarificationsItem TypesExamples
Educator Involvement
Key
7
Annual training conducted for Department and contractor staffTest Construction Specifications
Revised and enhanced annually Incorporates checklists Checklists follow documents during test
constructionRelentless focus on item quality, overall coherence of test content, statistical quality
Example: location of reading passages compared to prior location is strictly controlled
Test Construction
Document Quality
Test Development Center Editors Accommodations specialist Editorial and content checklists
Variety of staff take each test
Intense and synchronized process
8
Test Administration
Annual “Test Administration Debrief” meeting
Review of all products against specifications, including shipment samples
Webinars for preparation and training on CBT
Tests are administered and monitored by certified educators.
Multiple equated core forms of CBT are randomly assigned within classrooms.
9
Test SecurityExtensive and enhanced test security requirements
Test Security Statute, s. 1008.24, F.S.
Prohibited Activities Agreement
Seating charts, test group codes
Test materials chain of custody form
Student pledge
Data forensics on all grades and tests
Some student results automatically invalidated but have an appeals process.
Investigations, including Department of Law Enforcement and FDOE Inspector General
10
Scoring and Data Quality
Specifications and checklists
Extensive planning and dry runs with mock data
Contractor site monitoring by FDOE
Document processing and equating audits
Triangulation with FDOE and multiple contractors
Additional assistance from FSU for data checking 11
12
Handscoring Quality Assurance
Extensive quality assurance processes are utilized for FCAT Writing handscoring
FDOE and educators review, revise, and approve all training and qualifying materials.
FDOE staff provide on-site monitoring for training, qualifying, and first full week of scoring
One week of “pseudoscoring”
External audit and public reporting
Reporting Results
Statute (s.1008.22(c)14, F.S.) dictates testing and reporting timeframes.
Cross-agency planning team meets weekly for several months prior to reporting.
Hire seasonal data checkers to verify files/paper reports.
Choreographed file transfers (contractor, FDOE, web team, data warehouse), version control, analyses, and briefings
13
Pervasive Quality Assurance
Built into every process from planning and designing, to executing
Process and outcome documentation Process and outcome audits Transparency Continuous refinement and redesign Discipline and vigilance; trust but
verify 14
15
More information
Websites:
fcat.fldoe.org/default.asp
www.floridapathtosuccess.org/
Technical: fcat.fldoe.org/fcatpub5.asp
: @EducationFL
Florida Department of Education
FL Department of Education