Determining Validity For Oklahoma’s Educational Accountability System
Prepared for the American Educational Research Association (AERA)
Oklahoma State Department of Education
April 15, 2004
Education Perspectivefor Oklahoma, 2002-2003
• All Local Education Agencies (LEAs) in the State of Oklahoma participate in Title I programs.
• There are 541 total LEAs44 LEAs with a student enrollment of 2,500 or
greater201 LEAs with a student enrollment between
499 and 2,500296 LEAs with a student enrollment of less
than 500
Oklahoma Demographics, 2003Total school districts: 541
Independent (K-12): 430
Elementary (K-8): 111
Total School Sites: 1791
Teachers: 47,259
Oklahoma Demographics, 2003 Average Daily Membership: 618,358
Special Education (K-12): 13.6% 83,812
Alternative Education (K-12): 2.5% 15,312
Oklahoma Demographics, 2003American Indian/Alaskan: 17.9%
Asian/Pacific Islander: 1.5%
Black/Non-Hispanic: 10.9%
Hispanic: 7.0%
White Non-Hispanic/Other: 62.6%
State Standards, Assessments & Accountability Timeline
1990 - Oklahoma Education Reform and Funding Act of 1990
1991 - Initial development of core curriculum with statutory requirement for
review and revision every 3 years
1992 - Legislation mandated the development of Criterion-Referenced Tests (CRTs) in seven (7) content areas for Grades 5, 8, and 11
1993- First review and revision of core curriculum, Priority Academic Student Skills (PASS)
1995- Implementation of CRTs in Grades 5, 8, and 11
State Standards, Assessments & Accountability Timeline
•PASS revisions – 1997, 2000, 2003• Revisions have provided more specific, detailed and clear standards• PASS includes the Oklahoma School Testing Program (OSTP) “blueprints” for the CRTs
State Standards, Assessments & Accountability Timeline
1989 - Low-performing, high challenge schools identification process adopted in state law; includes sanctions and technical assistance
1999 - Academic Performance Index (API) adopted in state law; includes API Awards
State Standards, Assessments & Accountability Timeline
Oklahoma’s Accountability System
To guide instruction
To measure achievement
To promote a climate of
change
Proficiency for all
Content and Performance Standards
Assessments
Additional Indicators
Decision Rules
Sanctions and
Rewards
Validity Questions for Oklahoma’s Accountability System
Do our standards help guide instruction?
Do our assessments and performance indicators measure performance standards (i.e. achievement)?
Do our decisions rules accurately identify schools?
Do our sanctions and rewards promote change towards intended outcomes?
Key Elements in Determining Validity and Reliability
1. Ensure that the system has provided the intended outcomes.
2. Conduct research on additional information to corroborate findings.
3. Analyze design and implementation of each component of the system.
4. Conduct Analysis on several levels.
Sources of External Evidence
• Studies and research using additional indicators to corroborate findings or results of the system.
• Surveys on attitudes and opinions.
• Outside reviews for components of system.
Sources of Internal Evidence
• Ongoing review of policies and procedures.
• Data analysis for various levels such as State level, district level, and for particular types of schools.
• Evidence of Quality Control measures and Data Audits.
• Trend analysis over time.
PASS - external reviews have included:• 2001-02: Achieve, Inc.• 2001-02: Southwest Educational
Development Laboratory (SEDL)• 2001-02: Technical Issues in Large Scale
Assessment (TILSA)• 1997-2003: “Quality Counts” report cards by
Education Week
Oklahoma’s Steps Towards ValidationStudies and Reviews
Studies and Reviews
TILSA Review: Curriculum and Assessment Alignment Study (2001-02) with
Dr. Norman Webb, University of Wisconsin• Results of the study enabled Oklahoma to conduct
more in-depth alignment of standards and assessments
• Provided common language to facilitate communication in building curriculum and designing aligned assessments
Research Study (2002-03) by
Dr. John Poggio, University of Kansas
• Study to determine levels of cognitive complexity of multiple choice test items
Studies and Reviews
Implementation Survey (January 2002)
• After initial implementation of the API an attitude and opinion survey was collected.
• Additional surveys specifically on Adequate Yearly Progress (AYP) will be collected in October 2004.
Studies and Reviews
• Study on School Improvement Schools– Collecting additional local LEA assessments
from additional grades.
• Purpose: – To corroborate the results of the decision rules.– To look at trends of schools being identified as
school improvement .
Studies and ReviewsOngoing Study
Grants and ProjectsNo Child Left Behind (NCLB)
Enhanced Assessment Grant
• Oklahoma selected as lead state in collaborative curriculum/assessment alignment project with Council of Chief State School Officers (2003)
• Continues the alignment work done in 2001
• Includes vertical alignment of curriculum and assessments (Grades 3-8)
• Includes alignment of special education alternate assessments
Grants and Projects
Project with Council of Chief State School Officers (SCASS) - 2003
• Develop survey tools to determine the level of implementation of a state’s standards-based language arts curriculum
• To provide support for teachers in meeting NCLB requirement to increase student achievement
PoliciesA few examples of changes or development
as a result of reviewing the system.
• Developed policy for appeals of Adequate yearly progress.
• Developed a Technical Advisory Committee to review and offer expertise in regards to technical issues of the State testing program.
• Implemented new rules and regulations regarding data audits and security.
AYP Statistics• 367 (20.5%) of Oklahoma Schools did not
make AYP.• 198 (36.5%) of Oklahoma School Districts
did not make AYP.• 1.7 is the average number of subgroups that
did not make AYP by school.• 1.5 is the average number of subgroups that
did not make AYP by district.
AYP Statistics
• Total number of subgroups not meeting specific performance targets, ordered from greatest to least.– Reading target– Math target – Additional Indicator (attendance or graduation)– Participation target
AYP Statistics
• Total number of subgroups not meeting specific performance targets, ordered from greatest to least.– Reading target– Math target– Additional Indicator (attendance or graduation)– Participation target