Using Curricular Measuresfor Description &
AnalysisSEC Collaborative Meeting
Tampa, FLFebruary 21, 2005
John L. Smithson, Director, Measures of Enacted CurriculumWisconsin Center for Education Research, University of Wisconsin-Madison
Surveys of Enacted Curriculum
The SEC Data - Sets
DistinctionsOn-Line Off-line
•Data collection/processing/reporting•Descriptive Data•Limited Reporting Options•Easy Access / Indiv. Results
•Analytic analyses•Unlimited reporting options•Requires data manipulation
Conducting Inquiry Using SEC Data
Forms of Inquiry Collaborative or Evaluative
Teacher EnrichmentSchool ImprovementProfessional Lrng. Comm.
Program EvaluationIndicator ReportingProgram ManagementPerformance Modeling
• MSP-PD Study (In progress: Blank, Smithson, Porter, Garet, Birman)
Use of SEC data for program evaluation
A short history of SEC research
• Reform-Up-Close (Porter, Kirst, Osthoff, Smithson, Schneider, 1993)
Validation of teacher self-report survey data.
• Upgrading Mathematics (Gamoran, Porter, Smithson, White, 1997)
First content analysis of assessment using content language.Predictive validity of alignment index comparing instruction & assessments
• Data on Enacted Curriculum (Blank, Porter, Smithson, 2004)
Use of SEC data to facilitate school improvement effortsFirst content analysis of state standards
Alignment Relationships in Standards-based Reform
Instruction
Assessment
Standards
Intersection of what is taught with what is tested.
Intersection of what is taught with what is in standards.
Intersection of whatis taught with what is in the standards
Taught,tested, and inthe standards
Alignment Relationships in Standards-based Reform
A Quantitative Approach to Alignment
Porter-Smithson/SEC Alignment Process
Content analyses of curriculum documents and reports of practice by content experts using two-dimensional content language.
Multiple raters (w/ content & assessment expertise) using independent ratings in combination with team discussions.
Content Description [Topic(s) by Cognitive Demand(s)]
Yields Alignment Index based on:
Alignment Analyses
The analytic power of quantitative alignment measures
• Prediction
• Program Evaluation*
• Planning
• Performance Modeling
* Not teacher evaluation
Alignment Analyses
Students perform best when tested on material for which they have been provided the opportunity to learn.
Class Gains Student Gains
Alignment Index 0.451 0.259
Correlation of Alignment Index to Achievement Gains
From: Gamoran, et.al. (1997). Uprgading High School Mathematics Instruction. EEPA v19n4pp325-338.
Time 0 Time 1 Time 2
Time 0 Time 1 Time 2
Learning Gains by Course Type
Learning Gains Controlling for Content
RegentsAlgebraStretch RegentsMath A/B/UCSMPGen. Mth. / Pre-alg.
From:Upgrading High School Mathematics Instruction,(Gamoran, Porter, Smithson, & White, 1997),EEPAv19n4
12
11
10.5
12
9.5
10
11.5
12
11
10.5
12
9.5
10
11.5
Explaining variation in student learning gains
Alignment Index:Instruction to Standards
MathematicsAcross 4 Districts
Alignment Analyses for School Improvement
Using alignment as an outcome measure
(Measuring change in alignment over time)
Treatment 99
Control 124
Leaders 16
Counts
Alignment Analyses for Planning Purposes
Mapping Curriculum Materials
(www.seconline.org) SEC - Online
Administrative Functions:Administration Set-upReview Registrants, Completion RatesAdministrator Report Generator
SEC Reports
Floating Bars
Dynamic SampleSelectionDynamic DataDisaggregation
Sample andDisaggregateCounts
ComparisonCharts
IndividualResults
SEC Reports: Instructional Content
Coarse Grain Maps
Content Areas
by
Cognitive Demand
SEC Reports: Instructional Content
Fine Grain Maps
Topics
by
Cognitive Demand
Alignment as a Systemic Tool
Fine Grain
X 1 2 3 Y 1 2 3
z 1 2 3
Alignment= 0.6
C0.1 0 0
B0 0.1 0
0.2 0 0
A0.3 0.1 0
C
0.3 0.2 0
B 0.1 0.1 0.1
AA
B
C
0.5 0.1 0.1
0 0.2 0
0.1 0 0
Calculating Alignment1-((|x-y|(1-n))/2)
