Date post: | 17-Jan-2016 |
Category: |
Documents |
Upload: | kelly-sutton |
View: | 214 times |
Download: | 0 times |
DVAS Training 10-03-05
• Find out how Battelle for Kids can help
Presentation Outcomes
• Learn rationale for value-added progress measures
• Receive conceptual overview of value-added analysis
• View sample value-added reports
• See how value-added information
fits with school improvement
Why are traditional achievement measures alone an insufficient way to assess student achievement?
Rationale for Value-Added Progress Measures
The ChangingEducational Landscape
• 2000’s: Standards-based education
• 1960’s: Mastery learning
• 1970’s: Behavioral objectives
• 1980’s: Minimum competencies
• 1990’s: Outcomes-based education
Looking at the changing educational landscape, a clear pattern exists:
The focus has moved from what goes into a child’s education to what comes out of the process
What’s the Upshot?
From Teaching Inputs
• Context (room, furniture, master schedule, course of study)
• Resources (number/quality of books, computers, materials)
• Capacities (knowledge of subject and teaching/learning processes, classroom control, lesson planning)
To Learning Outputs
• State content standards and local curriculum aligned to standards
• Annual paper/pencil tests to measure achievement
• State report cards at district/building level
• Good teaching = High student performance
The primary measure in “output focused” system is student scores on statewide achievement tests.
The Shift in Focus
Stair-Step Expectations
Most achievement measures imply:
• Achievement test scores are enough to show growth
• Students start at the same place
• Students progress at the same rate
Reality
• Students start at different places
• Students progress at different rates
• Educators need more than individual test scores to evaluate school’s impact on student learning
Need for Progress Measures
To measure school effectiveness, we must pay attention to passage rates AND annual student progress
How do we maximize student progress each year, regardless of where they start?
85% of the public believes student progress is
the best measure of a school’s effectiveness
Question for EducatorsToday
— Phi Delta Kappa/Gallup Poll 2005
Conceptual Overview of Value-Added Measures in Ohio
How is performance data used to produce value-added information?
Two Value-Added Systems in Ohio
• Project SOAR
• Ohio’s Accountability System
Project SOAR
• Operated by Battelle for Kids• Began in 2002 with 42 school districts• Now has 106 districts and 3 charters schools• Provides analysis in 5 subjects for grades 3-10• Uses state and non-state test data• Uses a prediction based value-added approach• Expected growth is normative (“Average
Growth”)
Ohio’s System
• Operated by the Ohio Department of Education• Begins as a 4th grade pilot in 2006 in all
districts• Provides analysis in math & reading
in grades 4-8• Uses only state achievement tests• Uses a mean gain value-added approach• Expected growth is likely to be a fixed amount
What Do the Two Have in Common?
• Utilize the power of longitudinal data and linking together each student’s assessment data over time
• Compare students’ current test scores to baseline scores
• Provide value-added information in Web based reports
• Use the statistical power of EVAAS™ to produce the value-added analysis
Why EVAAS™ in Ohio?• EVAAS™ is a the value-added
methodology pioneered by Dr. William Sanders of SAS
• Applies most sophisticated statistical methodologies available to ensure reliability
• Allows for the use of all student test data• Provides valuable diagnostic information• Approaches for handling different types of
test data• Identified by RAND and others as a
preferred model• Used statewide in Tennessee for 10 years
What is Value-Added?Value-added in simplest form is an accurate measure of the present (observed scores) minus an accurate measure of the past (baseline scores) for the same group of students.
Mean Observed Score - Mean Baseline Score = Value-Added
Math ScoresYear 1 3rd Grade
Student 1 350Student 2 370Student 3 360Student 4 375Student 5 365
Year 2 4th GradeStudent 1 400Student 2 385Student 3 395Student 4 405Student 5 390
Mean Baseline Score 364 Mean Observed Score 395
31 would be a crude measure of the value-added
Why are Two Systems Needed?
• When both tests are on a common scale like the Ohio achievement tests, baseline can come from
the prior year’s test scores (Mean Prior Score Approach - Ohio System)
• When the tests are on different scales the
baseline must be calculated. (Mean Predicted Score Approach - Project SOAR)
Sophisticated statistics are required in both approaches to ensure that all students’ data are included, that the information is reliable, and to add predictive diagnostic power.
Different approaches are needed to provide reliable baseline scores from the different kinds of tests used in Ohio
What are Common Scales?• Vertical Scales increase in equal
intervals as you increase grade levels
• Horizontal Scales remain the same as you increase grade levels
Score at the 3rd 4th 5th 6th 7th 8th50th percentile 350 400 450 500 550 600
Ohio Achievement TestsProficient 3rd 4th 5th 6th 7th 8thScore 400 400 400 400 400 400
When you have common scales, the prior years’ scores can be used as the baseline.
How Are Test Data Used When They Are Not on a Common Scale?
• All available test data is collected and linked for each student
• Districts are grouped into pools based on common testing histories
• Relationships between and among all tests in pool are used to create predicted baseline scores
What students’ like them would typically score on this year’s test.
• Collect all individual student data available for a minimum of three years
• Link each student’s annual test data together to create a longitudinal record
How Do you Create a Longitudinal Record?
How Are Comparison Pools Created?Districts grouped into pools based on common testing histories at each grade-level cohort
Example: Mike’s cohort testing history or pool
How Are the Relationships Between Tests Within a Pool Defined?
Relationships between and among all tests in the pool are calculated and can be represented as a number or correlation.
Example: using 2 years of and 4 years of Mike’s testing history
How Much Prior Data is Used?
Up to 5 years of student test data and the relationships between tests are used to calculate predicted baseline score for this year’s subject-area tests.
How Are Predicted Scores Calculated?
Using the test data for students with similar prior performance on common
tests and the tests’ relationships to each other allows for the creation of
statistically reliable predicted scores for each student in each subject
Mike’s Prediction
Student Prediction 2
Student Prediction 3
Student Prediction 4
Student Prediction 5
Student Prediction 6
Student Prediction 7
Student Prediction 8
Student Prediction 9
Student Prediction 10
Student Prediction 11
Student Prediction 12
Student Prediction 13
Student Prediction 14
Student Prediction 15
Student Prediction 16
Student Prediction 17
Student Prediction 18
Student Prediction 19
Student Prediction 20
Are All Students Used in the Analysis?
Mike’s Prediction
Student Prediction 2
Student Prediction 3
Student Prediction 4
Student Prediction 5
Student Prediction 6
Student Prediction 7
Student Prediction 8
Student Prediction 9
Student Prediction 10
Student Prediction 11
Student Prediction 12
Student Prediction 13
Student Prediction 14
Student Prediction 15
Student Prediction 16
Student Prediction 17
Student Prediction 18
Student Prediction 19
Student Prediction 20
Mean Predicted Score “Baseline”
Mike’s Score
Student Score 2
Student Score 3
Student Score 4
Student Score 5
Student Score 6
Student Score 7
Student Score 8
Student Score 9
Student Score 10
Student Score 11
Student Score 12
Student Score 13
Student Score 14
Student Score 15
Student Score 16
Student Score 17
Student Score 18
Student Score 19
Student Score 20
Mean Observes Score
Your School
Only students with enough prior data to create a predicted score are included.
Mean Student Score – Mean Predicted Score (with some statistical reliability factored in)
= School Effect
How Do you Estimate the School’s Effect on Student Growth?
Sample Value-Added Reports
What information do value-added reports provide that was previously unavailable?
Achievement & Progress
High Achievement, High Progress
High Achievement High Mean Scores = 89% passage
High Progress Positive School Effects
2005 School Value-Added Report for OPT Math
High Achievement, Low Progress
High Achievement High Mean Scores = 85% passage
Low Progress Negative School Effects
2005 School Value-Added Report for OPT Math
Low Achievement, High Progress
Low Achievement Low Mean Scores = 69% passage
High Progress Positive School Effects
2005 School Value-Added Report for OPT Math
Student A Report
Student B Report
Student B Projection
Connecting to School Improvement Efforts
How can value-added progress measures enhance school and district improvement at the:
• District Level• School Level
District Improvement Efforts
• Identify patterns of progress across buildings, grade levels and subject areas
• Locate areas of strength to build upon• Locate areas for improvement
District Value-Added Summary Report
2005 District Value-Added Summary Report4th Grade
School Search Report
2005 School Search 4th Grade Math
School Improvement Efforts
• Identify patterns of progress across grade levels, subject areas and student subgroups
• Locate areas of strength to build upon• Locate areas for improvement
School Value-Added Report
2005 School Value-Added ReportReading
Performance Diagnostic Report
2005 Performance Diagnostic for Reading 5th Grade Means
School Diagnostic Report
2005 School Diagnostic for Reading 4th Grade Means
Search for Students
By Subgroup or Achievement
Student Search
Identified At-Risk Students
Student Search Results
In Summary, Value-Added Information Shows…
• How much progress students make in each subject area and grade level
• How much progress students at different previous achievement levels have made
• How students’ progress in one curricular area or program compares to their progress in another
• Whether individual students are making adequate progress to meet state standards
Without data, all we have are opinions!
School Strategic Planning Cycle
Pre-School Start of the Year Meetings• Examine value-added and other school performance
information by grade level and/or subject area
• Assess strengths and weaknesses and potential
actions, grade level by grade level, subject by subject
• Celebrate progress
• Set 1-2 goals for each grade level, department,
or team around strengths and weaknesses
• Create action plans and accountabilities
Grade level, department and/or team meet to work on team specific goals
Grade level, department and/or team meet to work
on team specific goals
Grade level, department and/or team meet to work on team specific goals
Select a lead person in the school district who understands value-added information, and can access, interpret and conduct presentations
Train school leaders (principals and lead teachers) to access, interpret and conduct presentations
Have school leaders share value-added information with school staff members
Have school staffs use value-added information to assess annual progress and set goals for next year
Implementation Checklist
Ohio’s Scale Up Plan
For more information, contact:
www.BattelleforKids.org
(866) KIDS-555