Date post: | 03-Jan-2016 |
Category: |
Documents |
Upload: | abigail-bell |
View: | 215 times |
Download: | 2 times |
Cross-Cutting Issues in Recent State Growth Modeling Efforts
Andrew Ho, DiscussantHarvard Graduate School of Education
Measuring Growth: A Key Feature of the Next Generation of AssessmentsNational Conference on Student Assessment
Friday, June 29, 2012
1) Lisa Keller’s presentation by Stephen Murphy– I’ll reflect on Lisa’s excellent overview of growth
modeling issues and make some explicit recommendations.
2) Richard Vineyard (and Carol Crothers), Nevada– SGPs with an eye toward EVAAS
3) Juan Copa, Florida– Categorical + Gain … + Covariate-Adjustment VAM
4) Maridyth McBee, Oklahoma– Categorical … + VAM
Four Presentations
• Lisa and Stephen raise a number of critical issues:– Defining “growth,” aligning models to purposes,
vertical scaling, missing data, background variables, error bands, transparency.
• I might push Lisa (and many academics, including myself) to be more explicit about what to do in particular situations.– We do a lot of “sensitivity studies” that show that
results differ across models. We don’t as often address why, or when Model A is better than Model B.
– Not only, “here are some important issues to consider,” but also, “and if you find X or care about Y, then you must do Z.”
Lisa Keller by Stephen Murphy (1)
• Shameless self-citation: Katherine Castellano (UC Berkeley) and I have a practitioner-oriented “Guide to Growth” due out this summer under CCSSO (contact me if you’re interested in a draft).– We cross-classify seven growth models by seven critical
questions and criteria.– I’ll be extending our framework to discuss the growth
models in Nevada, Florida, and Oklahoma.
• Lisa and Stephen’s presentation does an excellent job of listing some of the issues we consider and others that we should have discussed more than we did (particularly missing data and error bands).
Castellano and Ho’s Practitioner’s Guide to Growth (1)
Practical definitions of “growth” and “growth model”
• Status describes the academic performance of a student or group at a single point in time.
• Growth describes the academic performance of a student or group over two or more time points.
• A Growth Model is a collection of definitions, calculations, and rules that quantifies student performance over two or more time points and supports interpretations about students, their classrooms, their educators, and their schools.– With apologies (and sympathy) to the “growth purists,”
our call is to describe models as they are.
• Nevada uses SGPs but is considering “value added” alternatives such as EVAAS.
• Comments and suggestions:– Try mean SGPs instead of median SGPs.
• More shameless self-citation: Katherine Castellano and I have a paper (scholar.harvard.edu/andrewho) that shows considerable advantages to means, including
– much smaller theoretical standard errors– less variability in year-to-year comparisons– less scale-dependency– stronger correlations with familiar metrics.
• Measurement purists will tell you that you can’t average percentile ranks. We show that a simple tweak will satisfy many of their concerns while retaining the considerable advantages of averages.
– In my opinion, a move to EVAAS would improve the argument for “value-added” inferences by a small degree but sacrifice transparency and interpretability.
Richard Vineyard (and Carol Crothers), Nevada (2)
• Florida uses a hybrid categorical model and gain-based model and is adding a value-added model.– The covariate-adjustment value-added model, like the
SGP approach, is a “conditional status” model that does not align with “intuitive” descriptions of growth.
– Adjusts for student-level and classroom-level characteristics in an attempt to compare teachers to “similar teachers” as defined by the model.
– Could create some interesting incentives, where a teacher could “acquire” certain types of students to change their comparison group.
– Adds to existing incentives to “acquire” students that might have artificially low prior scores.
Juan Copa, Florida (3)
Sidebar: Conditional Status (SGPs, some VAMs)
Bertucci's
B. GoodBoloco
Border Cafe
Cafe Aliers
Cafe of India
Cafe Sushi
Cambridge Common
Cambridge 1Casablanca
Charlie's Kitchen
Chez Henri
Daedalus
Darwin's Dolphin Seafood
Finale
Fire & Ice
Garden at Cellar
Grendel's Den
Harvest
Henrietta's
Hi-Rise
Hong Kong
John Havard's
Legal Seafood
Le'sMr. Bartley's9 Tastes
Noir
Om
Pinocchio's
Red House
Redline
Rialto
Russell House
Sandrine's
Spice Thai
Tamarind Bay
Tanjore
Ten Tables
Tory Row
Upper Crust
Upstairs/Square
Veggie PlanetWagamama
AsmaraBaraka Cafe
Basta Pasta Bertucci's
Central Kitchen
Craigie on Main
Cuchi Cuchi
Dosa Factory
Flour Bakery
Green St.
India Pavilion
Koreana
Mary Chung
Middle East
Middlesex
Miracle of SciencePicante
Plough & Stars
Rendezvous
River Gods
Royal East
Salts
Sidney's
Tavern in Square
1369 Coffee
Zuzu!
1520
25Z
agat
ave
rage
sco
re (
0-3
0)
10 30 50 70Zagat estimated price of dinner with drink and tip ($)
Ratings of restaurants in Harvard and Central Squareplotted on the estimated cost of dinner.
The location of a restaurant with respect to the regression line is its rating compared with expectations given price. Flour Bakery is more accurately described as having higher rating than expected given cost than “high growth.”
• Oklahoma uses a categorical model. From our guide…
Maridyth McBee, Oklahoma (4)
Appealing Features: Transparency, communicability, simplicity.
Things to consider:•Loss of information due to coarse categories.•Selection of articulated cut scores represents an implicit vertical scale.
Value Added• Following the philosophies of many others (Braun,
2005; Reardon & Raudenbush, 2009; Rubin, Stuart, & Zanutto, 2004), I consider “value added” to describe a hypothesis, not a model.– School and teacher “effects” are akin to an average
student status beyond or below expectations. – We must remember that our expectations can and
should change as we base our expectations on different variables. So, too, will “effects.”
– Extremes are worthy of further investigation and require convergent evidence for causal attribution.
Value Added• Use and interpretation of “value-added” scores are far
too often devoid of any theory of action for the improvement of teaching or school leadership.– In medicine, one could…
• analyze data• use results to identify symptoms• consider symptoms to arrive at a diagnosis• use the diagnosis to prescribe a treatment.
– With teacher value added models, we…• analyze data• use results to rank teachers on a single scale… and then?
• Many states have thoughtful teacher evaluation systems for the improvement of teaching. – We might remember that VAMs are just one element in this
system. Thank you! scholar.harvard.edu/andrewho [email protected]