presented to Volusia Teachers Organization

AFT Innovation Grant SubcommitteesOctober 28, 2010

Dr. Serena Roberts

Think about the term “teacher effectiveness” and what it means to you

Turn to a partner and share your thinking

Based on what both of you have shared, come up with a common understanding to share with the group

How does your collective understanding of teacher effectiveness relate to teacher evaluations?

How does it relate to assessment-based accountability?

By the end of this session, participants will:Understand terms relating to measurements of

teacher effectivenessBe aware of other state’s value-added models

and the researchers associated with themKnow possible biases of different value-added

modelsBe able to compute simple gain scoresBe prepared to grapple with difficult questions

about teacher effectiveness and ways different measurement models can play out in assessments of teacher effectiveness

Part of a national movement

The role of AFT’s Innovation grant

Current focus with Race to the Top (RTTT) in Florida

2003 Rand Report, Evaluating Value-added Models for Teacher Accountability

“Although VAM holds great promise, it also raises many fundamental and complex issues. . . . If these issues are not adequately addressed, VAM is likely to misjudge the effectiveness of teachers and schools and could produce incorrect generalization about their characteristics, thus hampering systematic effort to improve education” (p. iii).

The entire Spring 2004 issue of Journal of Educational and Behavioral Statistics was devoted to value-added topics.

In April 2008 the University of Wisconsin at Madison hosted the inaugural National Conference on Value-added Modeling.

Value-added assessmentusually refers to a specific technique or assessment

settingValue-added measurement

is usually used in discussing attempts to measure such intangibles as the impact of school environment, the long-term effect of prior teaching, teacher experience, and socioeconomic characteristic

Value-added modelingrefers to the full range of statistical and mathematical

formulations encompassing all value-added assessment techniques

Layered Mixed Effects Models (LMEM)ignore sociodemographics, apportion credit for pupil

learning growth among multiple schools, use incomplete observations

Simple Fixed Effects Models (SFEM)highly correlated to LMEM so use of SFEM is preferred

because of their simplicity; however, use of SFEM rather than HLM can be controversial

Hierarchical Linear Models (HLM)The question of whether to adjust for minority status

and poverty “is, perhaps, the most important issue faced when developing a school accountability system. Either inclusion or exclusion of them is likely to lead to a biased system. Which bias is most tolerable may depend on whether the system is to be a high-stakes one” (Tekwe, et al, 2004, p. 12).

Tennessee Value-Added Assessment System (TVAAS)Measures teacher effectiveness based on student gains,

controlling for socio-economic status (SES) and other factors that influence achievement

Before 2004 the model was criticized for absence of explicit control. In 2004 they introduced controls of SES and demographics. Controls had negligible impact on estimated teacher effects in TVAAS, but not on a SFE estimator with which the TVAAS was compared. Their explanation was that the use of covariance of tests in different subjects and grades results in a student’s history of test performance substituting for omitted background variables.

Uses LMEMSanders

LouisianaRedesign of teacher preparation programs and use

of effectiveness of experienced certified teachers as the point of comparison

Includes aspects of cost-benefit analyses and a skills-as-outcomes model

Looks at which practices are most effective and what characteristics of teachers are most powerfully tied to their instructional success

Coding and analysis of data about practices and characteristics to assess teacher effects

Results to be used to shape hiring practicesHLM uses prior achievement, Limited English

Proficiency, gifted status, section 504 status, free or reduced price lunch status, student attendance, disability status, discipline record

Noell

One way to look at the value one’s teaching adds is to gather data to document how one’s students do when they move to the next course.How do students who earn A’s in reading in your 2nd

grade class do on their 3rd grade FCAT Reading exams?How do students who earn A’s in your Pre-Algebra class

do in Algebra? The value added by changing a teaching practice

sometimes called Response to Intervention (RTI)Non-statistical value added

“When people talk about their experience with a really good teacher, they’re not talking about test scores. . . . They’re talking about a teacher who gave them self-confidence, the ability to learn, an interest and curiosity about certain subjects” (Pallas, 2010, as quoted by Stewart).

Subcommittees:

Value Added

Formative Assessment

Alternative Assessment

Please get into the following groups for this data exercise

Elementary Middle HighAnthony JohnsonBecky PittardTrish Randall

Diane ClowDavid FinkleYania GonzalezLekita Howard

Beth BosseRon PaganoTom RussellMary Thomas

John CashAlba PerezSusan PersisMarilyn Travis

John AtkinsMatt KrajewskiVickie Presley

You will be given a data activity packet consisting of 8 sheets. Directions for each set of data are at the top of each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next.

If you have questions, please do not hesitate to ask for help.

Teacher 1, in School F, has a 74.3 average gain score in reading.

Teacher 2, in School A, has a 150 gain score in reading.

Based on this data, what can you conclude about the effect of each of these teachers on pupil learning?

What other data would be helpful in coming to a conclusion?

The importance of school effect

The New Teacher Project“Teacher effectiveness should consider multiple

sources of data and emphasize student learning outcomes.”

Teacher evaluations and support for effective instruction (October 6, 2010)

Teacher Evaluation 2.0 (October 2010)The Widget Effect: Our National Failure to

Acknowledge and Act on Differences in Teacher Effectiveness (2009)

Wisconsin Center for Education ResearchUsing Teacher Performance Assessments for Human

Capital Management (April 2010)New York Daily News

“New York City plan to grade teachers with ‘value-added’ data is destructive, says Diane Ravitch” (October 25, 2010)

Subcommittees:

Value Added

Formative Assessment

Alternative Assessment

If three teachers are ranked high, average, and low, using value-added methods, what is the difference among them in terms of the average number of test questions answered correctly by their students?

How many more correct answers does it take, on average, to raise a teacher from the low to the average ranking?

What are the factors on which these answers depend?

Which of the 13 RTTT projects require knowledge of what we talked about today for effective implementation?

Expand Lesson Study *Expand STEM Career and Technical Program Offerings Increase Advanced STEM Coursework Bolster Technology for Improved Instruction and

Assessment Improve Access to State Data Use Data to Improve Instruction Provide Support for Educator Preparation Programs Improve Teacher and Principal Evaluation Systems Use Data Effectively for Human Capital DecisionsFocus Effective Professional DevelopmentDrive Improvement in Persistently Low-Achieving Schools** Implement Proven Programs for School Improvement** Include Charter Schools in LEA Planning

One of the results of the Louisiana studies is: It is possible for teacher preparation programs to

prepare new teachers whose students demonstrate achievement comparable to the achievement of students taught by experienced teachers.

How will this effect traditional pay scales?

How will it effect the improved teacher and principal evaluation systems mandated by Race to the Top?

Please move to your grant subcommittee groups.

You will be given a data activity packet consisting of 4 sheets. Directions are on each sheet. Please do not look at subsequent sheets until you complete the previous sheet or are told to move to the next.

If you have questions, please do not hesitate to ask for help.