Thoughts on RelevanceRobert C. Granger, Ed.D.

Remarks prepared for Society for Research on Educational EffectivenessAlexandria, VA / March 1-3, 2009

2

• Why relevance?

• Relevant for what?

• Relevant for whom?

Main Messages

3

1. The R&D cycle of developing, testing, or identifying innovations at a small scale and then “scaling up” these pockets of excellence does not work.

2. The role of research is to better sift through practices implemented at scale, not to “discover” innovations.

3. Instead of focusing on individual student or teacher performance, focus on classrooms.

4

“There are many small pockets of excellence in the education market. But because of its fragmentation, few of these have been able to scale. With the proper incentives and infrastructure, many innovations could be surfaced or created and more importantly, adopted broadly, resulting in dramatic increases in education production – getting greater student achievement gains per dollar invested.”

Bill and Melinda GatesRoll Call Op EdFebruary 10, 2009

“A robust new federal Office of Educational Entrepreneurship and Innovation within the Department of Education would expand the boundaries of public education by scaling up successful entrepreneurs, seeding transformational educational innovations, and building a stronger culture to support these activities throughout the public sector.”

Sara Mead and Andrew RotherhamPaper for BrookingsMetropolitan Policy ProgramOctober, 2008

5

6

Comparison of Clinical Trial Phases

Phase I Objective: Determine the metabolic and pharmacological actions and the maximally tolerated dose.

Phase II Objective: Evaluate effectiveness, determine the short-term side effects, and identify common risks for a specific population and disease.

Phase III Objective: Obtain additional information about the effectiveness on clinical outcomes and evaluate the overall risk-benefit ratio in a demographically diverse sample.

Phase IV Objective: Monitor ongoing safety in large populations and identify additional uses of the agent that might be approved by the FDA.

Stages of Research in Research Prevention Cycle

7

8

IES Goal Structure for Grantmaking

Goal Description

1. Identification Projects

2. Development Projects

3. Efficacy and Replication Trials

4. Scale-Up Evaluations

5. Measurement Projects

9

A Snapshot: Intervention Trials in 2006

10

• National Institute of Mental Health (NIMH): 51 trials; 3 implementation/dissemination trials;

• National Institute of Drug Abuse (NIDA): 79 trials; 14 implementation/dissemination trials;

• National Institute on Alcohol Abuse and Alcoholism (NIAAA): 53 trials; 2 implementation/dissemination trials.

National Academies Table F-1, page 522.

11

12

“Contradiction and initially stronger effects are not unusual in highly cited research of clinical interventions and their outcomes. The extent to which high citations may provoke contradictions and vice versa needs more study. Controversies are most common with highly cited nonrandomized studies, but even the most highly cited randomized trials may be challenged and refuted over time, especially small ones.”

Ioannidis, John P.A., JAMA, 2005;294:218.

“Part II (of the report) describes the substantial progress in prevention science since 1994. It describes “numerous” (quotation marks added) efficacious or effective prevention programs…thus far, however, prevention programs have generally not been widely implemented in schools and communities and have done little to reduce behavioral health problems in America.”

National Academies, p. 295.

13

14

Barriers to Scaling Innovations

1. Program may not fit community needs, strengths, or capacities.

2. Real-world implementation may differ dramatically from the way originally tested.

3. Lack of ownership of the program.

4. Few evidence-based programs have the capacity to provide technical assistance and training.

5. An evidence-based program may not target outcomes relevant to community.

15

Barriers to Scaling Innovations (continued)

6. Key program components may be modified, thereby reducing outcomes.

7. Essential program components not always evident.

8. Lengthy period to develop community awareness, common vision, and program.

9. Potential for ineffectiveness or iatrogenic effects.

10. Challenges in obtaining funding for sustaining a unique program.

16

• Recruitment and training programs for liberal arts undergraduates

• Differentiated staffing with classroom aides

• Increased use of technology (TV, tape, teaching machines)

• Projects to introduce modern business methods into school management including a focus on teacher salaries. ($50 million in 1955 = $393 million in 2009).

The Decade of Experiment (1951-1961)

17

18

Comprehensive School Improvement Program: 12 Practices

1. Team teaching

2. Use of non-professional personnel in schools

3. Flexible scheduling

4. Variable-size pupil groups for instruction and new space arrangements

5. Use of audio-visual resources, including educational television

6. Programmed instruction

19

Comprehensive School Improvement Program: 12 Practices (continued)

7. Language laboratories

8. Educational data processing by machine

9. Independent study

10. Advanced placement and early admissions

11. Non-graded school programs

12. School and university partnerships for curriculum improvement, and pre- and in-service preparation

1. The purpose of a school is to promote learning.

2. Learning is a continuous process and must be related to an individual student’s abilities and needs.

3. Curriculum in all content areas should be built on a continuum from the beginning of the completion of formal education, rather than be frozen by grade levels or age of pupil.

4. There needs to be a constant and continuous examination of the ways by which schools facilitate learning in order to take advantage of discoveries and developments.

20

Comprehensive School Improvement Program: Key Assumptions

21

The Summative Judgment on “Lighthouse” Projects

“There was not a willingness on the part of the project’s neighbors to acknowledge its light giving nature…”

A Foundation Goes to School, p.42.

22

“We must instill in our students the expectation of tedium and disappointment and the duty of thorough persistence, by now well-achieved in the biological and physical sciences. We must expand our students’ vow of poverty to include not only the willingness to accept poverty of finances, but also poverty of experimental results.”

Campbell and Stanley, p. 2 & 3.

A Brief Aside

23

24

Another Approach (Practice at Scale)

“To avoid disillusionment several aspects may be noted. First, the claims made (earlier) for the rate and degree of progress which would result from experiments were grandiosely over optimistic and were accompanied by an unjustified depreciation of non-experimental wisdom.”

“The initial advocates assumed that progress in the technology of teaching had been slow just because (italics in original) (the) scientific method had not been applied: they assume traditional practice was incompetent just because it had not been produced by experimentation.”

“..applied practice and scientific knowledge are seen as the resultant of an accumulation of selectively retained tentatives remaining from the hosts that have been weeded out by experience.”

“..but the selective, cutting-edge of this process of evolution is very imprecise in the natural setting.”

“Experimentation enters at this point as the means of sharpening the relevance of the testing, probing, selection process. Experimentation thus is not in itself viewed as a source of ideas necessarily contradictory to traditional wisdom. It is rather a refining process superimposed upon the probably valuable cumulations of wise practice.”

Campbell and Stanley, p. 4.

Evolutionary Perspective on Cumulative Wisdom and Science

25

26

We know that about 15% of the variance in school-induced change in achievement is between schools and the rest is in the schools and that at least in relative terms, teachers who add value are widely dispersed. We also know that teachers matter but we are not sure why or how.

In elementary school classrooms, 30% of the time is spent on administrative matters.

Why Classrooms?

• Such settings are where development occurs.

• Attempts to change individual behavior without affecting its setting are rarely effective.

27

28

Conceptual Articles on Classrooms as Systems

Tseng, V., & Seidman, E. (2007). A systems framework for understanding social settings. American Journal of Community Psychology, 39(3-4), 217-228.

Cohen, D. K., Raudenbush, S. W., & Ball, D. L. (2003). Resources, instruction, and research.

Educational Evaluation and Policy Analysis, 25, 119-142.

29

Classrooms as Systems

Social Processes

Resources

Organizationof

Resources

MacroForces

Youth Outcomes

MacroForces

Social Settings

30

Bob Pianta and colleagues at the University of Virginia

http://www.virginia.edu/vpr/CASTL

31

Reed Larson and colleagues at the University of Illinois at Urbana-Champaign

http://www.youthdev.uiuc.edu/

32

Charles Smith and colleagues at the David P. Weikart Center for Youth Program Quality

For staff: http://www.forumforyouthinvestment.org/about/staff

For content: http://www.highscope.org/Content.asp?ContentId=117

• Classrooms are the unit of analysis.

• Improvements in measurement.

• Improvements in non-experimental causal analyses.

• Mixed-method work.

• An engineering approach to the improvement of change strategies.

Going Forward (Including a Modest Proposal for Secretary Duncan or Some Entrepreneurial Researchers)

33

34

Dear Mr. Secretary:

• $1 billion (100 @ $10 million).

• Make the welfare waiver approach work for education.

• Study shifts in policy but put classrooms in the middle.

35

THANKS!