Curriculum Renewal: Fidelity of Implementation

WERA/OSPI State Assessment Conference—Seattle Airport Hilton December 4, 2008

Peter Hendrickson, Ph.D.Everett Public Schools

Program Evaluation

Fidelity of Implementation Program Model Theory of Action or Logic Model To what extent, consistent?

Program Impacts Knowledge Skills Dispositions

Theory of Action What are you doing?

Who are you doing it with?

What resources do you have?

What effects do you expect?(Green, 2008)

Logic Model fragmentInputs Methods Outputs Short

Term Outcomes

Medium Term Outcomes

Long Term Outcomes

TrainerFacilitatorLead tchrPrincipals

OverviewIntenseFolo UpDrop inStudy Grp

All attend85% attend50% use

Tchr uses new methodK-2 form ltrs new method

Score classmate reliably

Students apply in school settings Grd 4, 5

AlignmentNew textsTchr guide

Post docsTrade oldHalf dayFAQ

All obs in 1 qtrHelp accessed

Note legible in parent conf

Use rubric to self assess

Apply in leisure settings MS, HS

FOI Critical Components Structural/

procedural Structural/

educative Instructional/

pedagogical Student

engagement

What to do

Need to know

Instructional strategies

Expectations for students

(Century et.al., 2007)

Degree implemented

On Model, Off Model

Various Models, e.g. Read 180 90 minute Local 55 minute HS Local 45 minute MS Local 45 minute, plus 30 SSR

Direct Observation--Tools Observation

protocol Train observers Timely feedback Additional

information Use the data

FOI Framework, p.1Component Not Present Partial Adequate Substantial

Fidelity

What to do

Procedures

Physical Organize

Needs to Know

Content

Pedagogy

FOI Framework, p. 2Component Not Present Partial Adequate Substantial

Fidelity

Instruct Strategies

Prior LearnDirect Instr

Guided Prac Assessment

Engage-ment

Instruction-al

Student

Zoomerang survey Objectives Design Format, pilot Administer, remind Organize data Analyze Report

Focus groups Representative Possible time Field test Conduct Transcribe,

validate Content analysis

Interviews Go to their turf Probe for key

elements Keep short Listen for

unexpected Don’t argue Act on findings

Software monitoring Who enrolled Time on software Books read Vocabulary growth Comprehension

growth Coasting?

Student Information System Present Tardy Gender Ethnicity Prior learning Special programs

References Bamberger, M., J. Rugh, & L. Mabry. (2006). Real World Evaluation.

Thousand Oaks, CA: Sage Publications. Three veteran program evaluators reveal tricks of the trade with limited resources, time or commitment.

Borman, G., Slavin, R.E., Cheung, A., Chamberlain, A., Madden, N.A., & Chambers, B. (2007). Final reading outcomes of the national randomized field trial of Success for All. American Educational Research Journal, 44 (3), 701-731. When implementing Success for All, considerable emphasis is placed on fidelity. External evaluators, extensive training for building principals, building-wide commitment and exhaustive instructional rubrics set the program apart from most other literacy models.

CEMSE. (2007). Fidelity of Implementation Questionnaires for Mathematics Curriculum in Grades 1-5. , Chicago, IL: University of Chicago.

CEMSE. (2008). Critical Components: Definitions and Explanations. Retrieved November 21, 2008 from the Worldwide Web at http://cemse.uchicago.edu/files/CCDefinitions summary 2008_11_110.pdf This fully fleshed math and science FOI checklist provides a useful base document for creating local checklists.

References Century, J., Freeman, C., Rudnick, M., & Leslie, D. (2007). A

conceptual framework for fidelity of implementation of instructional materials. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY. March 28th 2008.

Fullan, M. (2001). Leading in a Culture of Change. San Francisco, CA: Jossey-Bass.

Hendrickson, P. (2008). Unpublished principal interview protocols for several program evaluations. Everett, WA: Everett Public Schools.

--. (2006). Read 180 Placement, Assessment and Reporting Guide. NYC, NY: Scholastic. The PARG is a companion tool to the Research Protocol and Tools (2007).

Kerman, Sam. (1979). Teacher Expectations and Student Achievement. Phi Delta Kappan, v.60, n.10, p. 716-18, June 1979.

National Sciences Resource Center. (1977). Science for All Children. Washington, D.C.: National Academy Press.

Zoomerang is available free via www.zoomerang.com on the Worldwide Web to implement surveys for limited use. A full license is less than $500 per year. Survey Monkey works, too.

Contact Information

Peter Hendrickson, Ph.D.Curriculum SpecialistAssessment, Research, Program EvaluationEverett Public Schools

Tel: 425.385.4057E: phendrickson@everettsd.org