Implementation and Evaluation of

Project REAL

Rural Early Adolescent Learning

Allen MurrayACRES

March 14, 2009

Key Personnel for Project Real

• Principal InvestigatorsThomas Farmer, Ph.D. & Jill Hamm, Ph.D.

• InvestigatorsKimberly Dadisman, Ph.D. & Linda Mason, Ph.D.

• Intervention StaffAllen Murray, Intervention Director

Kirsten FeilAbby Hoffman

• Project Support StaffMallory Vinson, Project Manager

Jenny Westrick Kelli O’Brien

Courtney Mann

What is NRCRES?

• The National Research Center on Rural Education Support (NRCRES) was established in 2004 with funding from the Institute for Educational Sciences of the U. S. Department of Education. The center is based at the University of North Carolina at Chapel Hill.

• NRCRES conducts a focused program of control-trial research that addresses significant problems in rural education. More than 40% of all American schools are in rural areas and 30% of all students attend rural schools. The research and development work of NRCRES will seek solutions that will improve the quality of rural education.

What is REAL

• In short, Project REAL is a Professional Development Program for teachers in rural schools (with a focus on teachers of early adolescents).

• It is designed to help them promote the academic, behavioral, and social adjustment of rural youth as they transition from childhood to adolescence.

• The program is designed to help support youth who are most likely to experience difficulty in school, but the strategies and techniques have been carefully selected to promote the engagement of all students including typical learners and high achieving students.

• A primary emphasis of this program is to help teachers establish classroom and instructional contexts responsive to a broad and diverse range of learning needs.

• Common Challenges: – Poverty and changing employment base; – Financial impact of large geographic distances;– Shrinking population of communities and schools; – Changing economic foundation of communities. – Teacher recruitment, retention, and licensure issues;

• Common Strengths: – Faculty are a part of the community; – Schools are focal point for community; – Home ownership rates above the national average – Generational affiliation with local school and/or school system; – Social indicators (church, civic clubs) show strong community ties,– Higher rates of participation in extra-curricular activities

Overview of Project REAL Schools

• Pilot Sites: – 2 Appalachian states, 4 middle school transition schools

• Full Implementation Sites:– Northern Plains: 4 k-12 schools (completed)– Appalachia: 4 middle school transition schools (completed)– Midwest: 2 k-8, 2 middle school transition schools (completed)– Southwest: 4 middle school transition schools (completed)– Deep South: 4 k-12 schools (completed)– Southeast: 2 k-8, 2 middle school transition schools (5/09)– Far West, 4 middle school transition schools (5/09)– Pacific Northwest, 4 k-8 schools (5/09)

Total Number of Districts/Schools: 28 districts/56 schoolsTotal Number of Teachers: 392Total Number of Students: approximately 4000

Training and Research Sites

Rationale for Project Design

• The wide variances in grade level Standard Course of Study made a

content-specific focus and a universal design mutually exclusive.

• So the focus is on addressing issues that impact academics for all

populations and across curricula.

• Small school size means the grade-level team is often very small (and

may consist of only one teacher).

• Small teams can cause teachers to work in isolation, with little opportunity to see others model new teaching techniques, curricula, or management strategies.

• The program combines a topic-focused training model with a teacher-focused consultation model.

Training vs. Consultation

Training

The agenda is owned by the trainer.

Trainer has information to give others.

Trainer’s job is to get that information out.

A good model when many folks need the same content.

ConsultationThe agenda is owned by

the teacher.

The teacher needs help in finding a solution.

The consultant’s job is problem-solving.

A good model when individuals or schools have unique needs.

Training is available to local schools; Consultation may be the greater need • In many ways it is easier to provide training than

consultation. Training is more time efficient, can be done in larger scale, can be made universal, and can be individually pursued (self-instructional modules, etc.).

• Consultation, by its very nature, is a person to person process; it is not adaptable to self-instructional models (by reading, videos, on-line, etc.). It is time-intensive, requires knowledge of both the teacher and community, and needs a face-to-face component.

Training Consultation

• Our training is not designed as stand alone,

• Instead, it is designed to set the stage for consultation,

• Consultation is the “real goal”, especially for special educators or teachers of students with disabilities,

• Consultation is student specific, with hope that it will generalize.

Sets the stage for

Project REAL Delivery • Project Real combines in-person training with a

technology enhanced (distance delivery) consultation to both regular education teachers and specialists who have “at-risk” students in their class

• Focus is on how the transition to early adolescence impacts academic performance, both in class-wide issues and for “at-risk” students (as defined by teacher).

• Prior to the beginning of intervention, relationships are formed during site visits. These relationships, and the in-person summer institutes, keep the video-conferences from feeling “artificial”.

• Site visits are made during the intervention year to observe classrooms and follow-up on training topics.

Project REAL InterventionUniversal Components

• Site Visits, Needs Assessments, and Summer Institutes

• Training modules on topics including:– Early Adolescent

Development– Motivation and Academic

Engagement– Instruction for low-achieving

students– School and classroom social

dynamics– Information processing– Literacy support and Writing

Targeted Components

• Video-conference consultation with Project REAL staff

• A forum for intervention specialists to facilitate discussions of:

– Training modules– Strategies for at-risk students– Strategies for class-side

issues

Sample Training Unit

• Each module typically consists of some or all of these components:

– Narrated PowerPoint Presentation,– Brief (non-academic) article,– Further readings on the topic for those teachers

interested in a more in-depth look at the topic.– A video illustrating a teaching technique or topic in

adolescent development.– An activity that teachers will complete with each

other or in class.

Research Design for Project REAL

• Intervention schools matched with control schools– 1/2 with middle school transition configuration– 1/2 alternative configuration (e.g., k-8, k-12)– Schools are matched closely on demographic data

• Baseline student data is collected in the spring of 5th grade and fall of 6th grade.

• Student outcome data on school adjustment and academic achievement collected in spring of 6th grade.

• A follow-up year of data is collected.(7th grade, fall & spring)

Plan of AnalysisHierarchical Linear Regression Models

• Control variables:– Baseline, fall scores as controls – Ethnicity (Native American, White), Gender

• Intervention Main Effect• Intervention by Ethnic Group, Intervention by

Gender effects

Tests for improvement to fit of model for block of variables;

Test for significance of beta weight within significant block of variables

• Perceptions of Social-Academic Context– Peer norms for effort and achievement (Hamm, 2001)

– Emotional risk of participation (Hamm & Faircloth, 2005)

– Bullying Context (Song, 2008)

• Affective Relationships with Schooling– Sense of Belonging (Hagborg, 1994; 1998)

– School Valuation (Voelkl, 1996)

• Achievement– End-of-year state-level standardized test (school records)

– Grades (school records)

Three Key Outcome Domains

Findings for Intervention Effects: Northern Plains Site

Site Description

• Recruited from all 4th, 5th, & 6th grade classrooms of four public k-12 schools in a state in the Northern Plains– 72% agreed to participate, N=165 student participants– 45% Native American, 55% White– 92 boys, 73 girls

• Schools were eligible for U.S. Department of Education’s Rural and Low-Income School Program (RLISP) – locale code 7 or 8 and at least 20% of students are from families

living below the federal poverty level– In our sample, 64.6% of students were eligible for subsidized

meals through the National School Lunch Act

After controlling for the previous year scores on each outcome, and for student demographic characteristics, students in Project REAL intervention schools, compared to their peers in matched control schools reported . . . – More supportive peer norms for effort and

achievement;– More peer protection from bullying;– A greater sense of school belonging;– Greater valuing of school;– Higher year-end grades.

Results

Intervention effects were particularly pronounced for Native American (versus White) students for:– State-level standardized test scores– Sense of belonging– Perceived peer norms for effort and achievement

State Standardized Test Scores: Pre-Intervention

639.6643.1

660.3658.2

625

630

635

640

645

650

655

660

665

Intervention Control

Raw

Sco

re (m

ean)

Native American

White

State Standardized Test Scores: Post-Intervention

664.4

656.6

667.2

671.8

645

650

655

660

665

670

675

Intervention Control

Raw

Sco

re (m

ean)

Native American

White

Students' Perceptions of School Belonging/Valuing

3.7

3.44

3.14

3.42

2.8

2.9

3

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

Intervention Comparision

Sch

ool B

elon

ging

, mea

n

Native American

White

Intervention Outcomes

1

1.5

2

2.5

3

3.5

4

4.5

School Valuing SchoolBelonging

Peer Protection

Avera

ge S

tud

en

t R

ati

ng

Intervention

Control

Teacher Efficacy

3.5

3.6

3.7

3.8

3.9

4

4.1

4.2

4.3

6th Grade, Fall 6th Grade, Spring

Comparison

Intervention

Hierarchical Regression Analysis of Indicators of Students’ Academic Achievement

Note: Standardized Betas reported: +p <.06, *p < .05, **p < .01, and ***p < .001 (one-tailed).

Standardized State Test End of Course Grades

1 2 3 1 2 3

Native American-0.13** -0.13** -0.27 -0.15* -0.16* -0.29**

Female0.03 0.03 0.01 0.27*** 0.26*** 0.36**

Prior year score (spring) 0.71*** 0.71*** 0.71*** 0.32*** 0.32*** 0.30***

Intervention 0.01 -0.09 0.27*** 0.25**

Intervention * Native American 0.23* 0.18

Intervention * Female0.03 -0.15

R20.58 0.58 0.59 0.24 0.31 0.33

F for change in R2

72.2*** 0.1 2.2* 15.9*** 15.2*** 1.5

Hierarchical Regression Analysis of Indicators of Students’ Affective Relationship with Schooling

School Valuing/ Involvement School Belonging

1 2 3 1 2 3

Native American 0.17* 0.14* 0.01 0.08 0.06 -0.18

Female 0.11 0.10 0.16 0.07 0.07 0.17

Current year score (fall) 0.38*** 0.36*** 0.35*** 0.55*** 0.53*** 0.50***

Prior year score (spring) 0.20* 0.20* 0.20* 0.10 0.07 0.08

Intervention 0.15* 0.13 0.12* 0.07

Intervention * Native American 0.19 0.33**

Intervention * Female-0.09 -0.13

R2 0.38 0.40 0.41 0.40 0.42 0.45

F for change in R2 18.2*** 4.4* 0.9 20.6*** 2.9* 2.9*

Note: Standardized Betas reported: *p < .05, **p < .01, and ***p < .001 (one-tailed).

Hierarchical Regression Analysis of Indicators of Students’ Perceptions of the Social Academic Context of Classrooms and School

Peer Norms Emotional Risk Bullying Protection

Variable 1 2 3 1 2 3 1 2 3

Native American

0.19** .17** -0.10 -0.13* -0.11 0.13 0.11* 0.07 -0.09

Female 0.09 0.09 0.01 -0.03 -0.02 -0.22* 0.06 0.05 0.01

Current year score (fall)0.47*** 0.46***

0.44***

0.42***

0.39***

0.38***

0.50***

0.51*** 0.47***

Prior year score (spring)

0.22** 0.18** 0.21** 0.16** 0.15* 0.13 0.23* 0.16 0.18**

Intervention 0.13* -0.05 -0.12 -0.130.21*** 0.11

Intervention * Native American 0.36** -0.33* 0.21

Intervention * Female 0.11 0.29* 0.07

R2 0.43 0.45 0.48 0.27 0.29 0.33 0.50 0.54 0.55

F for change in R2

22.6*** 3.2* 3.6*11.4*** 2.2 3.6*

30.1***

10.0*** 1.5

Note: Standardized Betas reported: *p < .05, **p < .01, and ***p < .001 (one-tailed).

Equivalence of Intervention and Comparison Groups on Selected Child and School Attributes

TotalComparison Intervention p-value

Female child, % 44.2 40.0 46.7 .410

Native American child, % 43.0 40.0 44.8 .555

White child, % 55.2 60.0 52.4 .347

Other race/ethnicity, % 1.8 0 2.9 .189

Child in 4th grade at baseline, % 30.9 25.0 34.3 .217

Child in 5th grade at baseline, % 38.1 38.3 38.1 .976

Child in 5th grade at baseline, % 30.9 36.7 27.6 .229

School involvement at baseline, mean 4.2 4.1 4.3 .161

End of course grade at baseline, mean 652.1 650.9 652.7 .711

School size, mean 115 90 140 .085

Eligible for Free lunch, % 53.8 51.9 55.6 .896

Minority students, % 38.5 41.9 35.1 .897

Pupil/teacher ratio 10.6 8.7 12.6 .058Note. There were 165 participants (105 intervention and 60 control students) and 4 schools (2 intervention and 2 control schools).