E. Christine Baker-Smith Christopher C. Weiss Vanessa G. Ohta

Differential Success or Different Populations: Variations Across Sites within Penny Harvest

Program Evaluation

E. Christine Baker-SmithChristopher C. Weiss

Vanessa G. Ohta

Quantitative Methods in the Social Sciences (QMSS)Institute for Social and Economic Research and Policy

Columbia University30. October. 2010

Outline/Overview Previous Research on Service Learning Evaluating Service Learning – The Penny

Harvest Program in New York City & Columbus, Ohio• The Penny Harvest Program• Evaluation/Research Design

Using Factor Analysis to assess “program success”• Factor Analysis Theory• Our Factors• Variations by site

Previous Research on Service Learning Evaluation of Service Learning and

Civic Instruction has grown in past decade:• Service learning programs enhance host

of outcomes: Personal efficacy and enhanced social skills Academic learning Critical thinking skills Sense of social responsibility Civic engagement Increased later service

(Astin & Sax, 1998; Smith 2007; Billig 2000, 2003, 2004)

Previous Research on Service Learning Yet many of the research designs used have

significant limitations:• Focus on older students – usually middle

school and high school age. Artifact of program focus – most service learning programs oriented toward older students.

• However, some research documents benefits of service learning for younger grades. e.g., Michigan Learn and Serve study finds grater benefit for students in grades 2-5 than for those in older grades.

(Billig 2004; Billig and Klute 2003; Smith 2007)

Previous Research on Service Learning

Likewise, most research focuses on one particular group using:• Cross-sectional analysis• Self-selected samples• Single location analysis lacking

comparative possibilities Are benefits location and program-site specific?

Program fidelity, site comparability, etc. Are there ways to compare programs and sites

with regard to outcomes?

Evaluating Service Learning:The Penny Harvest Program in

NYC One evaluation of a program focused on

younger grades: The Penny Harvest Program

Founded in 1991 by Common Cents. Penny Harvest is an inclusive year-round program• Designed to develop community values through real-

world service experiences. Penny Harvest Program consists of four

integrated stages.

Scale and Recognition• The Common Cents Penny Harvest is now in

over 850 New York City schools, serving over half a million children.

• The program and curriculum guide have been internationally recognized • Named one of four top programs in international

competition on civic engagement by Bertelsmann LP

• Recognized by Bridgespan as a leader in the youth development field.

• Common Cents was recently selected as a key partner in NYC Service, Mayor Bloomberg’s initiative to increase civic participation amongst all New Yorkers.

• The organization has replicated the model it in six (primarily urban) learning sites across the country.

• New York region; New York City; Columbus, Ohio; Denver, CO; Seattle, WA; Florida

Penny Harvest“Transforming the multi-million-dollar

resource of idle pennies into the philanthropic property of children”

Stage 1:PENNY HARVEST

• Kick-off•Wheels of Caring• Collection





Stage 2:ROUNDTABLE

• Student Leaders• Group

Decision-Making• Grant

Awards





Stage 2:ROUNDTABLE



Awards

Stage 3:SERVICE

• Service in Community





Stage 2:ROUNDTABLE



Awards

Stage 3:SERVICE

• Service in Community

Stage 4: Reflect & Plan• Check

Presentation Ceremony

• Reflection time

School Student Body

Community Based

Organizations PH Student Leaders

PH Coach(es)

Common Cents/ Affiliate Partners

How The Common Cents Method Unites a School Community: Dynamic Development

Teachers

Dynamic Development

Program Goals for Multiple Levels Program goals (growth,

implementation of model)

Student goals

Teacher goals

Community goals

Student Goals

A. Efficacy

B. Connectedness

C. Ethical Development

Student Goals Measures

A. Efficacy

B. Connectedness


1. Academic Achievement2. Engagement3. Behavior in School4. Voice/Capacity5. Helping Others 6. Community

Orientation7. Self v. Others


A. Efficacy

B. Connectedness





A. Efficacy

B. Connectedness




Evaluating Service Learning: Research Design

Measurement techniques• Questions from 3 sources

1. University of California at Berkeley’s Service-Learning Research and Development Center’s Civic Responsibility Survey National Survey of Student Engagement (Furco, Muller, & Ammon, 1998)

2. MacArthur School Engagement Survey (Blumenfeld, 1998)3. Penny Harvest Pilot Roundtable Survey 2005-06

• Benefits: Generalizability (1&2):

Used in multiple locations nationally Used on various student-groups by age, gender, ethnicity & urbanity

Internal Validity (3): Used on similar population to test same program. Allows for modification of instrument with specific regard to unique

program


NYCResearch Design Selected a Sample of Schools based on a

set of criteria• Geographic location• Characteristics of the student population• Features of the program in school

Then invited grades 3, 4, and 5 in these schools to participate-all with parent permission surveyed.

Approximately 500 student interviews in Year 1 (fall& spring) across approx 10 schools.


NYC Research Design Issue of self-selection and comparison

groups.• All schools in NYC eligible for participation.• Factors related to whether a school participates also

likely related to program outcomes.• Limits comparability of non-program schools.

We focus only on schools with programs in this analysis.• Concentrate on variation of program effects.• Comparisons not to other schools in NYC, but to sites

where the questions we use have been fielded.


OhioResearch Design Selected a Sample of Schools based on a

set of criteria• Geographic location• Characteristics of the student population• Features of the program in school

Then invited grades 3, 4, and 5 in these schools to participate-all with parent permission surveyed.

Approximately 500 student interviews in Years 1 & 2 (fall& spring) across approx 6 schools.

Evaluating Service Learning:Research Design Pretest/Posttest Evaluation design

• Issues with pre-post test for NYC: Program in operation for over 15 years. Only true

pre-test for 3rd graders*• Importance of true pre-test. . . Columbus, Ohio

Program begins 2008 Initial survey conducted before program “kicks off” +Accomplishes true pre-test for all grades - Potential programs with program fidelity in first years

Evaluating Service Learning:

Analysis All created using P.C.F. factor analysis

Measures of Student Attitudes and Behaviors• Student engagement• Behavior in school• Voice/Capacity• Helping others (other orientation)

Factor Analysis (Kim & Mueller, 1982)

Common objective: represent group of variables in terms of a smaller number of hypothetical variables. • Exploratory: data reduction• Confirmatory: test theoretical hypotheses

Factor loadings • Standardized coefficients

“Student Engagement” Created from student responses to

six questions (italicized = scale reversed):

1. I pay attention in class.2. I complete my homework on time.3. I feel bored in school.4. My classroom is a fun place to be.5. I feel excited by the work at school. 6. I like being at school. 7. I am interested in the work at school.

Comparison of Loadings: Engagement

1 2 3 4 5 6 7

-0.2

0

0.2

0.4

0.6

0.8

1

OHw1OHw2OHw3OHw4

1 2 3 4 5 6 7

-0.2

0

0.2

0.4

0.6

0.8

1

NYw1NYw2

NYC Ohio

Behavior in School Created from student responses to

eight questions:1. I help people who are picked on.2. I share things with others.3. I work very well with other students.4. I find ways to solve problems that are fair.5. I pay attention in class.6. I complete my homework on time.7. I get in trouble at school. 8. I follow the rules at school.

Comparison of Loadings: Good Behavior

1 2 3 4 5 6 7 8

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

OHw1OHw2OHw3OHw4NYw1NYw2

Voice/ Capacity Created from student responses to

five questions:1. I am interested in doing something about the

problems in my school or neighborhood.2. If I work hard I can accomplish my goals.3. I help people who are picked on. 4. I help others with their schoolwork. 5. I talk to other students about helping our school

or neighborhood.

Voice and Capacity

1 2 3 4 5

-0.2

0

0.2

0.4

0.6

0.8

1


Helping Others Created from student responses to ten

questions:1. I think all students should learn about problems in their

neighborhood or city.2. People who have problems should only turn to their family

for help. 3. I think communities should take care of people who can’t

take care of themselves. 4. I think you should help all people, not just people you know.5. It’s better to work on a problem with a group than to work

alone. 6. I share things with others. 7. I help people who are picked on. 8. I help others with their schoolwork.9. I find ways to solve problems that are fair. 10. I cheer up people who are feeling sad.

Helping Others

1 2 3 4 5 6 7 8 9 10

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1


ActionTheory

Helping Others: Action Only

1 2 3 4 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9


Further Lessons Learned Survey design for young audiences.

• Theory vs. Action Timing and surveying children.

• School year and standardized testing constraints. New York vs. Ohio

Population testing for surveys• Different populations interpret differently.

Engagement variations vs. Behavior

Community Orientation Created from student responses to ten

questions:1. I talk to other students about helping our school or

neighborhood.2. I help others with their schoolwork. 3. I work very well with other students. 4. I help people who are picked on.5. I share things with others.6. I think you should help all people, not just people you know

well. 7. I am interested in doing something about problems in my

school or neighborhood.8. It’s important for all students to help out their school or

community. 9. It’s better to work on a problem with a group than to work

alone. 10. I think all students should learn about problems in their

neighborhood or city.

Self vs. Others Orientation Created from student responses to eight

questions:1. It’s sometimes hard for me to talk when I’m in a group

of people. 2. It’s better to work on a problem with a group than to

work alone. 3. I am interested in doing something about problems in

my school or neighborhood. 4. I share things with others. 5. I work very well with other students.6. I help others with their schoolwork.7. I talk to other students about helping out school or

neighborhood.8. I help people who are picked on.

Participation in Service Projects,NYC 2009-2010

% Participating inSchool-Based Service

Projects

% Participating inService Projects Out

of School

Overall 95.24 72.62

Male 94.29 67.35

Female 95.92 80.00

2nd Grade 100.00 84.21

3rd Grade 93.22 71.195th Grade 100.00 50.00

School Range 93.48%-100.00% 69.23%-83.33%

Participation in Service Projects,Ohio 2008-2009, 2009-2010

% Participating in

School-Based Service

Projects (2008-2009)

% Participating in

Service Projects Out of School (2008-

2009)

% Participating in

School-Based Service

Projects (2009-2010)

% Participating in

Service Projects Out of School (2009-

2010)

Overall 81.95 63.70 74.38 62.42Male 77.88 49.04 64.18 56.34Female 86.14 76.24 81.72 67.023rd Grade 66.67 64.00 56.41 48.784th Grade 89.69 69.07 68.09 60.825th Grade 83.67 58.16 85.11 70.216th Grade n/a n/a 94.44 66.67School Range

73.91-94.59 52.17-78.38 64.52-83.72 53.85-71.88

Comparing NYC and Columbus

New York City ColumbusParticipated in School-Based Service Projects

81.3% 86.6%

Participated in Service Projects Outside of School

41.9% 67.1%

Evaluating Child-Service ProgramsLIMITATIONS:

Sample size Sampling bias Age

Little understanding of how younger children are affected Longevity of study Difficult to determine true program effects Difficult to see changes in program effects

(deterioration, augmentation, etc.) Difficulty in defining a good

program/success

Date post:	24-Feb-2016
Category:	Documents
Upload:	teddy
View:	47 times
Download:	0 times

E. Christine Baker-Smith Christopher C. Weiss Vanessa G. Ohta

Documents