Cross-national measurement of school learning environments Creating indicators for evaluating UNICEF’s Child Friendly Schools

Erin Godfrey1 David Osher2

Leslie Williams1

Sharon Wolf1

Juliette Berg1

Catalina Torrente1

Elizabeth Spier2

Larry Aber1

International Society for Child Indicators

3rd International Conference28th July, 2011

The University of York

1New York University2American Institutes for Research

Overview of presentation

• Introduce importance of schooling and UNICEF’s “Child Friendly Schools” Initiative

• Describe methods for creating multi-item indicators to evaluate Child Friendly Schools

• Highlight implications for the development of cross-national indicators for use in comparative policy analysis and advocacy efforts

Schooling important aspect of child well-being

• Schooling is one experience that most children worldwide have in common and the most common means by which societies prepare their young for the future.” (UNICEF, 2009) More than a billion children worldwide are in

school • Primary mechanism for economic and

social development of nations

Schooling important aspect of child well-being

• In the past decade, significant progress has been made towards increasing school enrollment

• However, 101 million primary school age children do not attend school regularly (UNICEF, 2009)

• Poor educational quality, combined with other family and community factors constrain enrollment and attendance thwart learning and development

“Never let schooling interfere with your education.” - Mark Twain? Grant Allen?

UNICEF’s Child Friendly Schools Initiative

• Rights-based perspective rooted in CRC• Practical guide in pursuit of MDG 2• Embraces a multidimensional concept of

school quality addressing the total needs of “the child as learner”• Based in WHO’s mental health initiatives on

connectedness, caring and support• UNICEF’s interest in child-, family- and community-

centered approaches to school improvement• School effectiveness research (e.g. Battistich & Horn, 1997;

Christenson & Thurlow, 2004; Osher et al, 2007; Osher & Kendziora, 2010)

“Quality education is education that works for every child and enables all children to achieve their full potential.” (UNICEF, 2009)

UNICEF’s Child Friendly Schools Initiative

• Based on three key interrelated principles:• child-centeredness• democratic participation• inclusiveness

• Seeks to promote change and set quality standards across entire sector

• Recognizes local realities and promotes pragmatic “pathways towards quality” rather than a rigid blueprint

Global evaluation of CFS

• UNICEF supports CFS initiative in 95 countries worldwide

• First global evaluation of CFS conducted in 2008 by American Institutes for Research (AIR, 2009)

• 6 countries in 4 regions the Philippines and Thailand (East Asia and the

Pacific Region) Guyana and Nicaragua (Latin America and the

Caribbean Region) South Africa (East and Southern Africa) Nigeria (West and Central Africa Region)

Global evaluation of CFS

• Mixed-methods evaluation of 20-27 child friendly schools in each country Surveys from students, teachers and school

heads School and classroom observations Interviews/focus groups with school heads,

parents, teachers

Global evaluation of CFS

• Results provided important information about how key principles of CFS are understood and implemented in multiple contexts

• Lack of resources limited the ability to reach other goals Determine the extent to which CFS

objectives are being realized within and across countries

Create tools to monitor future progress

Global evaluation of CFS

• Relied heavily on multi-item scales• Initial analyses established internal

reliability of scales (Cronbach’s )• Advantages to factor analytic techniques

in structural equation modeling framework (Carle, Blumberg, Moore & Mbwana, 2011)

NYU Collaboration

• Deeper psychometric exploration of the validity and reliability of multi-item scales used as indicators of school quality Can a common core structure of perceptions of

school quality be identified across diverse countries,

or are country-specific measures needed to characterize these perceptions?

• First step to create cross-national tools to monitor CFS efforts

• Provide roadmap to others creating cross-national indicators

Initial focus

• Student perceptions of quality of learning environment Students’ experiences of connectedness, safety,

respect and engagement in school related to attendance, learning and drop out

Consistent with child-centered approach to indicator development (Ben-Arieh, 2008)

Not as tied to implementation differences• Three countries: South Africa, Nicaragua, the

Philippines Diverse in local context and CFS implementation

and evaluation

CFS Conditions for Learning Survey: three dimensions

• Emotionally supportive climate (15 items, = .74) How much students feel listened to, cared about, and helped

by teachers and other adults in the school

• Challenging, student-centered learning environment (14 items, = .79)

How strongly students perceive that teachers and other adults in the school encourage the active engagement of students in the learning process and the academic success of all students, and feel that what they are learning is interesting

• Safe, inclusive and respectful climate (27 items, = .83)

Two sub-dimensions: (1) the extent to which students perceive the school to be inclusive and respectful of all types of students and (2) how emotionally and physically safe students feel

Sample

• 3,072 children 807 in South Africa 1431 in the Philippines 835 in Nicaragua

• Grades 5 and 6 (roughly 9-12 years old)• 53% female

Analytic approach

• Exploratory and confirmatory factor analyses, two steps1. Determine best set of items for measuring

each domain within countries• More comprehensive information about

psychometric properties of items• Relative importance of each item for measuring

theoretical construct and substantive interpretation

Analytic approach

• Exploratory and confirmatory factor analyses, two steps1. Determine best set of items for measuring

each domain within countries2. Formally test measurement invariance to

determine whether items and scales operate the same way across countries • Particularly important in creating comparative

indicators out of scales• Otherwise, differences in mean levels of scales

or relationships between scales could be methodological artifacts, not true differences

Analytic approach

• Mplus used to handle complexities in data (Muthen & Muthen, 1998-2007)

Can treat indicators as categorical Can account for nesting of students within

schools• Items are reverse-coded to reflect

positive association with the construct

Step 1 results: Measurement models within countries

• Ran exploratory and confirmatory factor analyses separately for each dimension Emotionally-supportive climate Challenging, student-centered learning environment Safe, inclusive and respectful climate

• Each dimension had sound psychometric properties with at least a subset of items Items loaded significantly onto factor Fit statistics indicated adequate model fit (CFI and TLI ≥

.90; RMSEA ≤ .06; WRMR ≤ 1.0 (Hu & Bentler, 1999; Kline, 2005))

• Each dimension characterized by a core set of items, as well as some differences

Step 1 example:Emotionally-Supportive Climate Philippines Nicaragua South Africa17

Teachers at this school really care about students like me. 0.92 0.84 0.95

48 My teachers give me feedback on my assignments that help me to improve my work.

0.95 0.71 0.90

54Adults in this school are usually willing to give students extra help. 0.94 1.00 0.98

57Teachers notice if I am having difficulty with my lessons. 0.71 0.97 0.99

58 Teachers give students opportunities to improve their work if they do poorly on an assignment.

0.83 0.84 1.00

62 I can talk to teachers or other adults at school if I am having problems in class.

0.84 0.92 0.97

11 I can talk with at least one adult at school about things that are bothering me.

0.68 0.45 0.69

22My family knows what goes on inside this school. 0.59 0.54 0.56

60 Students at this school have the materials they need to support their learning.

0.70 0.56 0.78

37 This school does a good job teaching students what they really need to know in life.

1.00 0.41 0.98

50 This school does a good job in preparing students to continue on for more education after they graduate.

0.95 --- 0.96

47 This school does not try to help students who are behind in their work to catch up. (r)

0.56 0.91 ---

61Sometimes I am too hungry to pay attention in school. (r) --- 0.92 0.31

66 Families like mine are involved in making decisions that affect this school.

0.53 --- 0.80

69 It is difficult for students like me to get extra help from teachers. (r)

0.35 --- ---

Step 2 results: Measurement invariance across countries

• Perform formal tests of measurement invariance across countries (Gergorich, 2006; Temme, 2006) Estimate a model in which item intercepts and

loadings are allowed to vary across countries Estimate a model in which item intercepts and

loadings are constrained to be equal across countries

Evaluate change in model fit using the 2 difference statistic

• Tests whether each item measures the underlying dimension in the same way across countries

Step 2 results: Measurement invariance across countries

• For each dimension, we were able to identify a subset of items that demonstrated measurement invariance across countries

• These items measure each dimension in the same way across countries free from differential response bias due to

cultural, linguistic or other factors

Step 2 example:Emotionally-Supportive Climate Philippines Nicaragua South Africa

17 Teachers at this school really care about students like me.

1.02 1.02 1.02

22 My family knows what goes on inside this school. 0.62 0.62 0.62

37 This school does a good job teaching students what they really need to know in life.

1.01 1.01 1.01

54 Adults in this school are usually willing to give students extra help.

1.00 1.00 1.00

57 Teachers notice if I am having difficulty with my lessons.

1.08 1.08 1.08

58 Teachers give students opportunities to improve their work if they do poorly on an assignment.

1.09 1.09 1.09

11 I can talk with at least one adult at school about things that are bothering me.

0.94 0.90 0.74

48 My teachers give me feedback on my assignments that help me to improve my work.

1.28 1.78 0.97

60 Students at this school have the materials they need to support their learning.

1.01 0.57 0.84

62 I can talk to teachers or other adults at school if I am having problems in class.

1.23 1.45 1.02

Summary of results

• A common core set of items does characterize student perceptions of school quality across three diverse countries/regions

• A subset of these items demonstrated measurement invariance These can be used to create indicators of

school quality that can be compared across countries

Insight into substantive interpretation of each dimension

• Challenging, student-centered learning environment How much students perceive that teachers and

other adults in the school encourage the active engagement of students in the learning process and the academic success of all students, and feel that what they are learning is interesting

Core set of items does capture all of these features

Insight into substantive interpretation of each dimension

• Challenging, student-centered learning environment

• Emotionally supportive climate how much students feel listened to, cared about,

and helped by teachers and other adults in the school

Core set of items assess how much students feel cared about and helped, but not how much they feel listened to

Insight into substantive interpretation of each dimension

• Challenging, student-centered learning environment

• Emotionally supportive climate• Safe, inclusive and respectful climate

Two sub-dimensions: (1) the extent to which students perceive the school to be inclusive and respectful of all types of students and (2) how emotionally and physically safe students feel

Analyses revealed two different dimensions (1) general feelings of safety, inclusion, fairness and respect and (2) harsh treatment in school

Insight into substantive interpretation of each dimension

• Challenging, student-centered learning environment

• Emotionally supportive climate• Safe, inclusive and respectful climate• Limit interpretation of scales to these

aspects, consider adding additional items

Important cross-country differences

• Able to identify core similarities in the measurement of each dimension

• Also found notable differences in the structure and substantive interpretation of each dimension within countries

• Reflected each country’s specific cultural, linguistic, and sociopolitical context, as well as differences in CFS implementation and evaluation

Implications for CFS

• Create two sets of monitoring tools The core set of invariant items can be used to

create tools for cross-national comparisons This set can be augmented with additional country-

specific items that measure unique aspects of school quality

• Analyses based on only three countries Further work will extend analyses to all six countries

in global evaluation, others where these items are used

• Focus on reliability of scales Further work to explore concurrent and predictive

validity and sensitivity to change

Implications for indicator development

• Process holds promise for creating reliable and meaningful monitoring tools for international policies and programs with varying conditions for implementation and evaluation

• Provide a roadmap for future evaluation efforts

• Foster a new more stringent concern with psychometric properties of multi-item indicators