Girls Earthquake Science and Safety Initiative Evaluation Framework

7/30/2019 Girls Earthquake Science and Safety Initiative Evaluation Framework

http://slidepdf.com/reader/full/girls-earthquake-science-and-safety-initiative-evaluation-framework 1/75

Prepared by:

Ashley Cheung, Jana Kemp,Jessica Mann, Simon McNorton

The George Washington University

April 23, 2013

Client Organization: Teachers Without Borders

Project Liaison: Dr. Fred Mednick, Founder

Visiting Fellow/Lecturer, John Hopkins

University, School of Education

[email protected]

A Framework for Evaluating the Effect of

Earthquake Science Education

ShareAlike



A Framework for Evaluating the Effect of Earthquake Science Education

Executive Summary

Teachers Without Borders (TWB) provides resources, tools, and training to enhance the

knowledge, skills, and connectivity of teachers around the world. TWB is preparing to launch theGlobal Quake Science and Safety Initiative for Girls (GQSSI) which seeks to equip over 100,000

girls in earthquake vulnerable zones with the skills and knowledge they need to prepare their

communities for potential disasters. Collaborating with TWB, the authors of this report, Ashley

Cheung, Jana Kemp, Jessica Mann, and Simon McNorton, have developed the following

research question:

Does earthquake science education have an effect on disaster preparedness and

community resilience? If so, how can it be measured?

To address this research question, we have produced an evaluation framework including metrics

and indicators that can be used to assess the preparedness and resilience of communities.

Our initial research examined the context of disaster risk reduction globally and the role of

disaster education programs. To inform our construction of indicators and metrics, we conducted

an extensive review of current background literature and existing studies in two areas: 1) the

categorization of indicators used in disaster preparedness and community resilience

internationally; and 2) the evaluations of current disaster education programs. We found that

although disaster preparedness and resilience are most often measured at a national or macro-

level, there are a range of ways to observe changes at a local and household level. Furthermore,

evaluations of disaster education programs are uncommon and often lack rigorous design. Those

which do exist measure knowledge-based outcomes and do not reflect attitudinal or behavioral

impacts of disaster education.

Our analysis of existing programs and research led us to develop ten criteria for indicators that

can be used in local community evaluations of the GQSSI. The criteria also reflect the

knowledge and behavioral change objectives of the GQSSI curriculum. Reflecting these criteria,

the proposed evaluation framework identifies four key indicator categories that together make up

a composite measure of student knowledge gain and disaster preparedness and community




resilience: 1) earthquake science and safety knowledge; 2) risk perception; 3) structural and non-

structural safety; and 4) emergency planning and preparedness.

We have provided an evaluation framework in order to help researchers measure theseindicators. Acknowledging site-specific resource and time constraints, researchers may face three

key decisions in planning an evaluation: 1) research design; 2) data collection sources; and 3)

data collection methods. We designed tools that allow researchers to consider a combination of

research design elements and data collection methods to evaluate the program. We have provided

a guided toolkit, rather than a strict plan based on “promising practices”, because we recognize

that it may not be feasible for researchers to operate under the conditions that such a prescriptive

plan may require.

We have developed examples of evaluation instruments that researchers can use or modify when

measuring the impact of the GQSSI. We have provided student and teacher questionnaires which

aim to measure changes in the chosen indicators. In addition, we have offered options for shorter

surveys or focus groups which can be administered to parents, principals and local officials to

evaluate broad impacts of the GQSSI program on the community.

The extensive research and analysis we have provided and the tools we have created together

form an evaluation framework that TWB researchers can use to evaluate the impact of the

GQSSI program on disaster preparedness and community resilience.




Table of Contents

Executive Summary ....................................................................................................................................... i

List of Acronyms ......................................................................................................................................... iv

1. Introduction ........................................................................................................................................... 1

1.1. About Teachers Without Borders ................................................................................................. 1

1.2. About the Authors of this Report .................................................................................................. 2

1.3. About the Global Quake Science and Safety Initiative for Girls .................................................. 2

1.4. Research Question......................................................................................................................... 4

2. Background ........................................................................................................................................... 4

2.1. Disaster Risk Reduction as a Policy Priority ................................................................................ 4

2.2. What is Disaster Preparedness and Community Resilience? ........................................................ 5

2.3. The Role of Education in Disaster Risk Reduction ...................................................................... 6

2.4. What can be learned from Disaster Education Programs around the World? .............................. 7

2.5. Existing Evaluations of Disaster Risk Reduction Programs ......................................................... 9

2.6. Addressing the Lack of Evaluation ............................................................................................. 10

3. Existing Practices ................................................................................................................................ 11

3.1. Measuring Disaster Preparedness and Community Resilience ................................................... 12

3.2. Promising Practices in Evaluating Disaster Education Programs ............................................... 14

3.3. The Importance of Cultural Competence .................................................................................... 16

4. Developing Indicators and Metrics ..................................................................................................... 17

5. The GQSSI Evaluation Framework .................................................................................................... 20

5.1. Choosing a Research Design....................................................................................................... 20

5.2. A Discussion of Data Collection Methods .................................................................................. 226. Sample Instruments ............................................................................................................................. 26

6.1. Student and Teacher Questionnaires ........................................................................................... 26

6.2. Community Stakeholder Tools ................................................................................................... 27

7. Conclusion .......................................................................................................................................... 29

8. References ........................................................................................................................................... 31

Appendices .................................................................................................................................................. 34

Appendix A. Client Contact Information ................................................................................................ 34

Appendix B. Logic Model ...................................................................................................................... 35

Appendix C. Global Quake Science and Safety Initiative Curriculum ................................................... 36Appendix D. Data Collection Criteria Matrix ......................................................................................... 38

Appendix E. Sample Student Questionnaires ......................................................................................... 40

Appendix F. Sample Teacher Questionnaires ......................................................................................... 52

Appendix G. Questionnaire Response Coding Tool ............................................................................... 62

Appendix H. Community Survey ............................................................................................................ 70




List of Acronyms

4R – Robustness, Redundancy, Response, and Recovery

ADPC – Asian Disaster Preparedness Center AUDMP - Asian Urban Disaster Mitigation Program

EERI – Earthquake Engineering Research Institute

EFP – Equal Futures Partnership

ENDP – Education for Natural Disaster Preparedness

GEM – Global Earthquake Model

GQSSI – Global Quake Science and Safety Initiative

GWU – The George Washington University

STEM – Science, Technology, Engineering, and Math

TOSE – Technical, Organizational, Social, and Economic

TWB – Teachers Without Borders

UNDP – United Nations Development Program

UNESCO – United Nations Educational, Scientific and Cultural Organization

UNICEF – United Nations Children’s Fund

USAID – United Nations Agency for International Development

USGS – United States Geological Survey




1. Introduction

In the new millennium, the impact of earthquakes has been increasingly and indiscriminately

destructive. Rapid population growth, changing socio-economic conditions, and dense andunplanned urbanization mean that larger numbers of people than at any time in history are at risk

of experiencing an earthquake. Since 2000, almost one million people have lost their lives in

earthquake-related disasters (USGS, 2013). Two single events, the 2004 Indian Ocean

earthquake and the 2010 Haiti earthquake, are alone responsible for over half of those deaths.

The victims of these disasters are located in underdeveloped regions of the world that are poorly

equipped and unable to deal with the impact of earthquakes or the devastation that follows.

The importance of disaster risk reduction is now an international priority. Although

technological breakthroughs have allowed scientists to monitor seismic activity, earthquakes still

cannot be predicted. Organizations like Teachers Without Borders (TWB) have paired emerging

technology and earthquake education together to mitigate earthquake damage and create more

resilient communities. This report highlights how TWB is using education to build resilience at

the community level, and provides a recommended framework to evaluate the impact of the

organization’s initiatives.

1.1. About Teachers Without Borders

Teachers Without Borders (TWB) is an international not-for-profit organization based in Seattle,

Washington. TWB was founded by Dr. Fred Mednick who also serves as the primary liaison for

this project. TWB provides resources, tools, and training to enhance the knowledge, skills, and

connectivity of teachers around the world. The organization was founded in 2000 and now

operates six programs, serving over 30,000 online members and tens of thousands of offline

members in over 180 countries.

Through their Emergency Education Program, TWB is committed to working with teachers in

regions of the world vulnerable to natural and national disasters. Over time, the organization has

focused its efforts almost exclusively on education for teachers and students in seismically




vulnerable regions of the world. Launched in response to the 2008 Sichuan earthquake in China,

the primary focus of the program is to ensure that students and teachers are able to understand

the nature of earthquakes and prepare accordingly. TWB’s earthquake science and safety

program has also been adapted for, and tested in, Afghanistan, Haiti, India, and Pakistan. Their tagline for the program is: “Education from below the ground and up.”

1.2. About the Authors of this Report

The authors of this report are four graduate students in the Masters in Public Policy program at

the Trachtenberg School of Public Policy and Public Administration, part of The George

Washington University. The graduate students, Ashley Cheung, Jana Kemp, Jessica Mann, and

Simon McNorton, have produced this research as a capstone requirement of the degree program,

under the supervision of Professor Elizabeth Rigby and Research Advisor, Patrick Besha.

1.3. About the Global Quake Science and Safety Initiative for Girls

In 2012, the Senior Policy Advisor to the Undersecretary of Education solicited ideas to support

a White House initiative to support science, technology, engineering, and mathematics (STEM)

education for girls worldwide. TWB responded with a description of their existing disaster

education program. The White House encouraged TWB to collaborate with the United States

Geological Survey (USGS), Global Earthquake Model (GEM), and the Earthquake Engineering

Research Institute (EERI).

The collaboration issued a grant proposal and report, “The Global Quake Science and Safety

Initiative for Girls” (GQSSI). The project scope strengthens and leverages TWB’s pilots in

earthquake science and safety education by including regional engineering expertise, equipment,earthquake modeling, data analysis, and a process for measuring seismic activity in local schools

and frequently-visited community buildings.

As designed, the project intends to reach 100,000 students (with a special focus on girls’

inclusion) and teachers over three years, in regions of the world identified for their deadly




combination of dense population, poorly constructed buildings, and seismic vulnerability. A

logic model of the GQSSI program is provided in Appendix B. A logic model is a visual

depiction of the programs’ inputs, activities, outputs and outcomes.1 The graphic succinctly

summarizes the theory, underlying assumptions, and expected achievements of the programthroughout its lifetime.

The GQSSI curriculum builds on an existing framework created by Solmaz Mohadjer, TWB’s

Director of Emergency Education. Originally implemented in Tajikistan, the curriculum is

composed of 12 core inquiry-based science lessons, each of which provides students information

about the science and effect of earthquakes (Nature Education, 2013). The initial sessions focus

on why and how earthquakes happen, as students learn about plate tectonics and the stress that

occurs at their boundaries. Students then learn about how energy is released in the form of

seismic waves, and about structural and environmental hazards associated with movement at

geographic fault lines. A brief description of individual GQSSI lessons can be found in

Appendix C.

Toward the end of the curriculum, teachers introduce lessons on preparedness and mitigation

techniques. Students become knowledgeable about the potential structural and non-structural

hazards within their schools and homes. Students learn the importance of planning and practice,

and have the opportunity to create and present real emergency plans. As a culmination of the

science and preparedness modules, each student completes the curriculum by writing a story

about the effects of an earthquake, using their new technical knowledge and preparedness

techniques to demonstrate their understanding of the science and mitigation of earthquakes.

The content of each lesson is enhanced through hands-on activities and simple models to enable

learning through participation as well as lecture. Through its hands-on and student ownershipapproach, the applied nature of the GQSSI curriculum and its experiential learning model is an

ideal way to teach disaster awareness and risk-reduction.

1 Inputs are defined as tangible resources needed to foster program development and program activity. Activities are

the processes by which a program utilizes its inputs; they are the actual programmatic efforts. Outputs are the

measurable result of activities; they are the services and products that the staff members actually provide. Outcomes

are the changes in skill level, knowledge, behavior, etc. as a result of program activities (W.K. Kellogg, 2004, p. 2).




1.4. Research Question

To date, TWB has used pre- and post-tests to assess student educational outcomes as a result of

pilot disaster education programs. TWB has acknowledged the need for more quantitative

evaluation tools to understand the impact of disaster education programs on community

preparedness and resilience (see also Section 2.2) To that end, the authors of this report, herein

referred to as ‘we’, have assisted TWB in developing indicators and metrics that can be used to

quantify the impact of the GQSSI program in furthering disaster preparedness and community

resilience for earthquakes. To meet this need, we responded to the following research question:

Does earthquake science education have an effect on disaster preparedness and

community resilience? If so, how can it be measured?

To address this research question, we have produced an evaluation framework including metrics

and indicators that can be used to assess the preparedness and resilience of communities.

2. Background

2.1. Disaster Risk Reduction as a Policy Priority

The impact of disasters can vary widely across populations depending on socio-economic and

demographic differences. A region with higher levels of social and economic development and

with a modern physical and institutional infrastructure will likely be better prepared and more

resilient to disasters when they happen. When assessing disaster preparedness and resilience

across countries and regions, researchers rely heavily on these social and economic indicators to

determine which regions are more vulnerable to disasters. These may include mobile phone

penetration, car ownership, household size, per capita income, and education level.

Unfortunately, the overwhelming majority of casualties have taken place in communities that

were not prepared to face, or ill-equipped to respond to an earthquake.

Facing this challenge, governments, international organizations, and researchers have committed

to reducing vulnerability to earthquakes through disaster risk reduction (United Nations, 2004, p.




7).2 Disaster risk reduction relates to the reduction of risks associated with the onset of disasters.

The 2005 United Nations International Strategy for Disaster Reduction (herein the “Hyogo

Framework”) established five priorities for disaster risk reduction: a) ensuring that disaster risk

reduction is a national and a local priority; b) assessing and monitoring disaster risks and earlywarning systems; c) using knowledge, innovation, and education to build community resilience;

d) reducing the underlying risk factors, and e) building the capacity of communities to respond to

disasters.

The Hyogo Framework is the first internationally accepted framework for disaster risk reduction

and is the global blueprint for disaster risk reduction efforts through to 2015 (United Nations,

2007, p. 9). The Hyogo Framework was signed by 168 governments to acknowledge and plan for

education as a means of keeping children safe, and communities informed of what to do in case

of a disaster. It describes in detail what must be done across regions, communities, and

governments to reduce disaster loss.

Recognizing the powerful role of education in disaster risk management, the Hyogo Framework

has created a global dialogue for disaster preparedness programs. Some of these initiatives

involve developing institutions at a national level, while others involve increasing the capacity of

response teams so that they are better equipped to deal with disasters. Elsewhere, local awareness

campaigns and education programs help build resilience at a localized, community level. The

GQSSI program is one of these disaster education programs. This paper is concerned with the

evaluation of this program by determining its impact on disaster preparedness and community

resilience.

2.2. What is Disaster Preparedness and Community Resilience?

Preparedness is concerned with ensuring the readiness of societies by taking precautionary

measures. Within the context of disaster preparedness, this may include any pre-disaster,

protective, or preventative actions that can improve the safety of communities and the

2Vulnerability is defined as: “The conditions determined by physical, social, economic, and environmental factors

or processes, which increase the susceptibility of a community to the impact of hazards”.




effectiveness of disaster response (Şakiroglu, 2005, p. 12; Edwards, 1993, p. 293). Resilience has

been variously defined, and useful cross-examinations of different definitions in the disaster

preparedness context have been given (Burton, 2012, p. 8). The Hyogo Framework stresses the

following definition;

The capacity of a system, community, or society potentially exposed to hazards to adapt,

by resisting or changing in order to reach and maintain an acceptable level of functioning

and structure. This is determined by the degree to which a social system is capable of

organizing itself to increase this capacity for learning from past disasters for better future

protection and to improve risk reduction measures (United Nations, 2007, p. 4).

In practical application, attempts to improve preparedness and resilience involve changes to the physical environment and emergency planning and preparation in advance of disasters.

Furthermore, a capacity to learn from disasters and an increased awareness, or perception, of

disasters can be crucial in improving the ability of a system to protect itself.

2.3. The Role of Education in Disaster Risk Reduction

The need for education in disaster risk reduction efforts, made a priority by the Hyogo

Framework, has risen on the agenda of prominent international organizations over the last

decade. The United Nations Educational, Scientific, and Cultural Organization (UNESCO)

promotes education for disaster risk reduction and partnered with United Nations Children’s

Fund (UNICEF) in 2011 to create the joint Mapping of Global Disaster Risk Reduction

Integration into Education Curricula Initiative. The overall goal of the Initiative is to actively

integrate disaster risk reduction into school curricula worldwide by 2015 (Selby & Kagawa,

2012, p. 10).

Following this initiative, the two organizations produced Disaster Risk Reduction in School

Curricula: Case Studies from Thirty Countries in 2012. Highlighting thirty case studies, this

meta-analysis investigated interdisciplinary and cross-disciplinary examples of disaster risk

reduction demonstrated in curriculum, school culture, learning outcomes, and overall awareness.




The meta-analysis produced a comprehensive mapping that captures national experiences and

promising practices in regards to the integration of disaster risk reduction in school curriculum

(Selby and Kagawa, 2012, p. 10). UNESCO believes that mainstreaming disaster education

programs will raise awareness of disaster management among children, teachers andcommunities.

There are many models that support the theory that education can promote action and foster

change within communities. Education is positively associated with civic and social engagement

by increasing human capital and social mobility (Bartko, Eccles, and Stone, 2003, p. 234).3

Student engagement can be broken down into three components; affective-emotional, cognitive,

and behavioral engagement (Appleton, Christenson, and Furlong, 2008, p. 370).4

Although all

three components of engagement are important, we focus on how the earthquake science

program can foster student behavioral engagement within the community. Behavioral

engagement reflects student participation in activities. The objective of disaster education is to

increase behavioral engagement through teaching about how and why disasters occur and

informing students of the positive impact they can have on their communities.

2.4. What can be learned from Disaster Education Programs around the World?

The inherent link between schools and the wider community makes them ideal institutions

through which to foster a culture of disaster preparedness. Children are among the most

vulnerable populations, and the schools in which they learn are often weakly constructed.

Schools are anchors for community development, and can serve as refuge sites following a

natural disaster – if they are undamaged.

3 Generally, studies have examined relationships between student development outcomes and active participation in

programs, government and other clubs. Campbell (2006, p. 25) makes the argument that education is widely

recognized as having a strong correlation with multiple forms of civic and social engagement. Another study reveals

a positive relationship between education and forms of civic and social engagement (Milligan, Moretti, &Oreopoulus, 2003, p. 3).4 Affective-emotional engagement describes students’ social, emotional and psychological attachments to school and

can range of signals from academic enjoyment and pursuits of higher levels of knowledge to levels of interest and

happiness (Bohnert, Fredricks, & Randall, 2010, p. 576). Cognitive engagement examines how students demonstrate

deeper learning a mastery of a topic (Corno, 1993, p. 14).




Formal disaster education, paired with generational knowledge from previous disasters, supports

the dissemination of information to youth. Children then become change agents, transferring this

information to the wider community.

Many disaster education programs focus on a specific type of disaster to combat risks that are

most prevalent within the region. Earthquake preparedness is often regarded as a separate set of

activities unrelated to academia, rather than a complementary and critical component of existing

science coursework. The trend appears to be that such a curriculum is introduced after a natural

disaster, and thus the initiation of earthquake education and preparedness is treated as highly

reactive rather than proactive. The GQSSI is distinguished by its connection between science and

prevention and planning for seismically active regions, whether or not they have experienced a

disaster.

While earthquake science and safety education is a young field, examples of its integration into

schools do exist around the globe. In Iran, the International Institute of Earthquake Science and

Seismology was established in 1989 and the Manjil earthquake of 1990 spurred the creation of a

nationwide public education program in 1991. Integrated into the school system at all levels (K-

12), the program combines continued education through textbook materials, films, and posters;

an annual drill in all public schools; and a public exhibition to raise awareness at the community

level (Parsizadeh and Ghafory-Ashtiany, 2010, p. 33).

The Asian Urban Disaster Mitigation Program (AUDMP) of the Asian Disaster Preparedness

Center (ADPC) began in 1995 upon recognition of the increased disaster vulnerability of urban

populations in many of Asia’s densest cities (Asian Urban Mitigation Program, 2012). Nepal and

Indonesia, among other countries, have established school earthquake safety programs aimed ateducating and institutionalizing earthquake safety precautions, including the assessment and

renovation of school buildings (Asian Disaster Preparedness Center, 2004, p. 3; Asian Disaster

Preparedness Center, 2004, p. 1).




Soon after the Bhuj earthquake of 2001, the Government of India initiated the School Earthquake

Laboratory Program. The program was launched in the northeastern and northwestern Himalayan

regions, which are seismically active parts of India. The government set up earthquake

laboratories in 100 schools within those regions, designed to produce data for the Wadia Instituteof Himalayan Geology and simultaneously engage students in learning about earthquakes

(Wadia Institute, 2009). The program involves teaching earthquake science; providing students

an opportunity for participatory learning (in which they are involved in the acquisition of

seismological data); and “providing a platform for interaction among local bodies in disaster risk

reduction” in hopes of creating a culture of preparedness (Bansal and Verma, 2012, p. 451).

Understanding how other programs are designed and implemented will allow us to compare the

GQSSI within the greater context of earthquake education. A review of other programs’ goals,

processes, and achievements has allowed us to understand existing methods of evaluation, and

has provided support for the development of feasible and sound metrics.

2.5. Existing Evaluations of Disaster Risk Reduction Programs

Most research to-date on education about natural disasters has focused on awareness programs

that have taken place after the occurrence of an event, rather than in preparedness and planning

to mitigate against disasters. Research that examines education programs prior to the event of a

natural disaster is limited. Recent studies have begun to address the role of children in disaster

preparedness and community resilience by viewing education programs as part of overall

community capacity building. However, the majority of the studies have been cross-sectional and

correlational. It remains difficult to make causal references about the effectiveness of education

programs and their impact on community preparedness and resilience (Ronan, Crellin, and

Johnston, 2012, p. 1412).

Limited studies over the last decade have attempted to use a quasi-experimental research

methodology to examine the benefits of disaster education programs for youth in helping to

increase community resilience. The aforementioned UNESCO and UNICEF meta-analysis of

case studies from 30 countries found that few evaluations of these education programs existed.




Those which do exist have featured a heavy predominance of knowledge-based outcomes rather

than applied learning outcomes and do not reflect the community engagement and change

agency ambitions of disaster education (Selby and Kagawa, 2012, p. 9).

2.6. Addressing the Lack of Evaluation

The GQSSI is a promising program in the growing field of disaster education. TWB recognizes

the importance of utilizing rigorous evaluation techniques to demonstrate the benefits of these

types of programs for students and the community. Our evaluation framework is built on existing

work conducted by TWB, including pre- and post-assessments at pilot sites. Because the

earthquake science curriculum covers new ground in the field of earthquake science and safety,

these assessments have attempted to strengthen program value through: a) ensuring that the

curriculum has been validated by credible scientific experts; b) assessing prior knowledge; and c)

engaging credible local leadership and authorities so that TWB is not viewed as an outside party

with ulterior motives like policing contractors, exposing the government, or publicizing human

rights violations. TWB proposed that we assist the expansion of their existing monitoring and

evaluation criteria to include the kinds of measurements that such pilot projects have not been

able yet to handle, given capacity issues.

Evaluations of the GQSSI and other disaster education programs face some unique challenges.

Disaster situations such as earthquakes are somewhat random, unpredictable, and difficult to

replicate in a test environment, which poses difficulties for evaluation. The complexity increases

for the GQSSI because it will be implemented in several countries with unique characteristics –

language spoken, time zone, and a wide range of cultural norms. This variability presents

difficulties in creating consistent metrics applied uniformly across all regions and sites. Though

it is important to keep these challenges in mind, it is still possible to design a rigorous, evidence- based evaluation of such programs.




3. Existing Practices

To address the question of whether or not the GQSSI has an impact on community preparedness

and resilience, we sought to develop guidelines to assist researchers at the GQSSI sites inconducting evaluations of program outcomes.

5This included identifying key program outcome

indicators and an array of metrics that researchers can use for measurement. It also included

providing an outline of research design options and data collection methods for researchers to

consider, based on the needs and limitations of individual sites.

Designing an evaluation framework which can be used to assess the impact of the GQSSI

required identifying indicators that are appropriate and measurable. To inform our construction

of indicators and metrics we first conducted an extensive review of current background literature

and existing studies in two areas – the categorization of indicators used in disaster preparedness

and community resilience internationally, and the evaluations used in current disaster education

programs. Additionally, we explored the importance of cultural competence in data collection to

ensure that our guidance for GQSSI evaluations is sensitive and respectful of different

populations and their cultural norms.

Obtaining an understanding of current methodology, including indicator categorization, common

measures, and appropriate evaluation tools provided guidance for the design of an evaluation

framework and metrics which fit the needs of the GQSSI. A review of tested, promising

practices also helped distinguish which approaches can yield valid results as well as assisted in

identifying and addressing possible limitations to the various combinations of research design,

data collection methods, and tools available for an evaluation of the GQSSI.

5The word ‘researchers’ is used throughout this report as a collective term for staff conducting evaluations of the

GQSSI. This may differ depending on site location and resources. Researchers may include: TWB or GQSSI

program staff; third-party data collection personnel; or teachers.




3.1. Measuring Disaster Preparedness and Community Resilience

The focus of the GQSSI on attitudinal and behavioral change at the household and individual

level means that broad macro-level indicators used in national assessments of disaster preparedness and community resilience require refinement. However, many smaller assessments

and evaluations used micro-level indicators to better capture changes in disaster preparedness

and community resilience at the individual, household, and local community level.

In the U.S., research into disaster preparedness and resilience examines: a) pre-disaster hazard

vulnerability analysis and mitigation; and b) post-disaster emergency response and recovery.

Within the earthquake context, other models have categorized resilience into the 4R’s

(robustness, redundancy, resourcefulness, and rapidity) that capture the temporal phases of

resilience. In order, these reflect the integrity of physical and institutional structures to withstand

a disaster, followed by failsafe or redundancy measures in case structures or systems fail, and

finally a response and recovery phase that is made easier with increased resources (Bruneau et al,

2003, p. 737).6

A second model, TOSE (technical, organizational, societal, and economic),

captures categories of preparedness across different entities that exist in any society (Bruneau et

al. 2003, p. 738).7

As an example, a community may have excellent structural integrity

(technical) and high levels of socio-economic development, but may lack the proper institutions

(organizational) to manage disaster response efforts. The 4R and TOSE dimensions form useful

frameworks that help to conceptualize preparedness and resilience across macro-level units of

analysis such as in a national disaster preparedness strategy.

As mentioned, researchers have developed micro-level indicators to capture changes in disaster

preparedness and community resilience at the household and local level (although within the 4R

and TOSE frameworks). The Mulilis-Lippa-Duval emergency preparedness scale, for example,6 4R’s are further defined as: robustness, the ability to survive the impact of an earthquake; redundancy, the

availability of second or third-tier backup systems in case of failure; resourcefulness, the capacity of systems and

human capacity to mobilize resources in case of a disaster; and rapidity, the ability to respond in a timely manner sothat other systems are not jeopardized.7 For TOSE, the technical dimension refers to the robustness of physical entities; the organizational dimension refers

to the capacity of organizations to continue operating during and after an earthquake; the social dimension reflects

negative consequences within communities and households as a result of earthquakes; and is not dissimilar to the

economic dimension, which refers to the capacity to reduce economic losses resulting from earthquakes.




includes an excellent and oft-cited number of micro-level indicators (Mulilis, Duval, and Lippa,

1990, p. 357). Because this scale was created for use in developed countries, some of these

measures may not be applicable to the most vulnerable regions of the world. However, other

scales have adapted this approach to better reflect regional socio-economic, technological, or cultural differences (Sakiroglu, 2005, p. 12).

Individual indicators used in the Mulilis-Lippa-

Duval emergency preparedness scale include:

1. Possession of a flashlight, radio, first-aid kit, stocks of food and water.

2. Knowing the location of shut off valves for water, gas and electric power.

3. Fastening large furniture, windows, and doors securely.

4. Knowing information about what to do in case of an earthquake, including attending

meetings for the purpose of establishing earthquake preparedness.

5. Having earthquake insurance.

Further examples of micro-level indicators in households and communities can be found in

disaster education research by Faupel, Kelly, and Petee (1992). Micro-level indicators are often

categorized into groups that can be roughly assigned as: a) material; b) planning; and c)

knowledge-level measurements (Sakiroglu, 2005, p. 12).8

The first category, material indicators,

includes access to water storage, non-perishable food, and the securing of items like bookshelves

and windows. The second category captures planning indicators which include participation in

workshops, family and community evacuation plans, the existence of emergency supplies or an

emergency kit, and knowledge of safe locations. The third category, knowledge and risk

perception, includes disaster education, understanding of disasters, and previous experience of

disasters. Together, the three areas capture roughly the breadth of micro-indicators unrelated to

collective demographic or socio-economic proxy metrics (per capita income, household size,

etc.).

8The categorization into material, planning, and knowledge/risk perception was done independently but to a similar

conclusion as that of Mehmet Şakiroglu. Some national level and wider demographic and socio-economic indicators

and indices offer a more comprehensive and macro-related categorization (Simpson, 2006, p. 13).




3.2. Promising Practices in Evaluating Disaster Education Programs

In order to design effective and accurate measures for the GQSSI, current promising practices in

the evaluation of international disaster education programs should be reviewed. As discussed

earlier in this report, few evaluations of disaster education programs which use an experimental

design exist. We have drawn our review from those studies which have utilized these techniques,

as well as from an analysis of what would have strengthened other evaluations which used less

rigorous designs.

Though it has not typically been employed to evaluate disaster education programs, the literature

indicates the importance of using an experimental research design rather than merely cross-

sectional or correlational methods.9 Experimental designs can allow for the analysis of the causal

relationship of factors. Researchers manipulate variables and are able to analyze the impact of

the program (treatment/independent variable) on student awareness and disaster preparedness

and community resilience (dependent variables).

In recent years, evaluations of disaster education programs have introduced a quasi-experimental

design, allowing for some analysis of the causal relationship of factors. As discussed earlier, a

disaster education program cannot be tested with a true experimental design because disaster situations such as earthquakes are unpredictable. Researchers cannot simulate an earthquake and

conduct a randomized controlled trial to find causality – so researchers are limited to using

quasi-experimental approaches. Existing studies have used some form of comparative strategy

and have enlisted pre- and post-testing as a tool for evaluating programs. In an evaluation of a

disaster education program in New Zealand, Ronan, Crellin, and Johnson (2012) conducted a

one-group pre- and post-test of students receiving disaster education. The researchers used

previous results to ‘‘benchmark’’ current findings on students in an attempt to raise confidence

that changes were a result of the education program itself and not some extraneous factors.

9 Cross-sectional studies are merely descriptive, and though a lot of information can be drawn from these about how

studies are implemented and what is happening with students and communities, they are not causal or relational.

While correlational studies can suggest that there is a relationship between two variables, but cannot prove that one

variable causes a change in another variable (correlation does not equal causation).




In Japan, Shaw and Shiwaku (2008) compared learning outcomes of students in 12 schools,

specifically comparing other schools to Maiko High School, which has an environment and

disaster mitigation program. Another study by Ronan and Johnson (2003) used a control-

comparison pre- and post-test design, comparing a classroom of students receiving disaster education to a classroom receiving only a reading and discussion course (which the treatment

group was also receiving). This study appears to be the most experimental evaluation of a

disaster education program to-date.

Research indicates that evaluation of disaster education programs can be strengthened by the use

of multiple evaluation tools which address several units of analysis. The UNESCO and UNICEF

meta-analysis found that self-assessment, peer assessment and portfolio assessment evaluation

methods tend to remain mostly aspirational with relatively few examples of their concrete

implementation (Selby and Kagawa, 2012, p. 9). UNESCO and UNICEF identified the need for

evaluations to utilize a portfolio assessment in which different kinds of data are compiled based

on the performance of each student. Such an evaluation would use of a range of assessment tools

that seek to identify what knowledge students have acquired, what skills they have developed

and the degree to which their attitudes and behaviors shift through participation in disaster

education programs. Rather than the use of a portfolio assessment, the previously discussed

studies primarily used only student reported questionnaires, though the Ronan and Johnson

(2003) study had an additional questionnaire for parents.

While evaluations of disaster education programs vary in design, researchers commonly employ

measures of disaster knowledge, preparation, and resilience indicators which are similar to those

used in the emergency preparedness scales mentioned earlier. These include students’ disaster

awareness and perception of risks; factors such as awareness of the likelihood of natural

disasters, effects of disasters, and chance of injury. Two studies (Ronan and Johnson, 2003, p.1012; Ronan, Crellin, and Johnson, 2012, p. 1415) looked to assess students’ knowledge of

preparedness and risk mitigation behaviors. The inclusion of possible behaviors was based on

issues that are often highlighted in disaster education programs and emergency management

recommendations. Each study also looked at the hazard adjustments employed by students’

families – factors linked to readiness for a disaster, awareness of the warning systems, and




behaviors that are recommended to reduce the risk of accidents and injuries. The Shaw and

Shiwaku (2008) study also looked at students’ behaviors related to family and community

dialogue about disasters (both the students’ intent to discuss and the frequency of actual

discussions).

3.3. The Importance of Cultural Competence

While data collection methods vary by discipline and situation, the emphasis on ensuring

accurate data collection remains essential to maintaining the integrity of the evaluation design. It

is important to preserve reliable, comparable results throughout the whole data collection period.

Collecting information can be difficult due to language, geographic, and cultural differences

across multiple sites.

Awareness of the defining social and cultural characteristics of the populations within which

researchers are collecting data is referred to as cultural competence (O’Brien et al., 2006, p. 675).

Cultural competence moves beyond sensitivity and knowledge to incorporate the broad scope of

ethical research design, conduct and interpretation. It is critical for researchers to ensure effective

communication and interaction with participants, adequate analysis and interpretation of results

as they relate to the population, and appropriate engagement in study design and implementation.

According to the American Evaluation Association’s Guiding Principles (2011), data collection

must be done in a cultural context “to ensure recognition, accurate interpretation, and respect for

diversity.” The goal is to achieve an optimal cross cultural data set, maintaining validity and

comparable results, while remaining culturally sensitive to the population being studied. Due to

the differences across sites, it is difficult to collect data using the same techniques. For example,

in areas that are not as developed, the use of online surveys may not produce the same results as

if the survey questions were given orally. TWB has traditionally enlisted highly qualified, local

evaluators, well versed in face-to-face surveys. As TWB prepares to scale the program to

100,000 girls in multiple regions, the organization should solicit expertise on how to maintain the

accuracy of data collected from a cultural context.




4. Developing Indicators and Metrics

Our analysis of existing programs and research led us to develop ten criteria for indicators that

can be used for local community evaluations of the GQSSI. Additionally, these criteria alsoreflect the knowledge and behavioral change objectives of the GQSSI curriculum. The criteria

are as follows:

1. The indicators should reflect changes that can be expected from program

participation.

2. The indicators must be measureable.

3. The indicators should be associated with changes within the participant pool and

secondary impacts within the local community, for example, within families of

participants, who may also take additional preparedness measures.

4. The indicators should capture both physical and structural adjustments and

modifications, in addition to attitudinal and behavioral changes, that increase

preparedness.

5. The indicators should capture changes in awareness of disasters and their impacts in

addition to perceptions of potential hazards that result from earthquakes.

6. The indicators should include a learning assessment to measure changes in student

and teacher knowledge of earthquake science.

7. The indicators should reflect the capacity of program participants to use knowledge

gained to take additional preparedness measures not directly instigated by the

curriculum.

8. The indicators should reflect pre-disaster preventative action and preparation; and

post-disaster response and recovery. Where possible, indicators should reflect

robustness, redundancies, response, and recovery phases of a disaster.

9. The indicators should focus on micro-level observations at small units of analysis and

remain independent of macro-level demographic and socio-economic variables, for

example per capita income, household size, or literacy level.

10. The indicators should reflect standards in cultural competence and be appropriate

across project sites.




Reflecting these criteria, the proposed evaluation framework identifies four key indicator

categories that together make a composite measure of student knowledge gain and disaster

preparedness and community resilience:

1. Earthquake Science and Safety Knowledge

This indicator category is associated with the earthquake science and safety curriculum as

taught in the program and is designed to capture student knowledge transfer as a result of

the program.

2. Risk Perception

Risk perception includes changes in hazard awareness, knowledge of disaster impacts and

risk of disasters, and increased perception of personal and community vulnerabilities.

This indicator is designed to capture participant and community changes in risk

perception.

3. Structural and Non-Structural Safety

This category of indicators reflects changes in building design and construction that may

increase resistance to the variant impacts of earthquakes, in addition to changes to non-

structural components including windows, doors, large furniture, and other household

items.

4. Emergency Preparedness and Planning

The final indicator category captures changes in planning that may be considered micro-

level institutional changes, including evacuation plans and other preparedness measures.

Some of the metrics captured here are not proposed directly by the curriculum, but may

reflect further initiative and action as a result of program participation.

The evaluation framework is designed to measure these indicators using a variety of data

collection methods depending upon the resources available and situation in which the evaluation

is taking place. Table 1 summarizes examples of indicator metrics.




Table 1: Indicator Categories and Example Metrics

Indicator Category Example Metrics

Earthquake Science

and Safety Knowledge

• Student knowledge of earthquake science

• Student understanding of structural safety

• Student understanding of liquefaction

• Teacher comfort teaching earthquake science

Risk Perception

• Student/teacher awareness of disaster likelihood

• Student/teacher perception of importance of preparing

• Student teacher perception of potential damage

• Student/teacher perception of preparedness strategies

Structural and Non-

Structural Safety

• Student/teacher can identify vulnerabilities in home/school

• Student/teacher has taken action to correct vulnerabilities in

home/school

Emergency

Preparedness and

Planning

• Student/teacher has an emergency evacuation plan at home/school

and regularly practices plan

• Student/teacher has prepared an emergency kit or has access to

identified emergency items

• Students have identified family safe locations, emergency

numbers, and outside emergency contacts

• Teachers and schools have designated emergency staff and

emergency plans for evacuation and post-earthquake scenario

We have provided questionnaires at the end of this document that include a full set of associated

questions for each of these indicator categories. We recommend the use of these questions or similar metrics in assessments of disaster preparedness and community resilience prior to and

after the GQSSI is implemented. This is further discussed in Section 6.1. Example surveys

including final indicators and metrics can be found in Appendix E, Appendix F, and Appendix

H.




5. The GQSSI Evaluation Framework

We recognize that the GQSSI may be operating in a number of sites which will have unique

characteristics, and that researchers may encounter different limitations in resources such as staff and time. Acknowledging site-specific resource and time constraints, researchers may face three

key decisions in planning an evaluation: 1) research design; 2) data collection sources; and 3)

data collection methods. We designed tools that allow researchers to consider a combination of

research design elements and data collection methods to evaluate the program.

We have designed a reference tool which allows researchers to consider a combination of

experimental research design elements. Using this tool, researchers can decide which

components are feasible within the constraints they face. We have also chosen to provide

information about various methods for data collection that researchers can consider. We have

developed a reference chart which outlines strengths and weaknesses of different data collection

methods, and includes some guidance on which may be most appropriate in certain situations.

Our intention is to provide researchers with an outline of choices for the creation of a site-

specific evaluation plan. We have chosen this type of guided toolkit, rather than a strict plan

based on “promising practices”, because we recognize that it may not be feasible for researchers

to operate under the conditions that such a prescriptive plan may require.

5.1. Choosing a Research Design

There are three key components to an experimental research design: 1) selection of both a

control and a treatment group; 2) random assignment of participants to the control or treatment

group; and 3) administration of both a pre- and post-test. Table 2 provides a graphic

demonstration of these components and the decisions to be made by researchers.

Selecting a control group in addition to a treatment group adds an important element of

comparison to a research design. Having a control group helps identify whether the results seen

in the treatment group are causally related to the treatment participants are receiving, or if there




is another factor that both the control and treatment group are exposed to which may be causing

a certain impact. For example, a control group may be a school in an adjacent and non-

interacting district which may have been considered as a project site but not chosen.

Though selecting a control group can show a level of causality from the treatment, non-random

methods of assigning control and treatment groups still allow for selection or sampling bias. A

research design is made even stronger by using random assignment to select participants into

each group. Random selection aims to ensure that there is not a pre-experiment difference

between participants in each group. Every participant has the same chance of being randomly

selected into either the treatment or the control group, so neither group should be biased.

Using a pre- and post-test design allows researchers to measure the potential impacts of a

treatment by examining the differences in results before and after a program. Research designs

that do not include a pre-test cannot reliably demonstrate that results of a post-test are an

outcome of the treatment, as there is no baseline for comparison.

Table 2: Choosing a Research Design




These three components, if used together in a research design, indicate a strong and valid study.

The validity of a study indicates the accuracy and representativeness of its results, the strength of

the causal relationship, and the extent to which a measurement can be expected to produce

similar results on repeated observations. Using a randomly assigned, control-treatment, pre- and post-test research design adds internal validity to an evaluation. Internal validity means that it

can be determined that a treatment caused a measurable outcome, and to what magnitude

(Newcomer, 2011). An experimental design which incorporates the three above-mentioned

elements increases internal validity because these components help demonstrate that the

outcomes are caused by the treatment, rather than other factors.

There are many situations in social science research where a research design cannot incorporate

all three experimental elements. This will likely be the case in an evaluation of the GQSSI, given

location, resources, personnel, political, and logistical challenges across the multiple sites. For

example, a community that is providing the earthquake science and safety curriculum to students

in all schools may not have a comparison group available to serve as a control. If a site decides to

implement an accelerated curriculum, researchers may not have enough time to pre-test students,

teachers, and community stakeholders. Should conditions allow, we recommend that evaluations

of the GQSSI attempt to use a randomly assigned, control-treatment, pre- and post-test research

design. However, researchers at each GQSSI site should assess their situation and attempt to

incorporate as many of these components as possible while choosing the research design that

best fits their individual needs.

5.2. A Discussion of Data Collection Methods

The way in which information is collected from GQSSI program sites is important for a variety

of reasons. Appropriate data collection methods can ensure valid results, higher response rates,and strong community engagement. Cultural norms, literacy levels, and power relationships must

also be considered when choosing a mode of data collection. There is no one ‘best’ mode to

collect information; research can be conducted several ways – face to face, on the phone, through

the mail, or electronically – but certain audiences may be less receptive depending on the method




used. We suggest that TWB establish a broad repertoire of modes to ensure accessibility to, and

acceptability of, data collection methods.

The inclusion of community stakeholders is important for the purposes of both programimplementation and evaluation (McDonald, Kutara, Richmond, & Betts, 2004). Determining

who to collect data from can depend on family and community dynamics. Engaging with

community leaders, and getting to know program participants, can help distinguish who may be

the most relevant and reliable data sources. By establishing cooperation and trust, response rates

and accuracy may increase, and there may be less time focused on speculative interpretation of

the questions and the data gathered. For more information on engaging with community

stakeholders, please see Section 6.2.

Table 3 on page 25 contains four primary modes of collecting data that the GQSSI evaluation

may utilize: surveys, focus groups, interviews, and observations. Each option has strengths and

weaknesses, so carefully selecting the mode can help protect the validity of data collection.

1) Surveys

Written surveys allow researchers to ask a broad range of questions, some of which can

be closed-ended and some open-ended. Participants are all asked the same questions and

information is collected with a standardized procedure. Respondents are able to answer

questions at their own pace, and able to think them through rather than being rushed or

pressured to deliver an answer. Although surveys are often considered one of the

cheapest ways to collect information from large numbers of participants, it is also the

mode with the highest non-response rate. If not done properly, surveys may embed bias

in questions which can sway participants to answer one way or another. Researchers

should consider that surveys require a degree of literacy.

2) Focus Groups

A focus group is a form of qualitative data collection where groups of participants come

together and are asked to speak about their opinions, attitudes, and knowledge about a

certain topic or event. Focus groups are generally loosely structured to encourage




participants to discuss their experiences and attitudes with other participants. Although

time consuming, focus groups are generally cheap. Focus groups can help understand in

depth why participants feel a certain way, but the results cannot always be generalizable.

Participants may not want to voice their opinions due to group dynamics, or be unsureabout revealing true feelings with a moderator present. On the other hand, focus groups

can stimulate discussion and allow participants to develop their opinions and gain

perspectives. Focus group data can help reveal shared attitudes or understandings, but it

is important not to extrapolate those results to an entire population.

3) Interviews

Compared to focus groups, structured interviews represent a more systematic method to

collecting data, and can also help researchers gain awareness and perspective. The

person-to-person nature of interviews requires a large time commitment from researchers,

but can have a quick turnaround time. Questions should be designed to ensure uniformity

and comparability among respondents. Interviews can have open-ended questions which

help provide context and further explanation, but these can be difficult to quantify.

Interviews can be useful when translation is required or when literacy is an issue.

Researchers should be aware that participants may feel inclined to provide socially

desirable answers in the presence of interviewers.

4) Observations

Observations allow researchers to monitor programs without direct interference with

participants or activities. Researchers are able to gain a broader understanding of a

program’s operations, and to obtain perspective on aspects of the program that may not

be self-reported by participants. Participants may behave differently if they know an

outside observer is watching them, creating a false perception of the program (i.e. theHawthorne effect). Observers may come into a situation with a set agenda of what to look

for or what to pay attention to, which can encourage biased reporting. Observations are

useful when there are severe language or cultural barriers that prevent interaction with

program participants.




Table 3. Strengths and Weaknesses of Data Collection Methods

Strengths Weaknesses

Surveys • High volume of participants

• Standardized questions

• Easier to analyze and collate

information

• Can be mass produced at low costs

• Respondents are able to answer at

their own pace

• Potential low response rate

• Literacy barriers

• Potential question bias

Focus

Groups

• Open-ended questions allow for in-

depth response• Can be used to capture overall

perceptions of a program

• Low cost

• Large amounts of time for multiple

groups• Harder to analyze

• Difficult to assess changes over time

• Results are not generalizable• Group dynamics may inhibit

participant honesty

Interviews • Structured for comparability across

sites• Standardized questions

• Can be used where there are

language and/or literacy barriers• Quick turnaround time

• Expensive and time-intensive

administration• May encourage socially desirable

responses

Observations • Allow researchers to gain context

and perspective• Can be used where there are cultural

and/or language barriers

• Minimal interference from

researchers

• Low cost

• Results are not generalizable

• Subject to Hawthorne effect• Observers may have a preset agenda

In Appendix D we have included a data collection criteria matrix. The matrix details a series of

criteria with which to guide the researchers’ decisions about data collection. This chart is

intended to help TWB when deciding which methods of data collection will not only be most

useful, but also most feasible, when carrying out an evaluation of the GQSSI.




6. Sample Instruments

To complement our discussion of the options that researchers have in designing their evaluation

structure and methods, we have developed specific examples of evaluation instruments. We have

provided student and teacher pre- and post-program questionnaires which aim to measure

changes in the chosen indicators. In addition, we have offered options for shorter surveys or

focus groups which could be administered to parents, principals and local officials to evaluate

awareness of the GQSSI program and to identify broad impacts on the community. We recognize

that the instruments we provide are not exhaustive of all the options available for use in an

evaluation plan, but we believe that they can provide guidance for researchers in forming their

own evaluations.

6.1. Student and Teacher Questionnaires

In Section 4 we discussed specific indicators that aid in measuring the impact of the GQSSI. This

section demonstrates how these indicators can be used in sample student and teacher

questionnaires found in Appendix E and Appendix F.

We have designed these questionnaires to be flexible so they can be administered using a rangeof data collection methods, and recognize that available resources may vary. The final

questionnaire is simple, direct, and robust. It is offered in two versions to capture effects on both

students and teachers. Both versions are available as pre- and post-test questionnaires with only

slight differences.

Per our guide to indicators and metrics, the questionnaire assesses changes across four indicator

categories, 1) earthquake science and safety knowledge; 2) risk perception; 3) structural and non-

structural safety; and 4) emergency preparedness and planning. Questions take the form of

multiple choice, Likert scale, and dichotomous choice (yes/no), with three to six questions in

each category. It is important to note that in this version there are no open-ended questions. The

given questionnaire includes a cover page with instructions, a disclaimer, and basic demographic

questions including age, grade, and gender.




We recommend using the pre-test version of this questionnaire prior to program implementation

to assess the level of disaster preparedness and community resilience among participants before

the program (both students and teaching staff). Once the program is completed the provided post-test can be administered. When administering the post-test, researchers should consider that

program effects may not be immediately observable. To capture continued effects, we

recommend repeated assessments at regular time intervals following the program.

As discussed in Section 5.2 the questionnaire can be administered using data collection methods

that researchers consider appropriate; in the current design this is best done through distributed

surveys or in-person interviews. The latter is preferable where there are concerns with literacy

levels or language barriers. Where possible, all participants should be surveyed or interviewed to

ensure robust results.

A coding tool found in Appendix G includes correct answers to questions contained in section

one of the questionnaire: earthquake science and safety knowledge. The coding tool also

provides information that will assist interpretation of results from the other three sections.

6.2. Community Stakeholder Tools

One of the overarching goals of the GQSSI is to foster a greater awareness, preparedness and

resilience within the communities where the curriculum is being implemented. This section

provides guidance on how to measure the community impact of the earthquake science and

safety education program. Due to the differences across sites and the development challenges

facing each locale individually, the goal of measuring community impact does not necessarily

give way to a simple or easily replicable methodology. Acknowledging that rigorous data

collection methods may not be as applicable nor feasible in collecting community-level

information, we have created an alternative design that can still enable researchers to measure

community impact of the GQSSI.




In order to obtain community-level data, researchers must reach community-level stakeholders.

Identifying persons who represent greater areas of the community, aside from students and

teachers involved in the program, is the first step. Community stakeholders may include:

a) Police force, firemen, or other safety officials;

b) Principals of schools (where program is implemented);

c) Parents of children (who are involved in the program);

d) Mayor, Governor, or local leaders;

e) Other teachers (that are not involved in the program); and

f) Faith-based community leaders.

Twice throughout the course of the curriculum implementation, (mid-year and end of year) ashort questionnaire should be distributed to community stakeholders. This questionnaire can be

administered in various forms. For example:

a) Person-to-person, as a casual conversation;

b) Paper postcard or letter through the mail;

c) Email, if the resources are available;

d) Phones; and

e)

Town hall meetings or focus groups.

The key is to understand the awareness of the community and to identify changes that have

occurred as a result of, or in correlation with, the GQSSI curriculum implementation. Such a

questionnaire should thus be highly user-friendly and easily understandable. A sample survey for

community stakeholders can be found in Appendix H.

Asking only a few short questions will increase the likelihood of response even if a community

member has no recognition of the program. The straightforward nature of the questions can

allow researchers to easily gauge an individual’s interest in and interaction with the education

program. Such a questionnaire can provide researchers with meaningful, yet simple community-

level data concerning the wider effect of the GQSSI, thus addressing the greater goal of the

earthquake science and safety education program.




7. Conclusion

In this report we have provided guidelines for designing an evaluation framework which

accurately and appropriately measures the effect of the GQSSI on disaster preparedness andcommunity resilience. We have provided background research in disaster mitigation and

evaluations of disaster education programs to support the criteria and indicators we suggest to

use in an evaluation of the GQSSI. Our framework aims to provide GQSSI researchers with

contextual knowledge in the elements of experimental research design and data collection

methods so that they are well-equipped to examine and choose evaluation options based on the

individual sites in which they operate. The data collection metrics and instruments which we

provide can serve as an example to researchers, which can be modified as needed. Our intention

in providing this evaluation framework package is to demonstrate how the metrics, design, and

methods used in an evaluation can reflect indicators of the GQSSI program impact on students,

teachers, and the community.

In order to utilize the information and tools we have provided, we recommend some necessary

next steps for researchers at GQSSI sites to take. Researchers should first assess the climate of

their individual sites to help guide decision-making on appropriate evaluation design and

metrics. This may include assessing the availability of technology or other resources needed for

an evaluation, time limitations, literacy level of the intended program participants, and

community norms and acceptance. Researchers may also need to determine who will be

conducting the survey – TWB or GQSSI program staff; third-party data collection personnel; or

teachers. After researchers gain a full perspective on the environment in which the evaluation is

to be conducted, they should be able to begin to answer some of the critical design and methods

questions that we propose in this document. Once appropriate research design and data collection

methods are determined, researchers may need to develop their own evaluation instruments,

referencing the examples we have provided for guidance.

We recommend that TWB utilize the full range of tools within this evaluation framework. In

doing so, we believe researchers should be able to obtain measurable results of the impact that

the program has not only on student knowledge, but on community preparedness and resilience




in the event of an earthquake. Such impact data will strengthen the argument for the necessity of

disaster education research in these countries and other vulnerable regions of the world.

Increased identification of the concrete outcomes of the program can lead to additional fundingfor the GQSSI and allow it to expand to more areas which could benefit from disaster education

programs. Successful evaluation of the GQSSI may also fuel additional research in the fields of

disaster education and community mitigation strategies.




8. References

American Evaluation Association. (2011, April). Statement on Cultural Competence. RetrievedMarch 4, 2013, from American Evaluation Association website:

http://www.eval.org/ccstatement.asp

Appleton, J. J., Christenson, S. L., & Furlong, M. J. (2008). Student Engagement with School:Critical Conceptual and Methodological Issues of the Construct. Psychology in the

Schools, 45(5): 369-386.

Asian Disaster Preparedness Center. (2003). The School Earthquake Safety Program inKathmandu Valley: Building Safer Communities through Schools. Safer Cities 4,

AUDMP . Retrieved from http://www.adpc.net/audmp/library/safer_cities/4.pdf

Asian Disaster Preparedness Center. (2004). Creating Earthquake Preparedness in Schools: A

Case Study of Mitigation Efforts in Indonesia. Safer Cities 10, AUDMP . Retrieved fromhttp://www.adpc.net/audmp/library/safer_cities/10.pdf

Asian Urban Disaster Mitigation Program (AUDMP). (2012, July 30). Retrieved March 18,2013, from Asian Disaster Preparedness Center website:

http://www.adpc.net/audmp/audmp.html

Bansal, B. K., & Verma, M. (2012). Education and Awareness: A Key to Earthquake Risk Reduction. Journal Geological Society of India, 80, 451-454. Retrieved from

http://link.springer.com/article/10.1007%2Fs12594-012-0164-2

Bartko, T., & Eccles, J. (2003). Adolescent Participation in Structured and UnstructuredActivities a Person Oriented Analysis. Journal of Youth and Adolescence, 32(4), 233-

241.

Bohnert, A., Fredricks, J., & Randall, A. (2010). Capturing Unique Dimensions of YouthOrganized Activity Involvement: Theoretical and Methodological Considerations. Review

of Educational Research, 80(4), 576-610.

Bruneau et al. (2003, November). A Framework to Quantitatively Assess and Enhance theSeismic Resilience of Communities. Earthquake Spectra, Volume 19, No. 4. Earthquake

Engineering Research Institute.

Burton, C. G., Dr. (2012). Development of Metrics for Community Resilience to NaturalDisasters. Colombia, SC: The University of South Carolina.

Campbell, D. (2006). What is Education's Impact on Civic and Social Engagement?

Organization for Economic Cooperation and Development (OECD). Retrieved February,2013, from OECD website: http://www.oecd.org/edu/country-studies/37425694.pdf




Corno, L. (1993). The Best Laid Plans Modern Conceptions of Volition and Educational

Research. Educational Researcher , 22(2), 14-22.

Earthquake Hazards Program. (n.d.). Retrieved fromhttp://earthquake.usgs.gov/earthquakes/eqarchives/year/eqstats.php

Edwards, M. L. (1993). Social Location and Self-Protective Behavior. Implications for

Earthquake Preparedness. International Journal of Mass Emergencies and Disasters, 11(3).

Faupel et al. (1992, March). The Impact of Disaster Education on Household Preparedness for

Hurricane Hugo. International Journal of Mass Emergencies and Disasters, Vol. 10, No.1.

McDonald, D., Kutara, P., Richmond, L., & Betts, S. (2004). Culturally Respectful Evaluation.

The Forum for Family and Consumer Issues, 9(3).

Milligan, K., Moretti, E., & Oreopoulos, P. (2003, March). Does Education ImproveCitizenship? Evidence from the US and the UK. Cambridge, MA: National Bureau of

Economic Research.

Mulilis, J. P., Duval, T. S., and Lippa, R. (1990). The Effects of a Large Destructive LocalEarthquake on Earthquake Preparedness as Assessed by an Earthquake Preparedness

Scale. Natural Hazards, Volume 3, Issue 4.

Nature Education. (2013). Earthquake Science - Library. Retrieved from Scitable website:http://www.nature.com/scitable/partner/earthquake-science-8666053/library

Newcomer, K.E. (2011). Strategies to Help Strengthen Validity and Reliability of Data.

Unpublished paper. The George Washington University, Washington, DC.

O’Brien, R. L., Kosoko-Lasaki, O., Cook, C. T., Kissell, J., Peak, F., & Williams, E. H. (2006).

Self -Assessment of Cultural Attitudes and Competence of Clinical Investigators to

Enhance Recruitment and Participation of Minority Populations in Research. Journal of National Medical Association, 98 (5), 674-682.

Parsizadeh, F., & Ghafory-Ashtiany, M. (2010). Iran Public Education and Awareness Program

and its Achievements. Disaster Prevention and Management , 19 (1), 32-47. Retrieved

from www.emeraldinsight.com/0965-3562.htm

Ronan, K. R., & Johnston, D. M. (2003). Hazards Education for Youth: A Quasi-Experimental

Investigation. Risk Analysis, 23, 1009-1020.

Ronan, K. R., Crellin, K., & Johnston, D. M. (2012). Community Readiness for a New TsunamiWarning System: Quasi-experimental and Benchmarking Evaluation of a School

Education Component. Natural Hazards, 61, 1411-1425.




Sakiroglu, M. (2005, August). Variables Related to Earthquake Preparedness Behavior. Ankara,Turkey: Middle East Technical University.

Selby, D. and Kagawa, F. (2012, July). Disaster Risk Reduction in School Curricula: Case

Studies from Thirty Countries. United Nations Educational, Scientific and CulturalOrganization (UNESCO) and United Nations’ Children Fund (UNICEF). Retrieved from:

http://www.unicef.org/education/files/disaster risk reductioninCurricula-Mapping30countriesFINAL.pdf.

Shaw, R., & Shiwaku, K. (2008). Proactive Co-learning: A New Paradigm in Disaster Education.

Disaster Prevention and Management , 17(2), p. 183-198.

Simpson, D. M., Dr. (2006, September). Indicator Issues and Proposed Framework for a Disaster Preparedness Index. Louisville, KY: Center for Hazards Research and Policy

Development, University of Louisville.

United Nations. (2004). International Strategy for Disaster Reduction: Living With Risk, AGlobal Review of Disaster Reduction Initiatives (Volume II). (2004). Geneva,

Switzerland: United Nations.

United Nations. (2007). International Strategy for Disaster Reduction: Hyogo Framework for Action 2005-2015: Building the Resilience of Nations and Communities to Disasters.

Geneva, Switzerland.

W.K. Kellogg Foundation. (2004, January). Logic Model Development Guide. Battle Creek, MI:W.K. Kellogg Foundation. Retrieved from http://www.wkkf.org/knowledge-

center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx

Wadia Institute of Himalayan Geology. (2009). Tools and Facility: Seismological Laboratory.Retrieved from Wadia Institute of Himalayan Geology website:

http://www.wihg.res.in/geophyt_facilities.html




Appendices

Appendix A. Client Contact Information

Dr. Fred Mednick, Founder, Teachers Without BordersVisiting Fellow/Lecturer, John Hopkins University, School of

[email protected]

Teachers Without BordersP.O. Box 25607

Seattle, WA98165

Mission: Teachers Without Borders connects teachers to information and each other to create

local change on a global scale.

Website: http://www.teacherswithoutborders.orgTwitter Handle: @teachersnetwork

Facebook: Teachers Without Borders




Appendix B. Logic Model






Lesson 9: Structural Hazards

Students are introduced to basic concepts behind structural hazards in the context of earthquakes.

There are many cities that have a variety of building sizes, shapes and architectural styles andthis lesson discusses the basic ideas concerning how structures respond to earthquakes using a

tabletop exercise and three hands-on activities.

Lesson 10: Non-Structural Hazards

Students learn to identify potential earthquake hazards associated with non-structuralcomponents to their residential buildings and to provide recommendations for mitigating them.

Lesson 11: Earthquake Drills, Plans, and Supplies

Students learn to recognize the importance of advanced planning for their school in case of an

emergency. This provides guidance for conducting and preparing an emergency response plan

and drill regimen. Students test, evaluate, improve and present emergency plans to appropriateauthorities, emphasizing the importance of regularly practiced drills in schools.

Lesson 12: Making a Single Signature Book

Using information from previous lessons, students write stories about individuals or communitiesaffected by an earthquake and publish their stories in a single signature book.




Appendix D. Data Collection Criteria Matrix

As mentioned earlier in our report, there is no one “best” way to collect data. Decisions aboutdata collection are often based on similar criteria, regardless of the nature of the program or its

location. Cost, personnel, time, and audience receptivity, as examples, are common factors thatresearchers consider when deciding on their methodology. The considerations required of data

collectors can be universally applied and are critical to the success of any program evaluation. Inan effort to aid TWB’s decisions about data collection methods, we have created the chart

pictured below.

The Criteria Matrix below details a series of criteria with which to guide the evaluators’decisions about data collection. This chart is intended to help TWB in deciding which methods

of data collection will not only be most useful, but also most feasible, when carrying out anevaluation of the GQSSI. Each of the criteria applies differently to each method of collecting

data. We have only provided content with regards to a few criterion examples; additional criteria

can easily be added to the chart and populated by evaluators to further support their decision-making.

Time, cost, personnel, validity, and access/cultural factors are the criteria that our group deemedmost important to consider when designing a research design and data collection methodology

for GQSSI. Time implies considerations of how long data collection will take, both on the part of the researchers as well as the participants. Cost implies the monetary cost of conducting each

specific type of data collection. Validity implies the accuracy of the information gained fromeach form of data collection and its reliability as a true reflection of the program effects. Access

and cultural factors imply the feasibility of conducting the individual methods, and therecognition of cultural norms.







Appendix E. Sample Student Questionnaires

Questionnaire – Student Pre-Test

Global Quake Science and Safety Initiative

This document should be used as a survey for students. This is the version to be completed by participants

before the program begins.

Instructions:

Thank you for agreeing to participate in this survey of the earthquake science program. Your participation

in this survey will help us understand how much the earthquake science program may prepare you, your

school and your community in the event of an earthquake.

Your participation in this survey is voluntary. Please ensure you answer all questions truthfully. This is an

assessment of your knowledge and opinions. Your responses are anonymous and will not be shared. This

survey poses no risk to you and there is no penalty for refusal to participate. You may refuse to participate

in this survey simply by returning the survey without completing it.

If you have any questions regarding your participation in this survey, please contact:

1. Please do not place your name or your school name on this survey

2. Do not skip any items

3. Ask for clarification if you need to

4. Return the completed survey to the researcher when finished

5. A pen or pencil can be used to complete the survey

6. If any of the options do not apply, feel free to provide comment

Male □ Female □

Student□ Other □ (if other, please explain)

Age Grade

Site/Location

Date (dd/mm/yy):




Section OneEarthquake Science and Safety Knowledge

1. Which of the following are causes of earthquakes? (check all which apply)

A ○ Thunderstorms and heavy rain F ○ Tectonic plates move relative to each other

B ○ Large creatures inside the Earth move G ○

People build dams and reservoirs

C ○ People use heavy machinery to drill for oil H ○ Rocks deep below the surface of the earth are

heated up and move

D ○ Large plates of the Earth’s crust grind

against each other I ○ Famine and drought

E ○ People commit sinful acts and bad deeds J ○ Energy stored at the border of tectonic plates is

released

2. Put the following earthquake effects in order, marking them from 1 (first effect) to 5 (last effect)

A○ Energy is released

B ○ Earthquake waves reach the Earth’s surface and people feel shaking

C

○ Rocks deform and store energy at cracks in the Earth

D○ Accumulated energy becomes greater than forces keeping rocks together

E ○ Earth’s plates move relative to one another

3. Which of the following may occur as a result of an earthquake? (check all which apply)

A ○ Thunderstorms and heavy rain D ○ Flooding and fires

B ○ Liquefaction E ○ Landslides

C ○ Building collapse F ○ Tornados




4. Which of the following events occur during liquefaction? (check all which apply)

A ○ Flooding D ○ Thunderstorms

B ○ Buildings sink into the ground E ○ Underground pipes and cables break

C ○ Tsunami’s and large waves F ○ Saturated earth adopts liquid characteristics

5. Which of the following events may increase the chance of a landslide occurring? (check all which apply)

A ○ Deforestation D ○ Construction work

B ○ Steep slopes E ○ Heavy rain

C ○ Loose hillside materials F ○

An earthquake

6. Which of the following may pose a risk during an earthquake? (check all which apply)

A ○ Poor building designs F ○ Lighting fixtures

B ○ Machinery and vehicles G ○ Weak construction materials

C ○ Bookshelves and other shelving H ○ Gas canisters

D ○ Flash flooding I ○ Falling rocks

E ○ Windows and glass J ○ Electrical wiring, cables, and water pipes




Section TwoRisk Perception

1 2 3 4 5

1. What is your expectation of

an earthquake happening

within the next five years?

Very unlikely ○ ○ ○ ○ ○ Very likely

5 = an earthquake is very likely to occur

1 2 3 4 5

2. How important do you think

it is for you and your family

to prepare to protect against

earthquakes?

Not important ○ ○ ○ ○ ○ Very important

5 = it is very important that my family takes action to prepare and protect against earthquakes

1 2 3 4 5

3. How confident are you that

your family knows what

action to take in the event of

an earthquake?

Not

confident ○ ○ ○ ○ ○ Very confident

5 = my family has a very good evacuation and emergency plan if an earthquake happened

1 2 3 4 5

4. If an earthquake was tooccur tomorrow, how well

prepared do you feel your

family is to cope with

immediate consequences?

Very

unprepared ○ ○ ○ ○ ○ Very prepared

5 = my family is very well-prepared to cope with the immediate consequences of an earthquake, (i.e., food and

water supplies, shelter, access to healthcare, money




Section ThreeStructural and Non-Structural Safety

1. Which of the following vulnerabilities can you identify in buildings in your community? If none are

relevant, please leave empty (check all that apply).

A ○ Indirect load path F ○ Discontinuous vertical structure

B ○ Irregularity of building structure G ○

Heavy roof

C ○ Weak first story H ○ Heavy building structure

D ○ Loose or unsecured pipes and cables I ○ Weak window fixtures

E ○ Open shelves J ○ Vehicles parked on hills

2. Which of the following items in your home has your family secured in case of an earthquake? If none are


A ○ Bookshelves and books F ○

Chemicals and cleaning supplies

B ○ Cupboards G ○ Tables

C ○ Computers and televisions H ○ Wall hangings and paintings

D ○ Windows I ○ Utilities including pipes and cables

E ○ Gas canisters and kitchen equipment J ○ Doors

3. Which of the following steps have your family taken to strengthen the structure of your home in the last

year? If none are relevant, please leave empty (check all that apply).

A ○ General repairs, e.g. doors, windows,

cracks and re-plastering as neededD ○

Utility repairs, e.g. checking and replacing (if

needed) electrical wiring and water plumbing

B ○ Removal of damaged or weakened walls

and replacing with stronger materialE ○

Replacing defective or damaged wooden

trusses and roof supports

C ○ Reinforced or replaced walls including

cross-walls and reconstructionF ○

Reinforced, strengthened, or replaced roofs and

floors




Section Four Emergency Preparedness and Planning

1. Does your family have an emergency response and emergency evacuation plan?

○ Yes ○ No

2. How often does your family practice its emergency response and evacuation plan? For example, a drill?

○ Once a

month ○ Once every

six months ○ Once a

year ○ Less than

once a year ○ Never

3. Which of the following items do you have stored in an emergency kit in case of an earthquake? If none

are relevant, please leave empty (check all that apply).

○ Flashlight or torch ○

Radio

○ Mobile or cellular phone ○ Medical supplies

○ Water and food supplies ○ Tape

○ Blankets ○ Batteries

4. Do you and your family have a way to contact each other in case of an earthquake? For example, a phone

number or meeting place?

○ Yes ○ No

5. Do you have an emergency contact outside of your community? For example, an emergency official andphone number?

○ Yes ○ No




Questionnaire – Student Post-Test


This document should be used as a survey for students. This is the version to be completed by participantsbefore the program begins.

Instructions:


in this survey will help us understand how much the earthquake science program may prepare you, your

school and your community in the event of an earthquake.












Male □ Female □


Age Grade

Site/Location

Date (dd/mm/yy):






A ○ Thunderstorms and heavy rain occurs F ○ Tectonic plates move relative to each other

B ○ Large creatures inside the Earth move G ○

People build dams and reservoirs

C ○ People use heavy machinery to drill for oil H ○ Rocks deep below the surface of the earth are

heated up and move

D ○ Large plates of the Earth’s crust grind

against each other I ○ God commands that an earthquake occurs

E ○ People commit sinful acts and bad deeds J ○ Energy stored at the border of tectonic plates is

released

2. Put the following earthquake effects in order, marking them from 1 (first effect) to 5 (last effect)

A○ Energy is released

B ○ Earthquake waves reach the Earth’s surface and people feel shaking

C

○ Rocks deform and store energy at cracks in the Earth

D○ Accumulated energy becomes greater than forces keeping rocks together

E ○ Earth’s plates move relative to one another


A ○ Thunderstorms and heavy rain D ○ Flooding and fires

B ○ Liquefaction E ○ Landslides

C ○ Building collapse F ○ Tornados





A ○ Flooding D ○ Thunderstorms

B ○ Buildings sink into the ground E ○ Underground pipes and cables break

C ○ Tsunami’s and large waves F ○ Saturated earth adopts liquid characteristics


A ○ Deforestation D ○ Construction work

B ○ Steep slopes E ○ Heavy rain

C ○ Loose hillside materials F ○

An earthquake


A ○ Poor building designs F ○ Lighting fixtures

B ○ Machinery and vehicles G ○ Weak construction materials

C ○ Bookshelves and other shelving H ○ Gas canisters

D ○ Flash flooding I ○ Falling rocks

E ○ Windows and glass J ○ Electrical wiring, cables, and water pipes





1 2 3 4 5






1 2 3 4 5


it is for you and your family

to prepare to protect against

earthquakes?


5 = it is very important that my family takes action to prepare and protect against earthquakes

3. As a result of your participation in the program, does your family have a better understanding of

emergency evacuation and planning procedures?

○ Yes ○ No

4. As a result of your participation in the program, is your family better equipped to cope with the

immediate damages that may arise as a result of the earthquake?

○ Yes ○ No









Heavy roof




2. Which of the following items in your home has your family secured in case of an earthquake? If none are








3. Which of the following steps have your family taken to strengthen the structure of your home in the last

year? If none are relevant, please leave empty (check all that apply).












floors






○ Yes ○ No


○ Once a



year ○ Less than


3. Which of the following items do you have stored in an emergency kit in case of an earthquake? If none

are relevant, please leave empty (check all that apply).

○ Flashlight or torch ○

Radio




4. Do you and your family have a way to contact each other in case of an earthquake? For example, a phone

number or meeting place?

○ Yes ○ No

5. Do you have an emergency contact outside of your community? For example, an emergency official andphone number?

○ Yes ○ No




Appendix F. Sample Teacher Questionnaires

Questionnaire - Teacher Pre-Test


This document should be used as a survey for teachers. This is the version to be completed by participants

before the program begins.

Instructions:


in this survey will help the program organizers understand how much the earthquake science program

may prepare you, your school and your community in the event of an earthquake.












Male □ Female □


Age Grade Taught

Site/Location

Date (dd/mm/yy):





1 2 3 4 5

1. How comfortable are you

teaching about the causes of

earthquakes?

No prior

knowledge ○ ○ ○ ○ ○ Very knowledgeable

5 = I am able to teach students about the underlying science of plate tectonics, plate boundaries, plate friction,

seismic waves, the core and mantle, and the movement of the crust

1 2 3 4 5

2. How comfortable are you

teaching about the naturaleffects of earthquakes?

No prior


5 = I am able to teach students about the underlying science behind liquefaction, landslides, flooding, and

structural and non-structural damages/building collapse due to seismic waves

1 2 3 4 53. How comfortable are you

teaching students about

earthquake hazards and

safety precautions?

No prior


5 = is able to teach students about risks to buildings and non-structural risks, in addition to emergency plans

including evacuation routes and drills





1 2 3 4 5






1 2 3 4 5


it is for your school to

prepare to protect teachers

and students against

earthquakes?


5 = it is very important that the school takes action to prepare and protect against earthquakes

1 2 3 4 5

3. How confident are you that

teachers in your school are

able to lead students to safety

in the event of an

earthquake?

Not confident ○ ○ ○ ○ ○ Very confident

5 = teachers at my school know how to support students in the event of an earthquake happening while at

school, i.e. evacuation route, etc.

1 2 3 4 5

4. If an earthquake was to

occur tomorrow, how well

prepared do you feel your

school is to cope with

immediate damages that may

arise?

Very

unprepared ○ ○ ○ ○ ○ Very prepared

5 = the school is well prepared to cope with the immediate consequences of an earthquake, i.e. emergency

education plans, structural damage, contingencies









Heavy roof




2. Which of the following items in your school have been secured in case of an earthquake? If none are








3. Which of the following steps have been taken to strengthen the structure of your school in the last year?

If none are relevant, please leave empty (check all that apply).












floors





1. Does your school have an emergency response and emergency evacuation plan?

○ Yes ○ No

2. How often does your school practice its emergency response and evacuation plan? For example, a drill?

○ Once a



year ○ Less than


3.Which of the following items do you have stored in your classroom in case of an earthquake? If none are


○ Flashlight or torch ○ Radio




4. Has your school identified a staff member(s) responsible for emergency planning and preparedness?

○ Yes ○ No

5. Does your school have a procedure in place for contacting parents in the event of an earthquake?

○ Yes ○ No




Questionnaire - Teacher Post-Test


This document should be used as a survey for teachers. This is the version to be completed by participantsafter the program ends.

Instructions:


in this survey will help the program organizers to understand how much the earthquake science program

may prepare you, your school and your community in the event of an earthquake.












Male □ Female □


Age Grade Taught

Site/Location

Date (dd/mm/yy):





1 2 3 4 5

1. As a result of your

participation in the program,

how do you feel your ability

to teach about the causes of

earthquakes has changed?

No change ○ ○ ○ ○ ○

Significant

improvement in

knowledge

5 = My knowledge of the causes of earthquakes has significantly improved, meaning I can teach about theunderlying science of plate tectonics, plate boundaries, plate friction, seismic waves, the core and mantle, and

the movement of the crust

1 2 3 4 5

2. As a result of your



to teach about the natural

effects of earthquakes has

changed?


Significant

improvement in

knowledge

5 = My knowledge of the effects of earthquakes has significantly improved, meaning I am able to teach

students about the underlying science behind liquefaction, landslides, flooding, and structural and non-

structural damages/building collapse due to seismic waves

1 2 3 4 53. As a result of your



to teach about earthquake

hazards and safety

precautions has changed?


Significant

improvement in

knowledge

5 = My knowledge of earthquake hazards and safety precautions has significantly improved, meaning I am able

to teach students about risks to buildings and non-structural risks, in addition to emergency plans including

evacuation routes and drills





1 2 3 4 5






1 2 3 4 5


it is for your school to

prepare to protect teachers

and students against

earthquakes?


5 = it is very important that the school takes action to prepare and protect against earthquakes

3. As a result of your participation in the program, do you feel that teachers in your school are better

prepared to lead students to safety in the event of an earthquake?

○ Yes ○ No

4. As a result of your participation in the program, do you feel your school is to better prepared to cope

with immediate damages that may arise from an earthquake?

○ Yes ○ No









Heavy roof




2. Which of the following items in your school have been secured in case of an earthquake? If none are








3. Which of the following steps have been taken to strengthen the structure of your school in the last year?

If none are relevant, please leave empty (check all that apply).












floors






○ Yes ○ No

2. How often does your school practice its emergency response and evacuation plan? For example, a drill?

○ Once a



year ○ Less than


3.Which of the following items do you have stored in your classroom in case of an earthquake? If none are


○ Flashlight or torch ○ Radio




4. Has your school identified a staff member(s) responsible for emergency planning and preparedness?

○ Yes ○ No

5. Does your school have a procedure in place for contacting parents in the event of an earthquake?

○ Yes ○ No




Appendix G. Questionnaire Response Coding Tool

The following coding matrix is used to analyze the data provided in the student and teacher pre-and post-test questionnaires.

Student Questionnaires

Section One (Pre- and Post-Test)

Earthquake Science and Safety Knowledge


Correct answers: D, F, H, JOne point should be given for each circle correctly checked (maximum 4)

One point should be deducted for each circle incorrectly checked (minimum 0)

2. Put the following earthquake effects in order, marking them from 1 (first effect) to 5 (lasteffect)

Correct answer: C, D, A, E, B

One point should be given for each circle correctly numbered (maximum 5)One point should be deducted for each circle incorrectly numbered (minimum 0)


Correct answers: B, C, D, E

One point should be given for each circle correctly checked (maximum 4)One point should be deducted for each circle incorrectly checked (minimum 0)


Correct answers: B, E, FOne point should be given for each circle correctly checked (maximum 3)

One point should be deducted for each circle incorrectly checked (minimum 0)


Correct answers: ALL

One point should be given for each circle checked (maximum 10)One point should be deducted for each circle not checked (minimum 0)


Correct answers: ALL




One point should be given for each circle checked (maximum 10)

One point should be deducted for each circle not checked (minimum 0)

This section should be scored in total (maximum 36). A post-test increase in score indicates a greater knowledge of earthquake science and safety.

Section Two

Risk Perception (Pre-Test)

1. What is your expectation of an earthquake happening within the next five years?


The pre-test result indicates general awareness of earthquake occurring.

2. How important do you think it is for you and your family to prepare to protect against

earthquakes?

5 = it is very important that my family takes action to prepare and protect against

earthquakes

A pre-test result indicates awareness of the important of family preparedness.

3. How confident are you that your family knows what actions to take in the event of anearthquake?

5 = my family has a very good evacuation and emergency plan if an earthquake happened

A pre-test result indicates confidence in family emergency plans.

4. If an earthquake was to occur tomorrow, how well prepared do you feel your family is to

cope with immediate consequences?

5 = my family is very well-prepared to cope with the immediate consequences of anearthquake, (i.e., food and water supplies, shelter, access to healthcare, money

A pre-test result indicates confidence in family ability to deal with earthquake

consequences.

Section Two

Risk Perception (Post-Test)






A post-test increase in expectation of an earthquake occurring indicates increased

awareness of the likelihood of an earthquake.

2. How important do you think it is for you and your family to prepare to protect againstearthquakes?

5 = it is very important that my family takes action to prepare and protect against

earthquakes

A post-test increase in this measure indicates an increased perception of the importanceof family planning for earthquakes.

3. As a result of your participation in the program, does your family have a better

understanding of emergency evacuation and planning procedures?

The post-test response indicates the participant perception of improvements in family

emergency evacuation and planning procedures as a result of the program.

4. As a result of your participation in the program, is your family better equipped to cope

with the immediate damages that may arise as a result of the earthquake?

The post-test response indicates the participant perception of improvements in resilience – such as structural and non-structural changes – as a result of the program.

Section Three

Structural and Non-Structural Safety (Pre- and Post-Test)

1. Which of the following vulnerabilities can you identify in buildings in your community?If none are relevant, please leave empty (check all that apply).

A post-test increase in the vulnerabilities identified indicates a greater awareness of

vulnerabilities within the community. This is a proxy measure that may result in, at alater stage, changes or modifications to buildings that do increase preparedness.

2. Which of the following items in your home has your family secured in case of an

earthquake? If none are relevant, please leave empty (check all that apply).

A post-test increase in the number of items secured in case of an earthquake indicates

greater preparedness for hazards caused by non-structural items during an earthquake.

3. Which of the following steps have your family taken to strengthen the structure of your

home in the last year? If none are relevant, please leave empty (check all that apply).

A post-test increase in the steps taken to secure the structure of a home indicates greater preparedness for structural hazards caused during an earthquake.




Section Four

Emergency Preparedness and Planning (Pre- and Post-Test)


A post-test increase in those responding ‘yes’ indicates greater preparedness.


A post-test increase in frequency indicates greater preparedness.

3. Which of the following items do you have stored in an emergency kit in case of an

earthquake? If none are relevant, please leave empty (check all that apply).

A post-test increase in checked items indicates greater preparedness.

4. Do you and your family have a way to contact each other in case of an earthquake? For example, a phone number or meeting place?


5. Do you have an emergency contact outside of your community? For example, an

emergency official’s name and phone number?


Teacher Questionnaires

Section One (Pre-Test)


1. How comfortable are you teaching about the causes of earthquakes?

A post-test increase in score indicates a greater comfort in teaching students about the

causes of earthquakes. This may be indicative of increased earthquake science knowledgeamong teachers.

2. How comfortable are you teaching about the natural effects of earthquakes?

A post-test increase in score indicates a greater comfort in teaching students about the

effects of earthquakes. This may be indicative of increased earthquake science knowledgeamong teachers.

3. How comfortable are you teaching students about earthquake hazards and safety

precautions?




A post-test increase in score indicates a greater comfort in teaching students about thehazards of earthquakes and emergency planning procedures. This may be indicative of

increased earthquake science knowledge among teachers.

Section One (Post-Test)


1. As a result of your participation in the program, how do you feel your ability to teach

about the causes of earthquakes has changed?

5 = My knowledge of the causes of earthquakes has significantly improved, meaning Ican teach about the underlying science of plate tectonics, plate boundaries, plate friction,

seismic waves, the core and mantle, and the movement of the crust

A post-test score indicates the extent to which teachers feel the program had an impact

on their ability to teach about the causes of earthquakes.

2. As a result of your participation in the program, how do you feel your ability to teach

about the natural effects of earthquakes has changed?

5 = My knowledge of the effects of earthquakes has significantly improved, meaning I

am able to teach students about the underlying science behind liquefaction, landslides,

flooding, and structural and non-structural damages/building collapse due to seismic

waves

A post-test score indicates the extent to which teachers feel the program had an impact on their ability to teach about the effects of earthquakes.

3. As a result of your participation in the program, how do you feel your ability to teachabout earthquake hazards and safety precautions has changed?

5 = My knowledge of earthquake hazards and safety precautions has significantlyimproved, meaning I am able to teach students about risks to buildings and non-structural

risks, in addition to emergency plans including evacuation routes and drills

A post-test score indicates the extent to which teachers feel the program had an impact

on their ability to teach about earthquake hazards and related safety precautions

Section Two

Risk Perception (Pre-Test)






The pre-test result indicates general awareness of earthquake occurring.

2. How important do you think it is for your school to prepare to protect teachers andstudents against earthquakes?

5 = it is very important that my school takes action to prepare and protect against

earthquakes

A pre-test result indicates awareness of the importance that a school is prepared.

3. How confident are you that teachers in your school are able to lead students to safety inthe event of an earthquake?

5 = teachers at my school know how to support students in the event of an earthquake

happening while at school, i.e. evacuation route, etc.

A pre-test result indicates confidence in teacher’s ability to evacuate and support students when an earthquake happens

4. If an earthquake was to occur tomorrow, how well prepared do you feel your school is to

cope with immediate damages that may arise?

5 = the school is well prepared to cope with the immediate consequences of anearthquake, i.e. emergency education plans, structural damage, contingencies

A pre-test result indicates perception that a school is well prepared to deal with

earthquake consequences and continued operation.

Section Two

Risk Perception (Post-Test)


A post-test increase in expectation of an earthquake occurring indicates increased awareness of the likelihood of an earthquake.

2. How important do you think it is for your school to prepare to protect teachers andstudents against earthquakes?

5 = it is very important that my school takes action to prepare and protect againstearthquakes

A post-test increase in this measure indicates an increased perception of the importance

of schools planning for earthquakes.




3. As a result of your participation in the program, do you feel that teachers in your schoolare better prepared to lead students to safety in the event of an earthquake?

The post-test response indicates the participant perception of improvements in school

emergency evacuation and planning procedures as a result of the program.

4. As a result of your participation in the program, do you feel your school is to better prepared to cope with immediate damages that may arise from an earthquake?

The post-test response indicates the participant perception of improvements in school

resilience – such as structural and non-structural changes – as a result of the program.

Section Three

Structural and Non-Structural Safety (Pre- and Post-Test)

1. Which of the following vulnerabilities can you identify in buildings in your community?If none are relevant, please leave empty (check all that apply).

A post-test increase in the vulnerabilities identified indicates a greater awareness of vulnerabilities within the community. This is a proxy measure that may result in, at a

later stage, changes or modifications to buildings that do increase preparedness.

2. Which of the following items in your school have been secured in case of an earthquake?If none are relevant, please leave empty (check all that apply).

A post-test increase in the number of items secured in case of an earthquake indicates

greater preparedness for hazards caused by non-structural items during an earthquake.

3. Which of the following steps have been taken to strengthen the structure of your schoolin the last year? If none are relevant, please leave empty (check all that apply).

A post-test increase in the steps taken to secure the structure of a home indicates greater

preparedness for structural hazards caused during an earthquake.

Section Four

Emergency Preparedness and Planning (Pre- and Post-Test)



2. How often does your school practice its emergency response and evacuation plan? For

example, a drill?

A post-test increase in frequency indicates greater preparedness.




3. Which of the following items do you have stored in your classroom case of anearthquake? If none are relevant, please leave empty (check all that apply).

A post-test increase in checked items indicates greater preparedness.

4. Has your school identified a staff member(s) responsible for emergency planning and

preparedness?


5. Does your school have a procedure in place for contacting parents in the event of anearthquake?

A post-test increase in those responding ‘yes’ indicates greater preparedness




Appendix H. Community Survey

Community SurveyPlease help us by completing this short survey. Your participation in this survey will help Teachers

Without Borders to understand how much the earthquake science program may help prepare your school

[your community] in the event of an earthquake.

1. Are you aware of the earthquake science and safety education program taking place in your

school [schools in your community] (Global Quake Science and Safety Initiative for Girls)?

□ Yes □ No

2. If yes, what do you know about it? If you have experienced any benefits of the program, please

describe them below.

___________________________________________________________________________________

___________________________________________________________________________________

___________________________________________________________________________________

Return To: ______________________

______________________

______________________

______________________

Date post:	14-Apr-2018
Category:	Documents
Upload:	teachers-without-borders
View:	216 times
Download:	0 times

Girls Earthquake Science and Safety Initiative Evaluation Framework

Documents