| Date post: | 20-Mar-2023 |
| Category: |
Documents |
| Upload: | khangminh22 |
| View: | 0 times |
| Download: | 0 times |
Enhancing Evaluations of Complex Food Security Programs Operating
Under Climate Change
by
Steven Lam
A Thesis
presented to
The University of Guelph
In partial fulfilment of requirements for the degree of
Doctor of Philosophy
in
Population Medicine
in the field of Public Health
Guelph, Ontario, Canada
© Steven Lam, April, 2022
ABSTRACT
ENHANCING EVALUATIONS OF COMPLEX FOOD SECURITY PROGRAMS OPERATING UNDER CLIMATE CHANGE
Steven Lam
University of Guelph, 2022
Advisor(s):
Dr. Warren Dodd
Dr. Sherilee L. Harper
Dr. Kelly Skinner
Dr. Andrew Papadopoulos
Complex, multicomponent programs are increasingly required to address food security
challenges, especially in the context of the climate crisis. While program evaluation has the
potential to improve food security program design, minimal guidance exists on how to evaluate
complex programs. This dissertation research explored strategies for assessing complex food
security programs and opportunities for evaluation to examine climate change, drawing on
several knowledge syntheses and a qualitative study of an ongoing food safety program in
Vietnam called SafePORK. First, a review was conducted to systematically identify and examine
studies reporting on community-based food security monitoring. This review characterized
community-based food security monitoring programs globally and underscored the need to
understand how program evaluation tools can consider climate change. Then, another review
was conducted to systematically explore considerations of climate change within food-related
evaluation reports of United Nations agencies. This review showed that progress toward
mainstreaming climate action in evaluation is slow. Additionally, developing a program theory,
or a pathway showing how changes will occur in a particular context, was identified as a
promising strategy for considering the interactions of program components with climate
change. Next, a third review examined studies reporting on theory-driven food-related
evaluations to characterize how and why evaluations considered climate change. This review
highlighted knowledge gaps of how climate change influenced program outcome pathways or
how programs operated under climate change. Finally, a qualitative study of a food safety
program applying theory-driven evaluation explored the experiences of researchers and
program participants (n=30 interviews). Researchers found the process of developing the
program theory helped them to document and navigate through the adaptive nature of
complex programs. Program participants (slaughterhouse workers and pork retailers) shared
how they were adapting to the impacts of climate change, which was an important factor that
was overlooked within the design of the program. Taken together, this dissertation research
provides insights around how climate change could be better integrated into evaluation
processes.
iii
ACKNOWLEDGEMENTS
To my co-supervisors, Sherilee Harper and Warren Dodd: Sheri, thank you for suggesting I
pursue a PhD and helping to make this journey a reality. Warren, thank you for accepting the
co-supervisory challenge with enthusiasm that remained consistent throughout. I’m grateful for
the time and energy you both provided in pushing my thinking and writing.
To the members of advisory committee, Kelly Skinner and Andrew Papadopoulos: Kelly, I
appreciate the sage evaluation advice you shared at cafes and conferences. Andy, I’m grateful
for our conversations about almost everything. To the both of you, thank you for helping me to
succeed both within and outside of academia by connecting me with paid side projects.
To the Arrell Food Institute, thank you for the opportunity to join the 2017 inaugural cohort of
the Food From Thought research assistantship program. I’m grateful to have met and worked
with a stellar team of students across departments, especially Kathleen Johnson, Michelle
Thompson, and Cameron Fioret. Thank you all for your friendship, in dream life, and in real life!
To the Canadian Association for the Study of International Development, thank you for the
opportunity to participate in the 2017 Doctoral School. Through this experience, I connected
with some amazing peers, including Gloria Novovic, Jeremy John Escobar Torio, and Judyannet
Muchiri. “Thank you SO much” friends for the rants, laughs, and solidarity.
To the Harper labmates, moving to Edmonton in 2018 would not have been as fun without you
Crystal Gong, Jen Jones, Andrea Garcia, Amreen Babujee, Carlee Wright, Jacqui Middleton, and
iv
Kaitlin Patterson. Thank you for the “treat-yo-self Thursdays” and hangouts. Crystal: thanks for
always taking what I said with humour. Jen: your perseverance is contagious, and I might have
caught some of it, so thank you. Andrea: thank you for climbing with me. And thank you to the
wider lab, especially Amy Kipp and Chloe Zivot, for your support on different parts of this work.
To the Dodd labmates, I’m happy to have been in your company when I moved to Waterloo in
2019. Thank you Vinussa Rameshshanker, Chantelle Ramsundar, Emily Kocsis, Josalyn Radcliffe,
Monica Bustos, and Shoshannah Speers for the important conversations around global health.
Thank you especially to Sara Wyngaarden for being such a lovely friend.
To the Ontario Veterinary College, especially everyone at Shared Administrative Services for all
that you do. To my classmate, Aarabhi Rajendiran, thanks for being one of the few people I can
talk to for hours on ends about evaluation and non-evaluation matters. And to Abhinand
Thaivalappil, thanks for consistently having my back, from elementary school to grad school.
To the Canadian Evaluation Society, thank you for your conferences, student awards, and case
competitions, which enriched my PhD experience. Thank you to the evaluators and clients who
worked with me on interesting side projects, providing me with a much-welcomed mental break
from my dissertation alongside professional experience to further my evaluation journey.
To the Indigenous Health and Adaptation to Climate Change team, thank you for your support
during the first couple of years of my PhD. I appreciated the opportunity to work with and learn
from Lea Berrang-Ford, James Ford, and the rest of the team.
v
To the SafePORK team, thank you for your support during the latter half of my PhD. It’s always
wonderful working with you Hung Nguyen-Viet and Fred Unger. Thank you for being so
hospitable and welcoming. To Sinh Dang-Xuan and Trang Le, I’m very grateful for your research
assistance. I’m also happy we stayed connected since the days at the Center for Public Health
and Ecosystem Research. And thank you to the participants of SafePORK for sharing their time,
perspectives, and experiences with us.
Thank you friends for checking-in on me. Special thanks to Chloe Doan for telling me to “dream
bigger, boy.” Troy Stanton, thank you for making me ponder why I want a PhD and other big life
questions. Jennifer Bach, thank you for always being there for me wherever you happen to be in
the world. Huong Nguyen-Mai, thank you chị for always asking me about my Vietnamese and
helping to keep it in shape. Ly Nguyen, thank you for being that kind friend I aspire to be.
To my brothers Kevin and David, thank you for always supporting me with all my life choices. To
my mom, whose sacrifice before and after the war years made possible my life and those of my
brothers. Thank you mẹ for sending me home with food and plants I didn’t realize I needed.
Thank you to the rest of my family in Canada and Vietnam for always welcoming me into your
thoughts and homes. Lastly, thank you Dan, for always managing to make me laugh and keep
me full. Well done, my love.
vi
STATEMENT OF WORK
I wrote research chapters (Chapters 2-5) in collaboration with my advisory committee and other
colleagues. Under the supervision of Drs. Sherilee Harper and Warren Dodd, I wrote Chapter one
(introduction), Chapter six (applied chapter), and Chapter seven (conclusion). The following
scholarships funded my research: the Graduate Student Excellence Entrance Scholarship (2017),
the Ontario Veterinary College PhD Scholarship (2017-2021), the SSHRC General Graduate
Student Funding Travel Scholarship (2018), the CIHR Frederick Banting and Charles Best Canada
Graduate Scholarship (2019-2022), and the Canadian Evaluation Society Educational Fund
Graduate Scholarship (2020).
Chapter two | Community-based monitoring of Indigenous food security in a changing
climate: Global trends and future directions
Published article:
Lam S, Dodd W, Skinner K, Papadopoulos A, Zivot C, Ford J, Garcia P, IHACC Research Team,
Harper SL. (2019). Community-based monitoring of Indigenous food security in a changing
climate: Global trends and future directions. Environmental Research Letters. 14(7): 073002.
I led all stages of the review, including: developing the research question; designing the search
strategy; extracting, analyzing, and interpreting the data; and writing the paper. WD and SLH
supervised and contributed to all stages of the review. KS and AP contributed to
conceptualization and data interpretation. CZ contributed as a secondary reviewer and to the
vii
design of the screening tool. The IHACC team contributed to conceptualization. All co-authors
contributed to manuscript editing. In 2018, I received helpful feedback on this work from the
Ecohealth conference and Ontario Veterinary Colleague Graduate Research Symposium.
Chapter three | How are climate actions evaluated? A review of United Nations food
security evaluations
Published article:
Lam S, Dodd W, Berrang-Ford L, Ford J, Skinner K, Papadopoulos A, Harper SL. (2021). How are
climate actions evaluated? A review of United Nations food security evaluations. Global Food
Security. 28: 100509.
I led all stages of the review including: developing the research question; designing the search
strategy; extracting, analyzing, and interpreting the data; and writing the paper. Amreen Babujee
provided support as a secondary reviewer. WD and SLH supervised and contributed to all stages
of the review. LBF, JF, KS, and AP contributed to conceptualization and data interpretation. All
co-authors contributed to manuscript editing. In 2019, I received helpful feedback on this work
from the Canadian Evaluation Society conference.
Chapter four | How and why are Theory of Change and Realist Evaluation used in food
security contexts? A scoping review
Published article:
viii
Lam S, Dodd W, Wyngaarden S, Skinner K, Papadopoulos A, Harper SL. (2021). How are Theory of
Change and Realist Evaluation used in food security contexts? A scoping review. Evaluation and
Program Planning. 89: 102008.
I led all stages of the review including: developing the research question; designing the search
strategy; extracting, analyzing, and interpreting the data; and writing the paper. WD and SLH
supervised and contributed to all stages of the review. SW contributed as a secondary reviewer
and to the design of the screening tool. KS and AP contributed to conceptualization and data
interpretation. All co-authors contributed to manuscript editing.
Chapter five | Planning and evaluating a complex food safety program operating under a
changing climate: Experiences from Vietnam
Article in preparation for peer-review:
Lam S, Dodd W, Nguyen-Viet H, Unger F, Le TTH, Dang-Xuan S, Skinner K, Papadopoulos A,
Harper SL. “We have a lot of things to do:” On planning and evaluating a complex food safety
program operating under a changing climate.
I co-developed this research project with coordinators of the SafePORK program in Vietnam
(HN-V, FU). I conducted remote interviews with researchers while TL and S-DX conducted
remote interviews with program participants. To ensure data quality, I held de-brief sessions
with TL and S-DX throughout the interview process. Additionally, I conducted data checking (e.g.
ix
explored consistency between transcriptions and recordings). I wrote the manuscript under the
supervision of WD, SLH, HN-V, and FU. All co-authors contributed to manuscript editing.
Chapter six | How the world will measure progress on the Paris climate agreement and
keep countries accountable
Published commentary:
Lam S, Dodd W, Harper SL. (2021). COP26: How the world will measure progress on the Paris
climate agreement and keep countries accountable. The Conversation. Canada.
I wrote this piece to position the importance of evaluation in keeping countries accountable for
their climate action commitments. WD and SLH supported the conceptualization and editing of
the piece. Hannah Hoag, an editor at The Conversation Canada, provided helpful feedback and
edits.
x
TABLE OF CONTENTS
ABSTRACT .......................................................................................................................................................................... ii
ACKNOWLEDGEMENTS ............................................................................................................................................... iii
STATEMENT OF WORK ................................................................................................................................................ vi
TABLE OF CONTENTS .................................................................................................................................................... x
LIST OF TABLES ............................................................................................................................................................. xvi
LIST OF FIGURES .......................................................................................................................................................... xvii
LIST OF ABBREVIATIONS ........................................................................................................................................... xix
LIST OF APPENDICES ................................................................................................................................................... xx
POSITIONALITY STATEMENT .................................................................................................................................. xxii
The research topic ................................................................................................................................................. xxii
The research participants ................................................................................................................................... xxiv
The research context and process ................................................................................................................... xxv
My responsibility ................................................................................................................................................... xxvi
References.............................................................................................................................................................. xxviii
CHAPTER 1: INTRODUCTION .................................................................................................................................... 1
2.1 Introducing the work ................................................................................................................................... 1
2.2 Conceptualizing food security ................................................................................................................. 4
2.3 Measuring food security ............................................................................................................................ 6
2.4 Food security programs .......................................................................................................................... 10
2.5 Program monitoring and evaluation .................................................................................................. 11
xi
2.6 Dissertation objectives ............................................................................................................................. 17
2.7 Why does this dissertation matter today? ....................................................................................... 18
2.8 Methodological framing ......................................................................................................................... 20
2.8.1 Knowledge synthesis ....................................................................................................................... 20
2.8.2 Qualitative research ......................................................................................................................... 22
2.8.3 Combining knowledge synthesis and qualitative research .............................................. 27
2.9 Dissertation structure ............................................................................................................................... 28
2.10 Summary ....................................................................................................................................................... 29
2.11 References .................................................................................................................................................... 31
CHAPTER 2: COMMUNITY-BASED MONITORING OF INDIGENOUS FOOD SECURITY IN A
CHANGING CLIMATE: GLOBAL TRENDS AND FUTURE DIRECTIONS ....................................................... 43
Abstract ............................................................................................................................................................................ 43
2.12 Introduction ................................................................................................................................................. 44
2.13 Methods ........................................................................................................................................................ 48
2.13.1 Search strategy ............................................................................................................................. 48
2.13.2 Citation management ................................................................................................................. 49
2.13.3 Relevance screening and eligibility ....................................................................................... 49
2.13.4 Data extraction and synthesis ................................................................................................. 49
2.14 Results ............................................................................................................................................................ 53
2.14.1 Overview of relevant articles .................................................................................................... 53
2.14.2 Diversity of approaches to community-based monitoring .......................................... 54
2.14.3 Most studies were published in the past decade ............................................................ 55
2.14.4 Wildlife was a prominent type of food monitored.......................................................... 56
xii
2.14.5 More articles focused on men’s participation in monitoring ...................................... 57
2.14.6 Climate change indicators were rarely assessed .............................................................. 57
2.14.7 Varying degrees of Indigenous community engagement in monitoring ............... 59
2.15 Discussion ..................................................................................................................................................... 61
2.16 Conclusion .................................................................................................................................................... 71
2.17 References .................................................................................................................................................... 73
CHAPTER 3: HOW ARE CLIMATE ACTIONS EVALUATED? A REVIEW OF UNITED NATIONS FOOD
SECURITY EVALUATIONS .......................................................................................................................................... 85
Abstract ....................................................................................................................................................................... 85
3.1 Introduction ................................................................................................................................................. 86
3.2 Methods ........................................................................................................................................................ 89
3.2.1 Search strategy .................................................................................................................................. 89
3.2.2 Relevance screening ........................................................................................................................ 90
3.2.3 Data extraction, classification, and analysis ............................................................................ 91
3.2.4 Thematic analysis: examining evaluation challenges and opportunities..................... 96
3.3 Results ............................................................................................................................................................ 97
3.3.1 Climate action mainstreaming into food security evaluation ........................................ 100
3.3.2 Climate considerations in food security evaluation was superficial ............................ 103
3.3.3 High scoring food security evaluations integrated climate considerations into the
evaluation scope, approach, and results ................................................................................................. 104
3.3.4 Low scoring food security evaluations integrated climate considerations into the
evaluation results only ................................................................................................................................... 106
3.3.5 Challenges and opportunities for evaluating climate action ......................................... 107
3.4 Discussion ................................................................................................................................................... 114
xiii
3.4.1 Implications for research and practice ................................................................................... 116
3.4.2 Limitations ......................................................................................................................................... 118
3.5 Conclusion .................................................................................................................................................. 118
3.6 References .................................................................................................................................................. 120
CHAPTER 4: HOW AND WHY ARE THEORY OF CHANGE AND REALIST EVALUATION USED IN
FOOD SECURITY CONTEXTS? A SCOPING REVIEW ...................................................................................... 127
Abstract ..................................................................................................................................................................... 127
4.1 Introduction ............................................................................................................................................... 128
4.2 Methods ...................................................................................................................................................... 130
4.2.1 Search strategy ................................................................................................................................ 130
4.2.2 Relevance screening and eligibility .......................................................................................... 131
4.2.3 Data extraction and synthesis .................................................................................................... 132
4.3 Results .......................................................................................................................................................... 134
4.3.1 Characteristics of studies ............................................................................................................. 134
4.3.2 ToCs and RE were developed primarily using qualitative approaches ...................... 137
4.3.3 ToCs served a variety of purposes whereas RE supported evaluation purposes ... 138
4.3.4 The use of multiple theory-driven evaluation approaches together .......................... 139
4.3.5 Considerations of gender, context, and climate change ................................................. 140
4.3.6 Problem representations offered by studies ........................................................................ 143
4.3.7 Responses to problem framings ............................................................................................... 145
4.4 Discussion ................................................................................................................................................... 149
4.4.1 Strengths and limitations ............................................................................................................ 153
4.5 Conclusion .................................................................................................................................................. 153
xiv
4.5.1 Lessons learned ............................................................................................................................... 154
4.6 References .................................................................................................................................................. 156
CHAPTER 5: “WE HAVE A LOT OF THINGS TO DO:” ON PLANNING AND EVALUATING A
COMPLEX FOOD SAFETY PROGRAM OPERATING UNDER A CHANGING CLIMATE ........................ 163
Abstract ..................................................................................................................................................................... 163
5.1 Introduction ............................................................................................................................................... 164
5.1.1 Complex food safety programs ................................................................................................. 165
5.1.2 Context: Food safety in Vietnam ............................................................................................... 166
5.1.3 The SafePORK program ................................................................................................................ 168
5.1.4 SafePORK locations ........................................................................................................................ 170
5.2 Methods ...................................................................................................................................................... 172
5.2.1 Data collection and analysis ....................................................................................................... 172
5.2.2 Data validity and management ................................................................................................. 173
5.2.3 A note on COVID-19 and ethics ................................................................................................ 173
5.3 Results .......................................................................................................................................................... 174
5.3.1 “We imagine how it can work in practice:” Using program theory to track changes
to the program .................................................................................................................................................. 175
5.3.2 “When it rains, you have to cover the pork:” Attuning to climate change ............... 178
5.3.3 “We always talk about gender in SafePORK:” Attuning to gender equity ................ 182
5.4 Discussion ................................................................................................................................................... 187
5.4.1 Limitations ......................................................................................................................................... 191
5.5 Conclusion .................................................................................................................................................. 192
5.6 References .................................................................................................................................................. 193
xv
CHAPTER 6: HOW THE WORLD WILL MEASURE PROGRESS ON THE PARIS CLIMATE
AGREEMENT AND KEEP COUNTRIES ACCOUNTABLE ................................................................................. 199
6.1 Where are we? .......................................................................................................................................... 200
6.2 How are we doing? ................................................................................................................................. 201
6.3 Collecting and preparing information ............................................................................................. 204
CHAPTER 7: CONCLUSION .................................................................................................................................... 206
7.1 Reflecting on the journey ..................................................................................................................... 206
7.2 Synthesis of findings ............................................................................................................................... 209
7.2.1 Exploring program theory in food security program evaluation.................................. 210
7.2.2 Considering gender and other social identities in evaluation processes .................. 211
7.2.3 Integrating climate change into all stages of an evaluation .......................................... 212
7.3 Strengths and limitations ...................................................................................................................... 214
7.4 Suggestions for future research ......................................................................................................... 216
7.5 Concluding thoughts .............................................................................................................................. 217
7.6 References .................................................................................................................................................. 220
Appendices ................................................................................................................................................................... 224
xvi
LIST OF TABLES
Table 1.1. Food security metrics and their scale, strengths, and weaknesses. ......................................... 8
Table 2.1. Search strategy to identify published articles on Indigenous community-based
monitoring of Indigenous food security. ............................................................................................................ 51
Table 2.2. Inclusion and exclusion criteria to identify published articles on community-based
monitoring of Indigenous food security and climate change..................................................................... 52
Table 2.3. Summary of information extracted from articles on community-based monitoring of
Indigenous food security. ......................................................................................................................................... 53
Table 2.4. Types of community-based monitoring approaches described in articles that reported
the implementation of a monitoring system ..................................................................................................... 61
Table 3.1. Food security and climate change classification system applied to United Nations
evaluation reports. ....................................................................................................................................................... 93
Table 3.2. Assessment rubric for climate action integration into United Nations evaluation
reports. ............................................................................................................................................................................. 96
Table 3.3. Descriptive characteristics of United Nations food security evaluations (n = 136
evaluations) published from 2014 to 2019......................................................................................................... 99
Table 4.1. Criteria to select food-related Theory of Change and Realist Evaluation studies. ........ 132
Table 4.2. Characteristics of food-related Theory of Change and Realist Evaluation studies. ...... 136
xvii
LIST OF FIGURES
Figure 1.1. Visual representation of dissertation research including two chapters on how climate
change is considered in food security monitoring and evaluation. .......................................................... 29
Figure 2.1. Flow chart of the selection of studies that explored community-based monitoring of
Indigenous food security. ......................................................................................................................................... 54
Figure 2.2. Geographic distribution of articles on community-based monitoring of Indigenous
food security. ................................................................................................................................................................. 56
Figure 2.3. Articles on community-based monitoring of Indigenous food security with and
without climate change considerations, over time. ........................................................................................ 58
Figure 3.1. Flow chart illustrating the selection of United Nations food security evaluations (n =
136 unique evaluations within 133 reports) published from 2014 to 2019. .......................................... 98
Figure 3.2. Number of United Nations food security evaluations (n = 136) with a climate focus
published from 2014 to 2019. ............................................................................................................................... 101
Figure 3.3. Proportion of United Nations food security evaluations from 2014 to 2019 focused on
implicit adaptation (a), explicit adaptation (b), mitigation (c), both adaptation and mitigation (d),
and climate change impacts (e). ........................................................................................................................... 102
Figure 3.4. Extent of climate action integration into United Nations food security evaluations
from 2014 to 2019. .................................................................................................................................................... 104
Figure 3.5. Challenges and opportunities for assessing climate action as reported in United
Nations food security evaluations from 2014 to 2019................................................................................. 108
Figure 4.1. Flow chart showing the selection of food-related Theory of Change and Realist
Evaluation studies. ..................................................................................................................................................... 135
Figure 4.2. Food topics of reviewed Theory of Change and Realist Evaluation studies. ................. 137
Figure 4.3. Considerations of context, gender, and climate change in food-related Theory of
Change and Realist Evaluation studies. ............................................................................................................. 142
Figure 4.4. Concept map showing the relationship between problem framings, responses to
problem framings, and conditions under which responses occur in theory-driven evaluations. 146
Figure 5.1. SafePORK program objectives and their interrelationships. ................................................ 170
xviii
Figure 5.2. Map showing locations of study sites (Hung Yen, Nghe An, Hoa Binh provinces) and
the capital city of Vietnam (Hanoi). ..................................................................................................................... 171
Figure 5.3. Timeline of implementation, key milestones, and barriers of SafePORK. ....................... 178
Figure 7.1. Visual representation of key findings from this dissertation. .............................................. 209
xix
LIST OF ABBREVIATIONS
ASF African Swine Fever
CBM Community-Based Monitoring
CFIR Consolidated Framework for Implementation Research
IHACC Indigenous Health Adaptation to Climate Change
FBD Foodborne Disease
LMIC Low- and Middle-Income Country
M&E Monitoring and Evaluation
RE Realist Evaluation
SDG Sustainable Development Goals
ToC Theory of Change
UN United Nations
UNEG United Nations Evaluation Group
xx
LIST OF APPENDICES
Appendix 1: Search strategy to identify published articles on Indigenous community-based
monitoring of Indigenous food security and climate change in AGRICOLA©, PRISMA©,
MEDLINE®, CabDirect©, and the Web of Science™ CORE Collection databases................................. 224
Appendix 2: Complete list of included articles and select article characteristics on Indigenous
community-based monitoring of food security and climate change. ................................................... 228
Appendix 3: Complete list of references. .......................................................................................................... 238
Appendix 4: Questions to guide deductive qualitative analysis conducted on United Nations
food security evaluation reports published from 2014 to 2019. .............................................................. 246
Appendix 5. Codebook of climate-responsive food security evaluation challenges. ...................... 247
Appendix 6. Codebook of climate-responsive food security evaluation solutions........................... 250
Appendix 7: Thematic map of challenges & opportunities for climate-responsive evaluation. .. 253
Appendix 8: List of all included United Nations food security evaluation reports published from
2014 to 2019. ............................................................................................................................................................... 254
Appendix 9: Performance of United Nations food security evaluations published from 2014 to
2019 (n = 136) according to components of the climate assessment rubric. ..................................... 261
Appendix 10: Climate change mandates of United Nations agencies. .................................................. 262
Appendix 11: Comparison between average climate action integration score of United Nations
evaluations and average program duration. ................................................................................................... 263
Appendix 12: Comparison in climate-related reporting between high and low scoring climate
action integration of United Nations evaluations.......................................................................................... 264
Appendix 13: Search strategy used to identify Theory of Change and Realist Evaluation studies
from AGRICOLA©, CabDirect©, Web of Science™ CORE Collection, Medline®, Scopus®, EconLit©,
and Google™. ................................................................................................................................................................ 265
Appendix 14: Summary of the screening tools used in the title and abstract screening stage and
full text screening stage to identify relevant Theory of Change and realist evaluation studies. . 266
xxi
Appendix 15: The data charting form used to extract relevant information from Theory of
Change and Realist Evaluation studies. ............................................................................................................. 268
Appendix 16: List of references of all included studies on food-related Theory of Change and
Realist Evaluation. ...................................................................................................................................................... 272
Appendix 17: Coding process to analyze Theory of Change (ToC) and Realist Evaluation (RE)
studies. ........................................................................................................................................................................... 276
Appendix 18. Number of publications focusing on food-related Theory of Change and Realist
Evaluation studies (note: review cut off 10 March 2020). ........................................................................... 277
Appendix 19: Codebook to identify problems as defined by Theory of Change and Realist
Evaluation studies (2nd level). ................................................................................................................................. 278
Appendix 20: Codebook to identify how Theory of Change and Realist Evaluation studies are
responding to problems (3rd level). ..................................................................................................................... 280
Appendix 21: Interview guide and Consolidated Framework for Implementation Research
constructs used to explore the implementation and evaluation of the SafePORK program. ....... 283
Appendix 22: Audit trail of decisions made and rationales behind them. ........................................... 284
Appendix 23: Impacts of COVID-19 on data collection. .............................................................................. 286
Appendix 24: Presentation material for the SafePORK team. ................................................................... 288
Appendix 25: Characteristics of program participant interviewees. ....................................................... 297
Appendix 26: Codebook to identify overarching themes from SafePORK implementation. ........ 298
xxii
POSITIONALITY STATEMENT
Our social identities affect the way we see and interpret the world around us, and how the world
sees and interprets us, both of which influence the research process (Jacobson & Mustafa, 2019).
As the main research instrument, I have a responsibility to consider how my identities,
experiences, and motivations have shaped this work (Hammarberg et al., 2016; Pitard, 2017).
One way of attuning to positionality is to reflect on three key areas: (1) the research topic, (2) the
research participants, and (3) the research context and process (Holmes, 2020).
The research topic
My interest in program evaluation was shaped by my experiences as an accidental evaluator.
Toward the end of a research internship in Hanoi in 2015, I was asked by colleagues to support
an evaluation of an Ecohealth program. Gaining this experience created possibilities for me to
engage in the evaluation discipline, as I became increasingly involved in evaluations of programs
aiming to improve environmental health, rural development, and food security. To me, one of
the biggest privileges of this discipline is connecting with people who generously share their
thoughts and experiences about how programs can better respond to their needs and priorities.
My dissertation topic came from a place of curiosity developed from working in evaluation and
international development. Working alongside fellow evaluators, I found we would often default
to using traditional approaches to evaluation (e.g. assessing the ability of programs to achieve
their intended outcomes) regardless of the evaluation context. This inclination was influenced by
xxiii
our evaluation training, our comfort in using these tools, and our evaluation client’s suggestions.
I wondered whether these approaches were truly able to support learning, especially in food
security and other development contexts where programs often evolve and intended outcomes
could not be predicted in advance. Moreover, the climate crisis we are experiencing today
increasingly affects programs and measures used to evaluate their impacts. I undertook this PhD
research to better understand and strengthen alternative approaches to evaluation for
supporting complex programs and programs facing external influences such as climate change.
The goal was always pragmatic: to contribute to research on evaluation, and through this,
inform evaluation practice.
When planning an evaluation, I typically look for examples conducted in similar contexts for
inspiration. I especially draw on evaluation reports, which are a useful, timely, and actionable
source of information. Although important for informing decisions, many evaluation reports end
up sitting on shelves gathering dust (Leviton & Melichar, 2016). One way of making evidence
from evaluations more readily available is to consolidate what is learned from evaluations. To
address this research gap, I dedicated time and energy synthesizing data across evaluations to
identify common themes relating to food security program evaluation. The findings provided a
stage for further exploration in subsequent chapters.
Alongside my PhD studies, I work as an independent evaluation consultant. One of the
programs I have been supporting is a food safety program in Vietnam called SafePORK. Before
xxiv
entering my PhD, I had also supported evaluation activities of the previous phase of the
program called PigRISK. Working on these programs contributed to my interest in learning more
about strategies for monitoring and evaluating food security programs in low-resource settings.
Importantly, this connection I had to the SafePORK team provided an opportunity for me to
explore the SafePORK program as a case study in my dissertation.
My social identity as a Canadian-Vietnamese man with historical and cultural connections to
Vietnam also played a role in this research process. My parents were third-wave refugees who
had escaped Vietnam’s oppressive post-war climate in the mid-1980s. I was born and raised in
Canada and nourished by Vietnamese food and stories; if not for these, I might not have
pursued work and studies in Vietnam. The research topic of pork safety also became a personal
interest after I shared what I was working on with my mom. She reflected on the early 1970s
when she and grandma raised pigs on the side of their home as an additional source of income.
Listening to the challenges and opportunities they faced served as a reminder to prioritize
community voices in the findings and recommendations made toward pork safety.
The research participants
Several aspects of my identity gave me access to research participants (i.e. SafePORK researchers
and program participants) likely not afforded to other researchers, including my appearance
(Việt Kiều, Vietnamese overseas), experience living in Vietnam (from 2015 to 2017), and
proficiency with Vietnamese (fluent). I have also built relationships, rapport, and trust with many
xxv
of the same research participants. In these ways, I might be considered an ‘insider’, referring to
an individual who researches with a population they are a member of (Holmes, 2020). This
insider status allowed for more rapid acceptance by research participants. As several SafePORK
researchers told me, I was after all a SafePORK team member, and sharing details with someone
who ‘understands’ was not considered to be an issue. In other ways, I am an outsider, as a
colleague once shared with me “you look Vietnamese”, explaining that because I didn’t grow up
in Vietnam, I am not a real Vietnamese. I viewed this perception of me as beneficial for this
research as it led many participants to elaborate on the context of Vietnam, making often
implicit assumptions explicit. Additionally, the social distance of an outsider may have led some
interviews to be more conversational. At times, SafePORK researchers asked me questions, such
as my plans after the research was finished. Because of my insider-outsider status, and the
power differences between researcher and participant that come with this status, I self-reflected
throughout the dissertation on how to try to produce non-hierarchical research relationships.
Being attentive to these power differences is important for avoiding exploitative research.
The research context and process
My approach to working responsibly on research projects in my dissertation was two-fold. First, I
joined larger community-based research projects to help ensure my research and evaluation
interests aligned with community needs and priorities. For example, the idea of reviewing
community-based monitoring systems for Indigenous food security in Chapter 2 originated by
the IHACC team. I collaborated with my co-supervisors who are part of the Indigenous Health
xxvi
Adaptation to Climate Change (IHACC) team to further develop this idea. For Chapter 5, I
worked closely with the SafePORK coordinators to design a project that meets both dissertation
and SafePORK program learning goals. We decided to explore barriers and facilitators to
multidisciplinary food safety research and practice, drawing on the experiences of SafePORK.
Secondly, I employed strategies to enhance quality and reduce potential bias in the research
process. For example, I co-conducted different aspects of the research process where possible.
For my review-based research chapters, I conducted the search and screening of articles with a
second reviewer. For all research chapters, I conducted analysis and interpretation closely with
my PhD committee. When working within larger research teams, I aimed to deeply listen and
accommodate shifting research priorities. Finally, I strived to continuously reflect on my
positionality with the research topic, participants, and context, which helped me to engage with
the research process in a more meaningful way and undertake ethical research.
My responsibility
Universities have extractive tendencies whereby researchers collect data, analyze findings, and
communicate their results to their fellow academics, without meaningfully engaging with
research participants (Kolinjivadi et al., 2020). Pushing back against these colonial and extractive
models is challenging in academic environments that prioritize efficiency, productivity, and
quantifiable outputs. As a doctoral student, I participated in and benefited from this system by
conducting research and advancing my career. With these positionalities and privileges come
xxvii
responsibilities as a researcher. Firstly, I will continue building mutually beneficial research
collaborations, which could help to ensure the benefits from research are experienced by those
involved (Israel et al., 1998). Secondly, I will continue reflecting on power relations inherent in
the research in order to promote shared control of the research process (Råheim et al., 2016).
Thirdly, I will continue prioritizing community voices in the research process and findings. Doing
so could increase the relevance of the research and improve the usefulness of development
programs and policies for communities (Stein & Mankowski, 2004). Finally, I will use my
privileged position as an evaluator to advocate for evaluation metrics that prioritize research
use, quality, process, and outputs beyond the traditional academic paper (Jessani et al., 2020).
xxviii
References
Darwin Holmes, A. G. (2020). Researcher Positionality - A Consideration of Its Influence and Place
in Qualitative Research - A New Researcher Guide. Shanlax International Journal of
Education. https://doi.org/10.34293/education.v8i4.3232
Hammarberg, K., Kirkman, M., & De Lacey, S. (2016). Qualitative research methods: When to use
them and how to judge them. In Human Reproduction.
https://doi.org/10.1093/humrep/dev334
Israel, B. A., Schulz, A. J., Parker, E. A., & Becker, A. B. (1998). Review of community-based
research: Assessing partnership approaches to improve public health. In Annual Review of
Public Health. https://doi.org/10.1146/annurev.publhealth.19.1.173
Jacobson, D., & Mustafa, N. (2019). Social Identity Map: A Reflexivity Tool for Practicing Explicit
Positionality in Critical Qualitative Research. International Journal of Qualitative Methods.
https://doi.org/10.1177/1609406919870075
Jessani, N. S., Valmeekanathan, A., Babcock, C. M., & Ling, B. (2020). Academic incentives for
enhancing faculty engagement with decision-makers—considerations and
recommendations from one School of Public Health. Humanities and Social Sciences
Communications. https://doi.org/10.1057/s41599-020-00629-1
Kolinjivadi, V., Hecken, G. V., Casolo, J., Abdulla, S., & Blomqvist, R. E. (2020). Towards a Non-
Extractive and Care-Driven Academia. http://www.developmentresearch.eu/?p=801
Leviton, L. C., & Melichar, L. (2016). Balancing stakeholder needs in the evaluation of healthcare
quality improvement. BMJ Quality and Safety. https://doi.org/10.1136/bmjqs-2015-004814
Pitard, J. (2017). A journey to the centre of self: Positioning the researcher in autoethnography.
Forum Qualitative Sozialforschung. https://doi.org/10.17169/fqs-18.3.2764
Råheim, M., Magnussen, L. H., Sekse, R. J. T., Lunde, Å., Jacobsen, T., & Blystad, A. (2016).
Researcher-researched relationship in qualitative research: Shifts in positions and
researcher vulnerability. International Journal of Qualitative Studies on Health and Well-
Being. https://doi.org/10.3402/qhw.v11.30996
Stein, C. H., & Mankowski, E. S. (2004). Asking, witnessing, interpreting, knowing: Conducting
qualitative research in community psychology. In American Journal of Community
Psychology. https://doi.org/10.1023/B:AJCP.0000014316.27091.e8
1
CHAPTER 1: INTRODUCTION
2.1 Introducing the work
As with other social determinants of health, access to food has profound impacts on people’s
health (Tarasuk et al., 2019). Individuals who are food insecure are more likely to experience
nutrient deficiencies and have poorer diets compared to people who are food secure (CFS,
2012). The impacts of food insecurity on health go beyond nutrition and diet; food insecurity is
associated with multiple chronic conditions, a higher risk of depression, and poor self-rated
mental and physical health (Leung et al., 2020). Income, housing, education, and other social
determinants of health may also interact with and exacerbate food insecurity. Climate change
also affects the social determinants of health through its impacts on, among others, access to
adequate food (Ragavan et al., 2020).
While the right to food is a fundamental human right (UN General Assembly, 1948), we live in a
world with worrying food security challenges, particularly in low- and middle-income countries
(Farrell et al., 2018). The most recent estimate showed nearly 10% of the global population were
undernourished (FAO et al., 2021). At the end of 2020, more than 88 million people were
experiencing acute food insecurity, up 20% a year earlier (United Nations, 2021b). Projections
revealed this troublesome trend will continue, with climate change and COVID-19 exacerbating
the situation further and reversing gains made toward food security (Zurayk, 2020; Dodd et al.,
2021; FAO et al., 2021). With nine years left to achieve the 2030 Sustainable Development Goals
2
(SDGs), the world is not on track to “end hunger, achieve food security, and improve nutrition”
(i.e. SDG2) (von Grebmer et al., 2020).
There is no simple explanation for why countries lack food security. Food security can be
disrupted by many factors including environmental degradation, climate change, public policies,
poverty, inequality, conflict, food price volatility, and animal and human diseases (Mbow et al.,
2019; FAO et al., 2021; UN, 2021). The causes of food insecurity are often political, economic,
and social. In the words of the United Nations Secretary-General, “Famine and hunger are no
longer about the lack of food. They are now largely [hu]man-made” (UN, 2021). There is more
than enough food for everyone, but current food systems guarantee inequality in access to
food. If we are to achieve SDG2, we need to understand what works to address food inequality.
Because food security is a result of a wide range of factors, programs tackling food security
challenges are often complex, meaning they have “multiple interacting components” (Moore et
al., 2014, p. 1). The Medical Research Council lists characteristics of complex programs, including,
but not limited to: the number of components involved; the number of groups, settings, or
levels targeted; and, the level of flexibility of the program (Skivington et al., 2021). Programs are
complex not only because of the properties of the program but also the context into which the
program is introduced. Context is the pre-existing conditions, or anything external to the
program, that may affect implementation and outcomes (Thirsk and Clark, 2017). For example,
climate change can be a considered a contextual factor given its influence on all aspects of food
3
security, and by extension, efforts aiming to improve food security (Mbow et al., 2019).
Complexity arises from the interaction of program components with context (Hawe, 2015).
Monitoring and evaluation frameworks play a key role in improving food security programs.
Monitoring refers to the ongoing process of tracking program activities and results whereas
evaluation assesses the extent to which program goals were obtained (UNEG, 2013; Wanzer,
2021). Standard approaches to evaluation – such as the assessment of program outcomes
against pre-determined targets – help assess food security programs that are relatively linear
(e.g. school feeding leads to improved nutrition). However, these approaches become less
effective for supporting complex food security programs, such as those targeting behavioural,
social, and/or policy change. Additionally, the complexity stemming from climate change and
other contextual factors pose further challenges in assessing food security programs.
The “complex” and “complexity” aspects of food security programs have not always been well
acknowledged, described, or considered in program design and evaluation (Hendriks, 2020;
Hendriks, 2015; Jones et al., 2013; Leroy et al., 2015). I argue that if monitoring and evaluation
are to identify and learn from effective food security strategies, we need to account for the
influence of these aspects on implementation and outcomes. In this introduction, I illustrate how
my research sought to expand scholarship on monitoring and evaluation of complex food
security programs. I begin by providing a brief review of how the concept of food security has
evolved. Next, I outline the characteristics of food security programs that make them complex
4
and how such programs can experience complexity. Finally, I provide an overview of the
challenges and opportunities to monitoring and evaluating complex food security programs,
especially under climate change.
2.2 Conceptualizing food security
Originally taken to mean adequate world food supplies of basic foodstuffs, the definition of
food security has been continuously reconceptualized to bring the concept in line with
contemporary issues of human development (Stephenson, 2020; United Nations, 1975). Nearly
two decades ago, Maxwell and Frankenberger (1992) documented 194 different studies on food
security conceptualizations. The definition of food security most widely used today and adopted
by the highest level of global governance on food security (the Committee on World Food
Security) is described as “exist[ing] when all people, at all times, have physical and economic
access to sufficient, safe and nutritious food to meet their dietary needs and food preferences
for an active healthy life” (CFS, 2012, p. 1). From this definition, food security encompasses four
key dimensions: availability (e.g. physical presence of food), access (e.g. economic access to
food), utilization (e.g. nutrition intake), and stability (e.g. assurance of the dimensions over time
in the face of shocks). Increasingly, there is a push toward food and nutrition security,
recognizing both must be addressed for improved health (Mozaffarian et al., 2021).
Critical scholars argued these recent re-articulations of food security draw attention to nothing;
in conceptualizing food security as an outcome, the underlying power relations that have
5
created food security conditions remain out of sight (Shepherd, 2012; Stephenson, 2020; Turner
et al., 2018). Attuning to shifting food security definitions is important because definitions can
influence decisions regarding policy responses. Because food security is a prerequisite for the
right to food, applying rights-based approaches – e.g. focusing on root causes such as matters
of state policy and discrimination – is a promising strategy for addressing often neglected power
relations (Ayala & Meier, 2017). Of note, terms such as ‘deserve’ and ‘fairness’ might be more
effective than ‘rights’ in engaging specific audiences in addressing the root causes of food
insecurity (Godrich et al., 2021).
The shift in the terminology used to talk about food security also reflects a growing
understanding of its multidimensional nature. For example, food security is increasingly linked to
climate change. In its Special Report on Climate Change and Land, the Intergovernmental Panel
on Climate Change indicated climate change has complex interactions with food systems,
leading to food insecurity through impacts on food availability, access, utilization, and stability
(Mbow et al., 2019). The definition of food security has also been expanded to include the fifth
dimension, ‘cultural acceptability’, which acknowledges how food production and consumption
are also strongly influenced by culture (Friedrichsen et al., 2018; Mbow et al., 2019; Power, 2008).
Despite these new understandings, many analyses of ‘food security’ still narrowly focus on
raising the food supply; adaptation to climate change, changing consumption practices, and
cultural acceptability have been treated as challenges that sit alongside the primary goal of
achieving food security as if they were an afterthought (IPES-Food, 2015).
6
2.3 Measuring food security
The changing understandings of food security definitions pose challenges in measuring food
security, which is important for understanding the true scale of food security. Measurements
may focus on food availability, access, utilization, stability, or some combination of these
dimensions (Jones et al., 2013). Data collection may take into account quantitative, qualitative,
psychological, and/or social constructs of the experience of food security (Hendriks, 2015).
Importantly, a single measure that captures all aspects encapsulated in the food security
concept does not exist; each measure both captures and neglects phenomena intrinsic to the
concept of food security (Barrett, 2010). However, there have been efforts to develop a single
measure (e.g. Urke et al., 2014). Furthermore, decisions made around what to measure are not
only technical but also political considering measurement determines the subsequent actions
that flow from findings (Haysom & Tawodzera, 2018).
There is no agreed approach to measuring food security. A 2013 review identified 18 different
food security assessment approaches for the different framings of food security (Jones et al.,
2013). Approaches to measuring food security can be broadly categorized as either direct or
indirect, both of which have their strengths and weaknesses (see Table 1.1). Direct measures are
experience-based and attempt to directly measure families’ lived experiences of food security
(Jones et al., 2013). Indirect measures focus on the specific indicator and use the data generated
as a proxy to determine food security status. Care needs to be taken in the selection of
measurement approaches to avoid measuring an unintended construct of food security,
7
measuring multiple constructs without the ability to differentiate them, and collecting
information irrelevant to those using the data (Haysom & Tawodzera, 2018; Jones et al., 2013).
There is general confusion regarding the scale of food security these measurement approaches
are meant to reflect. Food security can be measured at different levels, including individual,
household, community, national, and global. For example, the Food Insecurity Experience Scale
directly measures individuals’ or households’ experiences through the lens of food access (Leroy
et al., 2015). The Global Food Security Index is country-centered and considers food security
according to the national capacity to promote food affordability, availability, and quality/safety
(Allee et al., 2021). Measurement tools are not static and can adapt over time. For instance, the
2020 edition of the Global Food Security Index incorporated the “Natural Resources and
Resilience” category into the main index, which assesses a country’s exposure to the impacts of
a changing climate (Economist Impact, 2021). Measurements can also be aggregated to
determine food security at a higher level. For instance, household expenditure measures can be
taken at the household level and aggregated to determine food security at the national level
(Russell et al., 2018). Importantly, care needs to be taken when extrapolating food security
estimates to arrive at higher data projections to avoid masking community-level differences.
Maintaining disaggregated data (e.g. demographics) is essential for understanding food security
trends and tailoring food security efforts to local contexts (Morales et al., 2020).
8
Table 2.1. Food security metrics and their scale, strengths, and weaknesses (adapted from Jones
et al., 2013; Leroy et al., 2015; Haysom and Tawodzera, 2018; Hendriks, 2020; Allee et al., 2021).
Approaches Description Scale Strengths Weaknesses
Direct
Food Insecurity
Experience Scales
including HFIAS,
MAHFP, and HFSSM
- Scale contains items
representing the
multidimensional
nature of food
insecurity
- Algorithm to convert
scale scores into food
insecurity categories
- Determines anxiety
levels, food
preferences, economic
access, food quantity
- Individual
- Household
- Measures food
security
according to
individual
experiences
- Captures the
psychosocial
dimensions of
food security
along with
physical
experiences
- Valid across
varied socio-
cultural settings
- Difficult to
establish cut-off
points for
classifying
households into
different levels of
food security
- Does not
capture food
safety dimension
- Different
frequency
response options
needed in
different settings
Indirect
Anthropometry - Focuses on weight,
height, body size, and
skinfold
measurements
- Determines
nutritional status
Individual
- Standardized
measurements
of weight and
height are
reproducible
across
individuals
- Evidence-
based cut-off
points
- Requires time
to collect
measurements
- Measures food
security indirectly
(e.g. uses
anthropometry
as a proxy for
food utilization
Whole Measures for
Community Food
Systems
Measures the
multitude of
interconnected
indicators that define
a healthy, whole
community (e.g.
access to healthy and
affordable food,
strong community
ties, thriving local
economies)
Community - Informs
dialogue about
and
measurements
toward
common values
- Helps to share
the stories of
complex
community
food projects
- Requires time
to learn and
implement the
tool
- Rubric model
may be
cumbersome in
scope
Global Food Security
Index
Index of 34 indicators
within three domains
National (can
inform global)
Provides cross-
national
Measures food
security indirectly
9
of food security: 1)
affordability; 2)
availability; and 3)
quality and safety
comparisons of
food security
status,
determinants,
and outcomes
(e.g. through
measuring the
conditions for
food security
rather than
actual food
security level)
Household
Expenditure Surveys
- Focuses on the
household
expenditure of food
and other necessities
- Determines caloric
intake per capita per
household and
economic access to
food
Household
(can inform
national)
- Dietary quality
and income
data can help
understand
food security
dimensions
- Less costly
and time-
consuming
alternative to
other measures
- Measures
available amount
- Conversion of
available food to
caloric intakes
involves major
assumptions that
can cause
measurement
errors
Coping Strategy Index - Questions how
households are
responding to food
shortages
- Determines how
households are
responding and/or
adapting to the
presence or threat of
food shortages
Household
(can inform
national)
- Uses simple
questions that
are easy to
understand and
implement
- Captures
directly the
notion of
adequacy and
vulnerability
Can be
misleading; both
richer and poorer
households may
report smaller
portions, but it
does not imply
an equal increase
in food insecurity
Dietary Intake
Assessments
including
DDS, FVS, FFS
- Focuses on different
items consumed in a
specific period
- Determines the sum
of the different foods
consumed by
individual/household
over a specific period
and quality of food
- Individual
- Household
- Addresses
both dietary
quality & caloric
intakes at an
individual level
- Possible to
understand
intra-household
food security
levels
- Rely heavily on
respondent’s
memory which
can lead to
measurement
error
- Requires
experienced
researchers to do
interviews
HFIAS, Household Food Insecurity Access Scale Indicator; MAHFP, Months of Adequate Household Food
Provisioning; HFSSM, Household Food Security Survey Module; DDS, Dietary Diversity Scores; FVS, Food
Variety Scores; FFS, Food Frequency Scores.
10
2.4 Food security programs
As with food security measures, programs working toward food security can also vary in scale.
Strategies can begin on an individual level. Using food safety as an example, educational
opportunities could support community members learn more about hygienic practices for safe
food handling. At an institutional level, food safety policy, regulation, and surveillance could
enhance both nutritional quality and the safety of food more widely. Additionally, food security
programs are diverse and vary from being complicated to complex, influenced by the specific
food security phenomenon, program, and/or systems in which the program is placed (Hawe,
2015). Lunch programs providing youth nutritious food could be considered a straightforward
way of encouraging good nutritional intake. More complex is agricultural training programs
promoting behavioural change to encourage expanded agricultural production, especially when
climate-resilient practices are not considered in the program design.
Many food security programs encompass attributes of complex programs (Skivington et al.,
2021). Because food security is a multi-dimensional phenomenon, food security programs often
have an extensive array of activities, from behavioural change to policy change, both of which
are interdependent. Furthermore, many different disciplines, sectors, and actors engage in food
security research and/or practice, resulting in different perspectives on how food security should
be achieved (Hendriks, 2015). These different perspectives often contribute to the design of
food security programs with strategies that change over time, making it difficult to attribute
changes in food security to a specific strategy (Foran et al., 2014). Complex food security
11
programs require flexibility in adapting food security strategies to new learning during
implementation (e.g. feedback from participants) and changes to context (e.g. climate impacts
and the need to adapt) (Hendriks, 2020).
Food security programs often experience complexity because food security is a part of broader
food systems, which encompass all activities and actors in the production, transport, processing,
package, storage, retail, consumption, loss, and waste of food (Mbow et al., 2019). Food systems
(and therefore food security programs) must respond to a host of external factors, especially
climate change. Climate irregularity is a key driving force behind rises in global food insecurity
(von Grebmer et al., 2020). Changes in extreme weather events (Devereux, 2007; Gbegbelegbe et
al., 2014) and temperature and rainfall variability (Myers et al., 2017; Niles & Brown, 2017)
threaten food security by decreasing global food production and increasing the risk of hunger
and undernutrition (Lake et al., 2012; Schmidhuber & Tubiello, 2007). Climate change introduces
unpredictability during food security program implementation, making it challenging to
anticipate food security outcomes in advance. Food security programs increasingly need to
invest in climate adaptation measures to decrease the risks on programming (Conevska et al.,
2018).
2.5 Program monitoring and evaluation
Monitoring and evaluation provide important information about the progress and performance
of food security programs (Uitto et al., 2016; FAO et al., 2018; Lam et al., 2019). Monitoring has
12
several meanings in a food security context. It may refer to the routine collection of information
about program processes to understand the progress being made toward food security targets,
informing program implementation and/or evaluation (UNEG, 2013). Monitoring can also be
part of a food security program itself when used to track and respond to changes in food
security situations (WFP, 2012). Evaluation also has multiple purposes, from rendering a value
judgment to generating learning for program improvement (Wanzer, 2021). The current
understandings of evaluation are not so different from Scriven’s (1991) definition of evaluation
20 years ago when evaluation was described as: “a systematic process to determine merit, worth,
or value of something, or the product of that process” (p. 139). Evaluation, as a way of analyzing
programs, for the most part still reflects its historical underpinnings; it is rooted in a particular
moment in time informed by the values, experience, and needs of a small group of white men
(Dean-Coffey, 2018).
While monitoring to track progress toward program goals is a more-or-less standard across
food security programs, there is no one ‘best’ approach to evaluating food security programs
(Hendriks, 2020; Levinson & Herforth, 2013). The appropriate approach is dependent on the
evaluation questions, intended uses, and attributes of the intervention and evaluation (Befani,
2020). In food security contexts, approaches to evaluation typically draw on positivist
approaches, regardless of the food security topic, study site, and target population. For example,
a systematic review of evaluations of interventions tackling children’s food security in low- and
middle-income countries (LMICs) identified 42 studies, most of which used a quantitative design
13
(88%, n = 37) (Holley & Mason, 2019). Several reviews of food security program evaluations
exclusively selected quantitative studies for review, including those on food safety education in
high-income countries (Young et al., 2015), smallholder livestock interventions in LMICs (Lindahl
et al., 2020), and agriculture for nutrition interventions globally (Bird et al., 2019).
Program evaluations privilege quantitative approaches because food security is often measured
quantitatively, and agricultural experiments tend to draw on experimental designs. While serving
an important role in identifying causal relationships, quantitative approaches that do not
provide an understanding of a program’s context are not very effective in capturing insights into
how and why food security programs work (Hendriks, 2020). Furthermore, the conventional
assessment of program performance against desired outcomes (e.g. results-based management)
is often not effective for complex food security programs where program activities and intended
outcomes can shift due to climate impacts and other contextual factors (Kidoido and Child,
2014; Bilali et al., 2019). Although important, there is a gap in the use of mixed-methods,
qualitative, context-sensitive, and theory-based designs in food security program evaluation
(Bird et al., 2019; Holley & Mason, 2019; Young et al., 2015).
There have been increasing developments of alternative approaches to monitoring complex
food security programs. In the context of agricultural innovation system interventions,
suggested monitoring approaches included outcome mapping, rapid appraisal of agricultural
knowledge systems, most significant change, and participatory impact pathway analysis (Faure
14
et al., 2020; The World Bank, 2012). These monitoring practices recognize the nonlinearity of
agricultural and other development interventions, accepting that outcomes emerge following
ongoing iterations of program implementation. There are also developments in monitoring
programs that track changes to food security status. For example, community-based monitoring
is increasingly considered a promising strategy whereby groups collaborate to track and
respond to local food security issues (Berner et al., 2016; Danielsen et al., 2014; Johnson et al.,
n.d.). If challenges with data ownership, funding, and protection of local monitors are addressed,
community-based monitoring offers an improved understanding of long-term food security
trends, skills development for local monitors, and provision of timely information for local
decision-making.
The need to uncover learnings from complex food security programs has also led to the
development of alternative approaches to evaluation. For example, one promising way of
enhancing the usefulness of program evaluations is to ensure they are based on a clear program
theory, often represented by a Theory of Change (McDermott et al., 2015; Rajala et al., 2021).
Theory of Change aims to move evaluation beyond an input-output notion of evaluation, and
instead, requires program designers to explicitly state how, why, and in what context impact will
be achieved (Douthwaite et al., 2017; Maru et al., 2018). Theories of Change are often visualized
as change pathways with linkages between program activities and outcomes as well as
assumptions underpinning these linkages. For example, in the context of a disaster risk
reduction intervention in Niger, the Theory of Change identified climate action as a key activity
15
influencing outcome pathways (Béné et al., 2020). Tracking the outcomes along the change
pathway – either qualitatively or quantitatively – can help program designers understand the
progress being made toward often elusive food security goals (McDermott et al., 2015).
It is generally assumed – but increasingly debated and largely untested – that qualitative,
contextually responsive, and theory-driven approaches to evaluation can effectively support
complex food security programs in the future (Belcher et al., 2017; Y. Maru et al., 2018).
However, this assumption has been accompanied by comparatively little effort to reflect on their
use in and applicability to food security contexts. Understanding context is important for
understanding how programs work, why they sometimes fail, and how they can be adapted
from one context to another (Craig et al., 2018). Theories of Change are immensely popular, but
also increasingly critiqued for not only neglecting context but also being under-developed
(Armitage et al., 2019; Belcher et al., 2017; Moore & Evans, 2017). If alternative approaches to
evaluation are to contribute to the design, implementation, and evaluation of complex
programs, it is important to clarify the context under which they are helpful. Knowing what
works and when can help evaluators anticipate challenges and select evaluation approaches
likely to generate learnings for food security research and practice.
Evaluators are increasingly recognizing the threat climate change poses to food security and
other development efforts, and in response, communities of practice have been formed to foster
evaluative thinking around climate (e.g. Earth-Eval) and food security (e.g. EvalForward) (Rowe,
16
2019; Ssekamatte, 2018; Uitto 2019). However, there is a gap in conceptualizing the intersection
of climate change and food security in evaluation, particularly around how programs affect (and
are affected by) climate change. Standard food security monitoring and evaluation are typically
not designed to track the response of food programs to climate change (Hendriks, 2020; Tendall
et al., 2015). Importantly, there is a need for insights into how food security monitoring and
evaluation can account for complexity introduced by climate change.
There is also growing calls for food systems transformations that support healthy diets in
sustainable, resilient, just, and equitable ways, with concurrent calls for rigorous monitoring and
evaluation (Fanzo et al., 2021; UN, 2021). A promising approach to help think about the
connections between food systems and climate change within evaluation is Blue Marble
evaluation (Vidueira, 2019). Originally applied in food systems transformation contexts, Blue
Marble evaluation is a principles-based approach to evaluating global systems change initiatives
(Richardson & Patton, 2021). This approach to evaluation encourages evaluators to both “zoom
out” for a big picture perspective and “zoom in” to incorporate context when developing and
evaluating transformation pathways. Additionally, Blue Marble evaluation encourages evaluators
to examine connections across scales, siloes, disciplines, and issues. While influencing systems
change is likely beyond the scope of many programs working toward food security, considering
how programs impact and are impacted by global challenges like climate change is critical for
ensuring climate-responsive programming. Of note, this dissertation does not focus on specific
approaches to evaluation a priori but explores those that emerge during this research process.
17
2.6 Dissertation objectives
Conventional approaches to evaluation often rest on the assumption that programs are
delivered as originally intended (Hendriks, 2020; McDermott et al., 2015; Steele Gray & Shaw,
2019). Yet, complex food security programs have strategies and outcomes that continually adapt
in response to learnings during implementation (Ansah et al., 2019; Breeman et al., 2015).
Additionally, food security programs are affected by complexity introduced by climate change
and other contextual factors, which are not well accounted for in conventional monitoring and
evaluation practices (Ebi & Del Barrio, 2017). Frameworks exist to help guide the development of
complex health programs (Moore et al., 2014; O’Cathain et al., 2019; Skivington et al., 2021) and
account for their context (Craig et al., 2018; Paparini et al., 2021). However, experience is still
quite limited and there is minimal information on how such frameworks might support food
security efforts specifically. To address this gap, my research objectives are to:
1) Characterize how food security is monitored under a changing climate;
2) Assess how evaluations of food security programs are considering climate change;
3) Characterize the use of theory-driven evaluation in food security contexts; and,
4) Capture lessons learned from designing and evaluating complex food safety programs.
Ultimately, this research aims to inform the development and evaluation of complex food
security programs. Additionally, this research aims to improve understandings of how
evaluations of food security programs might respond to climate change, which matters
18
profoundly within the current context of climate change, rising food security challenges, and
inequitable health outcomes.
2.7 Why does this dissertation matter today?
The Secretary-General of the United Nations convened a Food Systems Summit as a way of
setting the stage for global food systems transformation toward achieving the 2030 Sustainable
Development Goals (UN, 2021). The Summit acknowledged the need to understand how to
effectively monitor and evaluate food systems transformation to ensure transformation efforts
are creating positive change. A key dialogue event focused on convening the private sector,
government agencies, and other food security practitioners to discuss challenges and evidence
gaps in food security monitoring and evaluation (ICRISAT, 2021). For example, a key gap
discussed is the need for monitoring food security outcomes on a real-time basis. As such, this
dissertation is timely in addressing research gaps in food security monitoring and evaluation.
This research also informs strategies for evaluating climate action. To achieve any goal, targets
must be set and progress measured; when it comes to climate goals at the global level, that
assessment is called the “global stocktake.” The stocktake is outlined under Article 14 of the
Paris Agreement and is negotiated through United Nations climate change conferences. The
stocktake, which will occur every five years beginning in 2023, takes a look at the collective
progress the world’s nations have made on climate action (Tompkins et al., 2018). Governments
are still deciding how to collectively evaluate their climate action (Chin & Elliot, 2021). This
19
dissertation provides insights into how climate action might be tracked, drawing on lessons
learned from evaluations of food security programs. In doing so, evaluation can better support
efforts to limit warming to 1.5 degrees C.
Additionally, the most recent United Nations climate change conference (COP26) resulted in an
outcome document, known as the Glasgow Climate Pact, which calls on 197 countries to report
their progress toward more ambitious climate commitments; however, this document falls short
in naming specific accountability and reporting mechanisms (United Nations, 2021a). Post-
COP26, dialogue has been held among development evaluators to explore how evaluative
evidence can help deliver the Paris agreement, with a focus on agriculture and food security
contexts (EvalForward, 2021). This dissertation contributes strategies for collecting evaluative
evidence around climate action.
Food systems transformations are needed for more inclusive and sustainable food systems.
Importantly, because food systems impact (and are impacted by) climate change, food systems
transformation and climate action need to go together. As programs increasingly incorporate
climate action measures, programs will turn to evaluation to assess the coupled climate action
and food security outcomes alongside the dynamic interaction between climate change and
complex food security programming. International trends in monitoring and evaluation of food
system efforts showed climate change as high on the agenda (Kusters et al., 2019), highlighting
the urgency of understanding how to account for climate change in monitoring and evaluation.
20
2.8 Methodological framing
2.8.1 Knowledge synthesis
The Canadian Institutes of Health Research defines knowledge synthesis as “the
contextualization and integration of research findings of individual research studies within the
larger body of knowledge on the topic” (CIHR, 2010, p. 1). Knowledge syntheses can be used to
summarize what is known about a topic, understand inconsistencies across studies, and define
research agendas (Green et al., 2011; Kastner et al., 2012; Lockwood et al., 2019). In contrast to
traditional literature reviews that do not use structured methods to identify studies, knowledge
syntheses apply clearly defined and reproducible methods. Knowledge synthesis is characterized
by three core attributes: it is systematic, comprehensive, and transparent (Greyson et al., 2019).
These attributes are important for reducing bias and enhancing reproducibility during synthesis
steps. There are many types of knowledge syntheses including realist reviews, narrative reviews,
systematic reviews, and scoping reviews (Tricco et al., 2016).
Unlike systematic reviews, which explore questions from quality-appraised studies, scoping
reviews aim to provide an overview or map of the evidence (Colquhoun et al., 2014). Scoping
reviews are among the most common knowledge synthesis approach used in public health,
particularly for answering questions related to the effectiveness of public health interventions
(Lockwood et al., 2019). Scoping reviews are particularly useful when a topic has not yet been
extensively reviewed by bringing together often large and disparate bodies of knowledge
together into a more concise and manageable format (Munn et al., 2018). Often scoping reviews
21
are undertaken to examine the extent, range, and nature of research in a topic area (Arksey &
O’Malley, 2005; Pham et al., 2014).
2.8.1.1 Knowledge synthesis methods
Researchers and evaluators working in food security contexts need access to reliable information
to help inform program decisions (Hendriks, 2020). Because many evaluation reports tend to ‘sit
on the shelf’, scoping reviews can be useful for consolidating what is being learned from
evaluations and making that evidence more readily available (Chandna et al., 2019). Synthesizing
and translating knowledge into food security action (and ways to measure action) matters
greatly, especially in the present context of COVID-19, climate change, and growing rates of
food insecurity (Johnsen et al., 2020). As such, many research chapters in this dissertation (e.g.
Ch. 2-4) draw on scoping reviews to characterize literature in this area. Scoping the literature
identifies ideas where future research may be required but does not evaluate the quality of the
literature (Colquhoun et al., 2014). This distinction is important; these chapters intend to
understand what has been reported in the literature rather than make quality assessments.
Additionally, this dissertation integrates qualitative thematic analysis into scoping reviews to
understand the context behind the reviewed studies as well as strengthen qualitative analysis
skills before working with research participants (Hamel et al., 2021). For example, in Chapter 4, I
used thematic analysis to capture key recurrent messages or ‘themes’ from the reviewed
research and evaluation studies. I selected thematic analysis to move beyond the surface or
obvious content of the data to identify implicit patterns of meaning (Braun et al., 2018). Many of
22
the patterns identified were then explored further in subsequent. Reviewing, assessing, and
synthesizing the literature is a crucial early step before primary data collection, as Clarke argues:
“Those doing qualitative research cannot opt-out of knowing their relevant scholarly
conversations” (Clark, 2016, p. 1).
2.8.2 Qualitative research
Qualitative research is often used in evaluations of complex programs to increase knowledge of
program assumptions, components, and mechanisms (Pawson and Tilley, 1997; Thirsk and Clark,
2017; Rørtveit et al., 2020). Given my research objective of exploring how food security programs
operating under complexity can be supported by evaluation, I chose a qualitative design for the
final research chapter (Ch. 5). This qualitative study focuses on the experiences of researchers
and evaluators involved in food security programs and the broader socio-political contexts that
shape these experiences. Qualitative designs are suitable and often preferred, for research
focused on understanding how people navigate and make decisions in their food environments
(Centre for Food Policy, 2021; Hendriks, 2020). These realities can shed light on the context in
which complex food security programs operate, thus informing the design of evaluations.
Several philosophical assumptions underpinning qualitative research inform the research
process in Chapter 5. For example, ontologically, there are multiple truths and different realities
(Busetto et al., 2020; Guba & Lincoln, 1994; Tomaszewski et al., 2020). Embracing this
assumption, I sought to capture the diverse perspectives of participants, which may conflict but
which, taken together, contribute to a rich understanding of complexities (Creswell, 2014).
23
Qualitative research also assumes the position of the researcher – as the primary research
instrument – impacts all aspects of the research (Jaye, 2002). In my positionality statement, I
discuss how my identities and experiences are entwined in the research (Day, 2012).
Epistemologically, qualitative research takes the stance that knowing is context-dependent, co-
created, and situated (Englander, 2019). I aimed to give a detailed description of the context of
the study and how knowledge reflects a particular point in time (Craig et al., 2018).
To examine complex food security programs operating under complexity for lessons into
program evaluation, I drew on experiences from a food security program called “Market-based
approaches to improving the safety of pork in Vietnam” (SafePORK). SafePORK is a five-year
program (2017-2022) aiming to reduce the burden of foodborne disease in informal, emerging,
and niche markets through designing, piloting, and evaluating food safety interventions.
SafePORK builds on a previous five-year program called PigRISK (2012-2017). Because SafePORK
is a complex, multi-component program conducted by a multidisciplinary team with previous
experience implementing food safety programs (Lam et al., 2016), it provides a realistic example
of how complex food security programs are developed and evaluated. SafePORK
implementation has also been affected by external factors (e.g. African Swine Fever, COVID-19,
climate change), providing opportunities to explore how the program is responding to
complexity.
24
2.8.2.1 Qualitative research methods
To achieve the research objective of capturing lessons learned from designing and evaluating
complex food safety programs, I employed three data collection methods: 1) document reviews;
2) semi-structured interviews; and, 3) reflective memos. I also used these multiple lines of inquiry
to triangulate findings and strengthen the internal validity of the study (Carter et al., 2014;
Creswell, 2014).
Understanding the implementation of food safety programs in Vietnam requires an
understanding of Vietnamese food policies. The primary policy document reviewed was the
2010 Food Safety Law (No 55/2010/QH12), which provides for rights and obligations for
organizations and individuals in assuring safe food (The National Assembly, 2010). I also
reviewed key program documents of SafePORK and PigRISK, including meeting notes,
evaluation reports, annual reports, research briefs, policy briefs, and manuscripts. Finally, I
explored the published and grey literature to better understand the food safety context and
other programs contributing to food security outcomes in Vietnam.
The main data sources were semi-structured interviews with researchers and community
members. The interview questions focused on intervention characteristics, outer setting, inner
setting, characteristics of individuals, and implementation process. The interviews were
conducted remotely between August and September 2021. Qualitative health researchers have
long used online methods to conduct qualitative research (Varma et al., 2021). Douedari et al.
(2021) offer several criteria for reflecting on the appropriateness of remote methods, such as
25
research objectives, target participants, and data collection budget and timeframe. The
evaluation-focused research objective addressed in this study could be considered low-risk.
Additionally, I have worked with SafePORK for evaluation-related activities of SafePORK (both
in-person and remotely) and have visited SafePORK study sites. My research topic and
connections made the use of remote methods likely appropriate in this context.
For SafePORK researchers, I conducted interviews in English via Microsoft Teams. For program
participants, interviews were conducted in Vietnamese via phone calls by two SafePORK team
members. These two team members (one man, one woman) were young, early career
researchers working with the International Livestock Research Institute. They divided up the
participant interviewee list among themselves based on familiarity with interviewees to increase
the likelihood of good conversations. Existing trust was viewed by the team as important,
especially during remote data collection where face-to-face interaction was not possible.
Program participants who completed an interview received 50.000 Vietnamese đồng
(approximately US$2.20) as compensation for their time. Program participants were selected by
team members based on participation in SafePORK to ensure a diversity of experiences and
perspectives. Interviews were audio-recorded with verbal informed consent and ranged from 60-
120 mins (with researchers) and 15-20 mins (with participants).
I wrote reflective memos throughout the entire qualitative data collection process as a means of
documenting contextual information and reflecting on the data collection process (Mulhall,
26
2003; Phillippi & Lauderdale, 2018). I wrote down important insights gained from the document
review. I also documented reflections from the interviews, including initial impressions,
surprising revelations, emerging themes, and potential next steps in my study. Additionally, I
recorded similar reflections from de-brief sessions with team members, program coordinators,
and my PhD advisors. Finally, I kept an audit trail of key decisions made (e.g. revisions to the
interview guide) to enhance the trustworthiness of the qualitative inquiry (Carcary, 2009).
I identified patterns in the data using a deductive approach to thematic analysis (Braun et al.,
2018). I paid specific attention to several elements. First, I considered how context was reported
as per guidelines for process evaluations of complex programs (Moore et al., 2014). Secondly, I
explored explicit or implicit mentions of climate change given its growing impact on food
security and other development programs. Finally, I considered how differences and similarities
in the social identities of researchers and program participants (e.g. population, age, gender)
shaped experiences with SafePORK. This approach enabled us to explore how SafePORK is
responding to the needs and priorities of different groups of people. Coding and analyses were
performed using NVivo© software (QSR International, Burlington, MA, USA).
2.8.2.2 Data validity and informed consent
This research was conducted in collaboration with the coordinators of SafePORK who provided
regular guidance and feedback on this work and are co-authors on the manuscript. I shared
preliminary findings with the program coordinators to inform ongoing implementation,
evaluation, and reporting needs. All interviewees verbally provided their informed consent to
27
participate before the interview. This work received research ethics approval from the University
of Guelph (20-02-003) and Hanoi University of Public Health (018-110/DD-YTCC).
2.8.2.3 A note on language
Interviews conducted in Vietnamese were transcribed by the two SafePORK team members. I
translated Vietnamese transcripts into English before coding. For key quotations used in the
findings, I went back to the original transcripts to capture the nuances of Vietnamese (van Nes
et al., 2010).
2.8.3 Combining knowledge synthesis and qualitative research
Scoping reviews have received criticisms for their positivistic leaning and commitment to
procedural objectivity (Dalmer, 2020). Generally speaking, qualitative researchers reject this
notion of objectivity (Guba & Lincoln, 1994). Another criticism of systematic reviewing is that it
can encourage the superficial reading of the material found, with reviewers simply scanning and
extracting the relevant information (Hammersley, 2020). By contrast, qualitative researchers
emphasize the need for careful reading (Day, 2012). Mindful of these criticisms, it is important to
note that my purpose in reviewing studies is to learn from and contribute to evaluation practice
broadly. Because evaluating complex programs requires a wide range of evaluative evidence
(Paparini et al., 2021), I drew on a range of studies rather than a limited set of studies (e.g.
quantitative studies only). Additionally, I engaged in close reading of studies to understand the
context in which evaluations have been used. These close readings shaped the direction of my
qualitative in-depth study exploring food security programs in their real-life settings, enabling a
28
deeper understanding of how and why programming decisions are made. The use of knowledge
syntheses and qualitative research methodologies contributes to the limited research on
evaluation exploring evaluation approaches in food security contexts (Hendriks, 2020).
2.9 Dissertation structure
I wrote my dissertation in a manuscript-based style to help ensure my work reached researchers
and evaluation practitioners much more quickly than what was possible with a book-style
dissertation. Chapters 2 to 5 represent stand-alone research-based chapters that have been
published or are ready to submit for peer review. Chapter 6 represents a practice-based chapter,
common in the public health discipline, which is designed to be more accessible and facilitate
the use of research findings. This chapter was published in The Conversation Canada. For
chapters already published, the reference style of the journal was used.
Underscoring each chapter is an inquiry and/or description of how monitoring and evaluation in
food security contexts might be conducted effectively during a changing climate. And because
climate change impacts on food security will not be equitable, these chapters pay attention
toward for whom food security programs work. In the early chapters, I learn from other
researchers and evaluators who have shared their practical experiences; specifically, I draw on
knowledge synthesis approaches and principles to explore how climate change is addressed in
food security monitoring (Chapter 2) and evaluation (Chapter 3). Engagement with this literature
was the impetus for knowledge synthesis and qualitative analysis of how alternative approaches
29
to evaluation (especially theory-driven evaluation) were used in food security contexts (Chapter
4). Building on emergent themes from previous chapters, the final research chapter (Chapter 5)
explores the experiences of researchers and participants involved in a food security program in
Vietnam (SafePORK) and examines the role of theory-driven evaluation in supporting this
program. For the practice-based chapter, Chapter 6, I reflect on how global progress on climate
action could be measured, drawing on food security as a case study (Figure 1.1).
Figure 2.1. Visual representation of dissertation research including two chapters on how climate
change is considered in food security monitoring and evaluation (Ch. 2 & 3), two chapters on
theory-driven evaluation (Ch. 4 & 5), and one chapter translating research into practice (Ch. 6).
2.10 Summary
This chapter introduces the importance of enhancing monitoring and evaluation tools for these
tools to continue to learn from and improve complex food security programs. In the context of
30
current calls to integrate climate change into all evaluations, this dissertation also explores how
food security program evaluations could account for climate change and other contextual
factors. Ultimately, this dissertation offers insights into how alternative approaches to evaluation
could support complex food security programs under climate change.
31
2.11 References
Allee, A., Lynd, L. R., & Vaze, V. (2021). Cross-national analysis of food security drivers:
comparing results based on the Food Insecurity Experience Scale and Global Food Security
Index. Food Security. https://doi.org/10.1007/s12571-021-01156-w
Ansah, I. G. K., Gardebroek, C., & Ihle, R. (2019). Resilience and household food security: a review
of concepts, methodological approaches and empirical evidence. In Food Security.
https://doi.org/10.1007/s12571-019-00968-1
Arksey, H., & O’Malley, L. (2005). Scoping studies: towards a methodological framework.
International Journal of Social Research Methodology, 8, 19–32.
Armitage, D., Arends, J., Barlow, N. L., Closs, A., Cloutis, G. A., Cowley, M., Davis, C., Dunlop, S. D.,
Ganowski, S., Hings, C., Rotich, L. C., Schang, K., Tsuji, S., & Wiens, C. (2019). Applying a
“theory of change” process to facilitate transdisciplinary sustainability education. Ecology
and Society, 24(3). https://doi.org/10.5751/ES-11121-240320
Ayala, A., & Meier, B. M. (2017). A human rights approach to the health implications of food and
nutrition insecurity. In Public Health Reviews. https://doi.org/10.1186/s40985-017-0056-5
Barrett, C. B. (2010). Measuring food insecurity. In Science.
https://doi.org/10.1126/science.1182768
Befani, B. (2020). Choosing Appropriate Evaluation Methods.
Belcher, B., Suryadarma, D., & Halimanjaya, A. (2017). Evaluating policy-relevant research: lessons
from a series of theory-based outcomes assessments. Palgrave Communications, 3(1).
https://doi.org/10.1057/palcomms.2017.17
Béné, C., Riba, A., & Wilson, D. (2020). Impacts of resilience interventions – Evidence from a
quasi-experimental assessment in Niger. International Journal of Disaster Risk Reduction.
https://doi.org/10.1016/j.ijdrr.2019.101390
Berner, J., Brubaker, M., Revitch, B., Kreummel, E., Tcheripanoff, M., & Bell, J. (2016). Adaptation in
Arctic circumpolar communities: Food and water security in a changing climate.
International Journal of Circumpolar Health, 75. https://doi.org/10.3402/ijch.v75.33820
Bird, F. A., Pradhan, A., Bhavani, R. V., & Dangour, A. D. (2019). Interventions in agriculture for
nutrition outcomes: A systematic review focused on South Asia. Food Policy.
https://doi.org/10.1016/j.foodpol.2018.10.015
32
Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2018). Thematic analysis. In P. Liamputtong (Ed.),
Handbook of Research Methods in Health Social Sciences.
Breeman, G., Dijkman, J., & Termeer, C. (2015). Enhancing food security through a multi-
stakeholder process: the global agenda for sustainable livestock. Food Security, 7(2), 425–
435. https://doi.org/10.1007/s12571-015-0430-4
Busetto, L., Wick, W., & Gumbinger, C. (2020). How to use and assess qualitative research
methods. Neurological Research and Practice. https://doi.org/10.1186/s42466-020-00059-z
Carcary, M. (2009). The research audit trial - enhancing trustworthiness in qualitative inquiry.
Electronic Journal of Business Research Methods.
Carter, N., Bryant-Lukosius, D., Dicenso, A., Blythe, J., & Neville, A. J. (2014). The use of
triangulation in qualitative research. In Oncology Nursing Forum.
https://doi.org/10.1188/14.ONF.545-547
Centre for Food Policy. (2021). Understanding Lived Experience of Food Environments to Inform
Policy: An Overview of Research Methods.
CFS. (2012). Coming to Terms with Terminology: Food Security, Nutrition Security, Food Security
and Nutrition & Food and Nutrition Security. “Committee on World Food Security”
Proceedings of the 39th Session of the Committee on World Food Security.
Chandna, K., Vine, M. M., Snelling, S. J., Harris, R., Smylie, J., & Manson, H. (2019). Principles,
approaches, and methods for evaluation in indigenous contexts: A grey literature scoping
review. Canadian Journal of Program Evaluation. https://doi.org/10.3138/cjpe.43050
Chin, N., & Elliot, C. (2021). Why the First Global Stocktake Should Not Overlook Long-Term
Climate Strategies. World Resources Institute. https://www.wri.org/insights/why-first-
global-stocktake-should-not-overlook-long-term-climate-strategies
CIHR. (2010). A knowledge synthesis chapter.
Clark, A. M. (2016). Why qualitative research needs more and better systematic review. In
International Journal of Qualitative Methods. https://doi.org/10.1177/1609406916672741
Colquhoun, H. L., Levac, D., O’Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M., & Moher,
D. (2014). Scoping reviews: Time for clarity in definition, methods, and reporting. In Journal
of Clinical Epidemiology (Vol. 67, Issue 12, pp. 1291–1294).
https://doi.org/10.1016/j.jclinepi.2014.03.013
33
Conevska, A., Ford, J., Lesnikowski, A., & Harper, S. (2018). Adaptation financing for projects
focused on food systems through the UNFCCC. Climate Policy, 0(0), 1–16.
https://doi.org/10.1080/14693062.2018.1466682
Craig, P., Di Ruggiero, E., Frohlich, K. L., Mykhalovskiy, E., & White, M. (2018). Taking account of
context in population health intervention research: guidance for producers, users and funders
of research.
Creswell, J. W. (2014). Research Design: Qualitative, Creswell, J. W. (2014). . Research design
Qualitative quantitative and mixed methods approaches. Research Design Qualitative
Quantitative and Mixed Methods Approaches.
Dalmer, N. K. (2020). Unsettling Knowledge Synthesis Methods Using Institutional Ethnography:
Reflections on the Scoping Review as a Critical Knowledge Synthesis Tool. Qualitative
Health Research. https://doi.org/10.1177/1049732320949167
Danielsen, F., Topp-Jørgensen, E., Levermann, N., Løvstrøm, P., Schiøtz, M., Enghoff, M., &
Jakobsen, P. (2014). Counting what counts: using local knowledge to improve Arctic
resource management. Polar Geography, 37(1), 69–91.
https://doi.org/10.1080/1088937X.2014.890960
Day, S. (2012). A reflexive lens: Exploring dilemmas of qualitative methodology through the
concept of reflexivity. Qualitative Sociology Review.
Dean-Coffey, J. (2018). What’s Race Got to Do With It? Equity and Philanthropic Evaluation
Practice. American Journal of Evaluation. https://doi.org/10.1177/1098214018778533
Devereux, S. (2007). The impact of droughts and floods on food security and policy options to
alleviate negative effects. Agricultural Economics, 37(S1), 47–58.
https://doi.org/10.1111/j.1574-0862.2007.00234.x
Dodd, W., Kipp, A., Bustos, M., McNeil, A., Little, M., & Lau, L. L. (2021). Humanitarian food
security interventions during the COVID-19 pandemic in low-and middle-income countries:
A review of actions among non-state actors. In Nutrients.
https://doi.org/10.3390/nu13072333
Douedari, Y., Alhaffar, M., Duclos, D., Al-Twaish, M., Jabbour, S., & Howard, N. (2021). ‘We need
someone to deliver our voices’: reflections from conducting remote qualitative research in
Syria. Conflict and Health. https://doi.org/10.1186/s13031-021-00361-w
Douthwaite, B., Mur, R., Audouin, S., Wopereis, M., Hellin, J., Moussa, A.S., Karbo, N., Kasten, W.,
Bouyer, J. (2017). Agricultural Research for Development to Intervene Effectively in Complex
34
Systems and the Implications for Research Organizations.
Ebi, K. L., & Del Barrio, M. O. (2017). Lessons learned on health adaptation to climate variability
and change: Experiences across low- and middle-income countries. In Environmental Health
Perspectives (Vol. 125, Issue 6). https://doi.org/10.1289/EHP405
Economist Impact. (2021). Methodology.
https://impact.economist.com/sustainability/project/food-security-
index/Home/Methodology
El Bilali, H., Callenius, C., Strassner, C., & Probst, L. (2019). Food and nutrition security and
sustainability transitions in food systems. In Food and Energy Security (Vol. 8, Issue 2).
https://doi.org/10.1002/fes3.154
Englander, M. (2019). General knowledge claims in qualitative research. Humanistic Psychologist.
https://doi.org/10.1037/hum0000107
EvalForward. (2021). The role of evaluation for climate action.
https://www.evalforward.org/webinars/evaluation-climate-action
Fanzo, J., Haddad, L., Schneider, K. R., Béné, C., Covic, N. M., Guarin, A., Herforth, A. W., Herrero,
M., Sumaila, U. R., Aburto, N. J., Amuyunzu-Nyamongo, M., Barquera, S., Battersby, J., Beal,
T., Bizzotto Molina, P., Brusset, E., Cafiero, C., Campeau, C., Caron, P., … Rosero Moncayo, J.
(2021). Viewpoint: Rigorous monitoring is necessary to guide food system transformation in
the countdown to the 2030 global goals. In Food Policy.
https://doi.org/10.1016/j.foodpol.2021.102163
FAO, IFAD, UNICEF, W. and W. (2018). The state of food security and nutrition in the world.
FAO, IFAD, UNICEF, W. and W. (2021). The State of Food Security and Nutrition in the World 2021.
Farrell, P., Thow, A. M., Abimbola, S., Faruqui, N., & Negin, J. (2018). How food insecurity could
lead to obesity in LMICs. Health Promotion International.
https://doi.org/10.1093/heapro/dax026
Faure, G., Blundo-Canto, G., Devaux-Spatarakis, A., Le Guerroué, J. L., Mathé, S., Temple, L.,
Toillier, A., Triomphe, B., & Hainzelin, E. (2020). A participatory method to assess the
contribution of agricultural research to societal changes in developing countries. Research
Evaluation. https://doi.org/10.1093/reseval/rvz036
Foran, T., Butler, J. R. A., Williams, L. J., Wanjura, W. J., Hall, A., Carter, L., & Carberry, P. S. (2014).
Taking complexity in food systems seriously: An interdisciplinary analysis. World
35
Development, 61, 85–101. https://doi.org/10.1016/j.worlddev.2014.03.023
Friedrichsen, C. N., Daroub, S. H., Monroe, M. C., Stepp, J. R., & Wani, S. P. (2018). Mental Models
of Soil Management for Food Security in Peri-Urban India. Urban Agriculture & Regional
Food Systems. https://doi.org/10.2134/urbanag2017.08.0002
Gbegbelegbe, S., Chung, U., Shiferaw, B., Msangi, S., & Tesfaye, K. (2014). Quantifying the impact
of weather extremes on global food security: A spatial bio-economic approach. Weather
and Climate Extremes, 4, 96–108. https://doi.org/10.1016/j.wace.2014.05.005
Godrich, S. L., Barbour, L., & Lindberg, R. (2021). Problems, policy and politics – perspectives of
public health leaders on food insecurity and human rights in Australia. BMC Public Health.
https://doi.org/10.1186/s12889-021-11188-8
Green, S., Higgins P., T., J., Alderson, P., Clarke, M., Mulrow D, C., & Oxman D, A. (2011). Cochrane
Handbook: Cochrane Reviews: Ch 8: Assessing risk of bias in included studies. In Cochrane
Handbook for: Systematic Reviews of Interventions (Vol. 6, pp. 3–10).
j:%5CWMCSU%5CWORK PROGRAMME%5CPROJECTS TRAINING%5C2_EXTERNAL
EDUCATION & TRAINING%5CCOURSES - Planned Training%5CUoB_EVIDENCE REVIEW
Training%5CCOURSES RUN
Greyson, D., Rafferty, E., Slater, L., MacDonald, N., Bettinger, J. A., Dubé, È., & MacDonald, S. E.
(2019). Systematic review searches must be systematic, comprehensive, and transparent: A
critique of Perman et al. In BMC Public Health. https://doi.org/10.1186/s12889-018-6275-y
Guba, E. G. E., & Lincoln, Y. S. Y. (1994). Competing Paradigms in Qualitative Research. In
Handbook of qualitative research (pp. 105–117).
https://doi.org/http://www.uncg.edu/hdf/facultystaff/Tudge/Guba%20&%20Lincoln%20199
4.pdf
Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C.
(2021). Defining rapid reviews: a systematic scoping review and thematic analysis of
definitions and defining characteristics of rapid reviews. In Journal of Clinical Epidemiology.
https://doi.org/10.1016/j.jclinepi.2020.09.041
Hammersley, M. (2020). Reflections on the Methodological Approach of Systematic Reviews. In
Systematic Reviews in Educational Research. https://doi.org/10.1007/978-3-658-27602-7_2
Hawe, P. (2015). Lessons from Complex Interventions to Improve Health. Annual Review of Public
Health, 36(1), 307–323. https://doi.org/10.1146/annurev-publhealth-031912-114421
Haysom, G., & Tawodzera, G. (2018). “Measurement drives diagnosis and response”: Gaps in
36
transferring food security assessment to the urban scale. Food Policy.
https://doi.org/10.1016/j.foodpol.2017.12.001
Hendriks, S. (2020). Food security policy, evaluation and impact assessment. Earthscan.
Hendriks, S. L. (2015). The food security continuum: a novel tool for understanding food
insecurity as a range of experiences. Food Security, 7(3), 609–619.
https://doi.org/10.1007/s12571-015-0457-6
Holley, C. E., & Mason, C. (2019). A Systematic Review of the Evaluation of Interventions to
Tackle Children’s Food Insecurity. In Current Nutrition Reports (Vol. 8, Issue 1, pp. 11–27).
https://doi.org/10.1007/s13668-019-0258-1
ICRISAT. (2021). MONITORING AND EVALUATION KEY STEPS FOR FOOD SYSTEM
TRANSFORMATIONS. https://www.icrisat.org/monitoring-and-evaluation-key-steps-for-
food-system-transformations/
IPES-Food. (2015). THE NEW SCIENCE OF SUSTAINABLE FOOD SYSTEMS.
Jaye, C. (2002). Doing qualitative research in general practice: Methodological utility and
engagement. Family Practice. https://doi.org/10.1093/fampra/19.5.557
Johnsen, J. T., Buckner, L., & Ray, S. (2020). Knowledge synthesis and translation in global food
and nutrition security to evaluate and accelerate priority actions. BMJ Nutrition, Prevention
& Health. https://doi.org/10.1136/bmjnph-2020-000104
Johnson, N., Behe, C., Danielsen, F., Nickels, S., & Pulsifer, P. L. (n.d.). Community-Based
Monitoring and Indigenous Knowledge in a Changing Arctic :
http://www.inuitcircumpolar.com/uploads/3/0/5/4/30542564/cbm_final_report.pdf
Jones, A. D., Ngure, F. M., Pelto, G., & Young, S. L. (2013). What Are We Assessing When We
Measure Food Security? A Compendium and Review of Current Metrics. Advances in
Nutrition: An International Review Journal, 4(5), 481–505.
https://doi.org/10.3945/an.113.004119
Kastner, M., Tricco, A. C., Soobiah, C., Lillie, E., Perrier, L., Horsley, T., Welch, V., Cogo, E., Antony,
J., & Straus, S. E. (2012). What is the most appropriate knowledge synthesis method to
conduct a review? Protocol for a scoping review. BMC Medical Research Methodology.
https://doi.org/10.1186/1471-2288-12-114
Kidoido, M., & Child, K. (2014). Evaluating value chain interventions: A review of recent evidence.
37
Kusters, C., ten Hove, H., Bosch, D., Herens, M., Wigboldus, S. (2019). Conference Report:
Monitoring and Evaluation for Inclusive and Sustainable Food Systems.
Lake, I. R., Hooper, L., Abdelhamid, A., Bentham, G., Boxall, A. B. A., Draper, A., Fairweather-Tait,
S., Hulme, M., Hunter, P. R., Nichols, G., & Waldron, K. W. (2012). Climate change and food
security: Health impacts in developed countries. Environmental Health Perspectives, 120(11),
1520–1526. https://doi.org/10.1289/ehp.1104424
Lam, S., Barot, M., Nguyen-Viet, H., & Unger, F. (2016). Changes in researcher capacity in
assessing food safety risks and value chains: Insights from PigRisk team. ILRI. Hanoi.
Lam, S., Dodd, W., Whynot, J., & Skinner, K. (2019). How is gender being addressed in the
international development evaluation literature? A meta-evaluation. Research Evaluation,
28(2), 158–168. https://doi.org/10.1093/reseval/rvy042
Leroy, J. L., Ruel, M., Frongillo, E. A., Harris, J., & Ballard, T. J. (2015). Measuring the food access
dimension of food security: A critical review and mapping of indicators. Food and Nutrition
Bulletin. https://doi.org/10.1177/0379572115587274
Leung, C. W., Kullgren, J. T., Malani, P. N., Singer, D. C., Kirch, M., Solway, E., & Wolfson, J. A.
(2020). Food insecurity is associated with multiple chronic conditions and physical health
status among older US adults. Preventive Medicine Reports.
https://doi.org/10.1016/j.pmedr.2020.101211
Levinson, F. J., & Herforth, A. (2013). Monitoring and Evaluating the Food Security and Nutrition
Effects of Agricultural Projects.
Lindahl, J. F., Mutua, F., & Grace, D. (2020). Evaluating farm-level livestock interventions in low-
income countries: A scoping review of what works, how, and why. In Animal Health
Research Reviews. https://doi.org/10.1017/S1466252320000146
Lockwood, C., dos Santos, K. B., & Pap, R. (2019). Practical Guidance for Knowledge Synthesis:
Scoping Review Methods. In Asian Nursing Research.
https://doi.org/10.1016/j.anr.2019.11.002
Maru, Y., Sparrow, A., Stirzaker, R., & Davies, J. (2018). Integrated agricultural research for
development (IAR4D) from a theory of change perspective. Agricultural Systems, 165, 310–
320. https://doi.org/10.1016/j.agsy.2016.09.012
Maru, Y. T., Sparrow, A., Butler, J. R. A., Banerjee, O., Ison, R., Hall, A., & Carberry, P. (2018).
Towards appropriate mainstreaming of “Theory of Change” approaches into agricultural
research for development: Challenges and opportunities. Agricultural Systems, 165, 344–
38
353. https://doi.org/10.1016/j.agsy.2018.04.010
Maxwell, S., & Frankenberger, T. R. (1992). Household food security: concepts, indicators, and
measurements: A technical review.
Mbow, C., Rosenzweig, C., Barioni, L. G., Benton, T. G., Herrero, M., Krishnapillai, M., Liwenga, E.,
Pradhan, P., Rivera-Ferre, M. G., Sapkota, T., Tubiello, F. N., & Xu, Y. (2019). Food Security. In
P. Z. P.R. Shukla, J. Skea, E. Calvo Buendia, V. Masson-Delmotte, H.-O. Pörtner, D.C. Roberts,
J. P. P. R. Slade, S. Connors, R. van Diemen, M. Ferrat, E. Haughey, S. Luz, S. Neogi, M.
Pathak, J. Petzold, & J. M. P. Vyas, E. Huntley, K. Kissick, M. Belkacemi (Eds.), Climate Change
and Land: an IPCC special report on climate change, desertification, land degradation,
sustainable land management, food security, and greenhouse gas fluxes in terrestrial
ecosystems.
McDermott, J., Johnson, N., Kadiyala, S., Kennedy, G., & Wyatt, A. J. (2015). Agricultural research
for nutrition outcomes - rethinking the agenda. Food Security, 7(3), 593–607.
https://doi.org/10.1007/s12571-015-0462-9
Moore, G., Audrey, S., Barker, M., Bond, L., Bonell, C., Cooper, C., Hardeman, W., Moore, L.,
O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2014). Process evaluation in complex public
health intervention studies: The need for guidance. In Journal of Epidemiology and
Community Health. https://doi.org/10.1136/jech-2013-202869
Moore, G. F., & Evans, R. E. (2017). What theory, for whom and in which context? Reflections on
the application of theory in the development and evaluation of complex population health
interventions. In SSM - Population Health (Vol. 3, pp. 132–135).
https://doi.org/10.1016/j.ssmph.2016.12.005
Morales, D. X., Morales, S. A., & Beltran, T. F. (2020). Racial/Ethnic Disparities in Household Food
Insecurity During the COVID-19 Pandemic: a Nationally Representative Study. Journal of
Racial and Ethnic Health Disparities. https://doi.org/10.1007/s40615-020-00892-7
Mozaffarian, D., Fleischhacker, S., & Andrés, J. R. (2021). Prioritizing Nutrition Security in the US.
In JAMA - Journal of the American Medical Association.
https://doi.org/10.1001/jama.2021.1915
Mulhall, A. (2003). In the field: Notes on observation in qualitative research. Journal of Advanced
Nursing. https://doi.org/10.1046/j.1365-2648.2003.02514.x
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic
review or scoping review? Guidance for authors when choosing between a systematic or
scoping review approach. BMC Medical Research Methodology.
39
https://doi.org/10.1186/s12874-018-0611-x
Myers, S. S., Smith, M. R., Guth, S., Golden, C. D., Vaitla, B., Mueller, N. D., Dangour, A. D., &
Huybers, P. (2017). Climate Change and Global Food Systems: Potential Impacts on Food
Security and Undernutrition. Annual Review of Public Health, 38(1), 259–277.
https://doi.org/10.1146/annurev-publhealth-031816-044356
Niles, M. T., & Brown, M. E. (2017). A multi-country assessment of factors related to smallholder
food security in varying rainfall conditions. Scientific Reports, 7(1).
https://doi.org/10.1038/s41598-017-16282-9
O’Cathain, A., Croot, L., Duncan, E., Rousseau, N., Sworn, K., Turner, K. M., Yardley, L., &
Hoddinott, P. (2019). Guidance on how to develop complex interventions to improve health
and healthcare. BMJ Open. https://doi.org/10.1136/bmjopen-2019-029954
Paparini, S., Papoutsi, C., Murdoch, J., Green, J., Petticrew, M., Greenhalgh, T., & Shaw, S. E.
(2021). Evaluating complex interventions in context: systematic, meta-narrative review of
case study approaches. BMC Medical Research Methodology.
Pawson, R., Tilley, R. (1997). Realistic evaluation.
Pham, M. T., Rajić, A., Greig, J. D., Sargeant, J. M., Papadopoulos, A., & Mcewen, S. A. (2014). A
scoping review of scoping reviews: Advancing the approach and enhancing the consistency.
Research Synthesis Methods. https://doi.org/10.1002/jrsm.1123
Phillippi, J., & Lauderdale, J. (2018). A Guide to Field Notes for Qualitative Research: Context and
Conversation. Qualitative Health Research. https://doi.org/10.1177/1049732317697102
Power, E. M. (2008). Conceptualizing food security for aboriginal people in Canada. Canadian
Journal of Public Health, 99(2), 95–97. https://doi.org/10.1016/j.gloenvcha.2007.09.002
Ragavan, M. I., Marcil, L. E., & Garg, A. (2020). Climate change as a social determinant of health.
Pediatrics. https://doi.org/10.1542/PEDS.2019-3169
Rajala, E., Vogel, I., Sundin, A., Kongmanila, D., Nassuna-Musoke, M. G., Musundire, R.,
Mulangala, M. N., Chiwona-Karltun, L., Magnusson, U., & Boqvist, S. (2021). How can
agricultural research translation projects targeting smallholder production systems be
strengthened by using Theory of Change? Global Food Security.
https://doi.org/10.1016/j.gfs.2020.100475
Richardson, R. A., & Patton, M. Q. (2021). Leadership-evaluation partnership: Infusing systems
principles and complexity concepts for a transformational alliance. New Directions for
40
Evaluation. https://doi.org/10.1002/ev.20461
Rørtveit, K., Saetre Hansen, B., Joa, I., Lode, K., & Severinsson, E. (2020). Qualitative evaluation in
nursing interventions—A review of the literature. In Nursing Open.
https://doi.org/10.1002/nop2.519
Rowe, A. (2019). Sustainability-Ready Evaluation: A Call to Action. New Directions for Evaluation,
2019(162), 29–48. https://doi.org/10.1002/ev.20365
Russell, J., Lechner, A., Hanich, Q., Delisle, A., Campbell, B., & Charlton, K. (2018). Assessing food
security using household consumption expenditure surveys (HCES): A scoping literature
review. In Public Health Nutrition. https://doi.org/10.1017/S136898001800068X
Schmidhuber, J., & Tubiello, F. N. (2007). Global food security under climate change. Proceedings
of the National Academy of Sciences of the United States of America, 104(50), 19703–19708.
https://doi.org/10.1073/pnas.0701976104
Shepherd, B. (2012). Thinking critically about food security. Security Dialogue.
https://doi.org/10.1177/0967010612443724
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N.,
French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A
new framework for developing and evaluating complex interventions: Update of Medical
Research Council guidance. The BMJ. https://doi.org/10.1136/bmj.n2061
Ssekamatte, D. (2018). The role of monitoring and evaluation in climate change mitigation and
adaptation interventions in developing countries. African Evaluation Journal, 6(1).
https://doi.org/10.4102/aej.v6i1.254
Steele Gray, C., & Shaw, J. (2019). From summative to developmental: Incorporating design-
thinking into evaluations of complex interventions. Journal of Integrated Care.
https://doi.org/10.1108/JICA-07-2018-0053
Stephenson, E. S. (2020). Akaqsarnatik: Food, power, and policy in Arctic Canada. McGill
University.
Tarasuk, V., Fafard St-Germain, A. A., & Mitchell, A. (2019). Geographic and socio-demographic
predictors of household food insecurity in Canada, 2011-12. BMC Public Health.
https://doi.org/10.1186/s12889-018-6344-2
Tendall, D. M., Joerin, J., Kopainsky, B., Edwards, P., Shreck, A., Le, Q. B., Kruetli, P., Grant, M., &
Six, J. (2015). Food system resilience: Defining the concept. In Global Food Security (Vol. 6,
41
pp. 17–23). https://doi.org/10.1016/j.gfs.2015.08.001
The National Assembly. (2010). Law No. 55/2010/QH12. https://vanbanphapluat.co/law-no-55-
2010-qh12-of-june-17-2010-on-food-safety
The World Bank. (2012). Monitoring Agricultural Innovation System Interventions.
Thirsk, L. M., & Clark, A. M. (2017). Using Qualitative Research for Complex Interventions: The
Contributions of Hermeneutics. International Journal of Qualitative Methods.
https://doi.org/10.1177/1609406917721068
Tomaszewski, L. E., Zarestky, J., & Gonzalez, E. (2020). Planning Qualitative Research: Design and
Decision Making for New Researchers. International Journal of Qualitative Methods.
https://doi.org/10.1177/1609406920967174
Tompkins, E. L., Vincent, K., Nicholls, R. J., & Suckall, N. (2018). Documenting the state of
adaptation for the global stocktake of the Paris Agreement. Wiley Interdisciplinary Reviews:
Climate Change, 9(5), 1–9. https://doi.org/10.1002/wcc.545
Tricco, A. C., Antony, J., Soobiah, C., Kastner, M., Macdonald, H., Cogo, E., Lillie, E., Tran, J., &
Straus, S. E. (2016). Knowledge synthesis methods for integrating qualitative and
quantitative data: A scoping review reveals poor operationalization of the methodological
steps. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2015.12.011
Turner, C., Aggarwal, A., Walls, H., Herforth, A., Drewnowski, A., Coates, J., Kalamatianou, S., &
Kadiyala, S. (2018). Concepts and critical perspectives for food environment research: A
global framework with implications for action in low- and middle-income countries. In
Global Food Security. https://doi.org/10.1016/j.gfs.2018.08.003
Uitto, J.I., Puri, J., & van den Berg, R. D. (2016). Evaluating Climate Change Action for Sustainable
Development. Springer.
Uitto, Juha I. (2019). Sustainable Development Evaluation: Understanding the Nexus of Natural
and Human Systems. New Directions for Evaluation, 2019(162), 49–67.
https://doi.org/10.1002/ev.20364
UN. (2021). The food systems summit. https://www.un.org/food-systems-summit
UN General Assembly. (1948). Universal Declaration of Human Rights | United Nations. In
General Assembly resolution.
UNEG. (2013). UNEG Handbook for Conducting Evaluations of Normative Work in the UN System.
42
United Nations. (1975). Report of the World Food Conferance Rome, 5-16 November 1974.
United Nations Publication.
United Nations. (2021a). COP26 closes with ‘compromise’ deal on climate, but it’s not enough,
says UN chief. https://news.un.org/en/story/2021/11/1105792
United Nations. (2021b). With 30 Million Facing Famine, Secretary-General Announces Prevention
Task Force, Warns Security Council against Cutting Aid as Solution to Economic Woes.
https://www.un.org/press/en/2021/sgsm20619.doc.htm
Urke, H. B., Cao, Z. R., & Egeland, G. M. (2014). Validity of a single item food security
questionnaire in Arctic Canada. Pediatrics. https://doi.org/10.1542/peds.2013-3663
van Nes, F., Abma, T., Jonsson, H., & Deeg, D. (2010). Language differences in qualitative
research: Is meaning lost in translation? European Journal of Ageing.
https://doi.org/10.1007/s10433-010-0168-y
Varma, D. S., Young, M. E., Kreider, C. M., Williams, K., Vaddiparti, K., Parisi, C., & Semeah, L. M.
(2021). Practical Considerations in Qualitative Health Research During the COVID-19
Pandemic. International Journal of Qualitative Methods.
https://doi.org/10.1177/16094069211043755
Vidueira, P. (2019). Blue marble evaluation: a personal, professional and organizational
transformation. Global Alliance for the Future of Food.
von Grebmer, K., Bernstein, J., Wiemers, M., Acheampong, K., Hanano, A., Higgins, B.,
Cheilleachair, R.N., Foley, C., Gitter, S., Ekstrom, K., Fritschel, H. (2020). 2020 Global Hunger
Index.
Wanzer, D. L. (2021). What Is Evaluation?: Perspectives of How Evaluation Differs (or Not) From
Research. American Journal of Evaluation. https://doi.org/10.1177/1098214020920710
WFP. (2012). Monitoring Food Security.
Young, I., Waddell, L., Harding, S., Greig, J., Mascarenhas, M., Sivaramalingam, B., Pham, M. T., &
Papadopoulos, A. (2015). A systematic review and meta-analysis of the effectiveness of
food safety education interventions for consumers in developed countries. In BMC Public
Health. https://doi.org/10.1186/s12889-015-2171-x
Zurayk, R. (2020). Pandemic and Food Security: A View from the Global South. Journal of
Agriculture, Food Systems, and Community Development.
https://doi.org/10.5304/jafscd.2020.093.014
43
CHAPTER 2: COMMUNITY-BASED MONITORING OF INDIGENOUS FOOD SECURITY IN A CHANGING CLIMATE: GLOBAL TRENDS AND FUTURE DIRECTIONS
Abstract
Climate change is expected to exacerbate existing food security challenges, especially in
Indigenous communities worldwide. Community-based monitoring is considered a promising
strategy to improve monitoring of, and local adaptation to climatic and environmental change.
Yet, it is unclear how this approach can be applied in food security or Indigenous contexts. The
objectives of this paper are to: 1) review and synthesize the published literature on community-
based monitoring of Indigenous food security; and, 2) identify gaps and trends in these
monitoring efforts in the context of climate change. Using a systematic search and screening
process, we identified 86 published articles. Relevant articles were thematically analyzed to
characterize elements of community-based monitoring in the context of climate change. Results
show that the number of articles published over time was steady and increased more than two-
fold within the last five years. The reviewed articles reported on monitoring mainly in North
America (37%) and South America (28%). In general, monitoring was either collaborative (51%)
or externally-driven (37%), and focused primarily on tracking wildlife (29%), followed by natural
resources (16%), environmental change (15%), fisheries (13%), climate change (9%), or some
combination of these topics (18%). This review provides an evidence-base on the uses,
characteristics, and opportunities of community-based monitoring, to guide future food security
monitoring efforts in the context of climate change.
44
2.12 Introduction
Enhancing the resilience of different food systems to climate change via monitoring is an
important global health opportunity of the 21st century (1). Through routine monitoring and
assessments of climate-related risks and their interplay with food security, along with linking
early warnings to early responses, food and nutrition crises can be better managed (2,3). Such
efforts are especially crucial now as climate change increasingly challenges food security1 (4,5).
Changes in extreme weather events (6,7), temperature and rainfall variability (8,9), and sea level
(10), threaten food security by decreasing global food production and increasing the risk of
hunger and undernutrition (3,11). Projections indicate these food-related health impacts will far
exceed all other climate-related health risks (12). The effects of climate change on food systems
are expected to be widespread, complex, and variable, both geographically and temporally (13).
Investing in food systems adaptation to climate change across scales, particularly via monitoring,
decreases the risks and uncertainties for food and health systems (14).
The impacts of climate change will not be evenly distributed (15), with Indigenous peoples2
facing complex challenges to their food systems (16,17). Food systems encompass a number of
activities which give rise to a number of food security outcomes (i.e. stability of food access,
1 Food security can be defined as “a situation that exists when all people, at all times, have physical, social and
economic access to sufficient, safe and nutritious food that meets their dietary needs and food preferences for an
active and healthy life” (63). 2 Indigenous peoples can be defined as “the assembly of those who have witnessed, been excluded from, and have
survived modernity and imperialism. They are peoples who have experienced the imperialism and colonialism of the
modern historical period beginning with the Enlightenment. They remain culturally distinct, some with their native
languages and belief systems still alive” (129, p. 114).
45
utilization, and availability); we use the term ‘Indigenous food systems’ to refer to systems of
production, process, distribution, and consumption, which are specific to particular geographic
regions (18,19). Several factors make Indigenous food systems particular sensitive to climate
change impacts. These include: histories and ongoing pressures of colonialism and land
dispossession that have disconnected Indigenous peoples from their land and local knowledge
of food practices (20,21), high burden of existing food security challenges (17), structural
inequities characterized by lack of access to land and other resources (22–24), and habitation in
areas undergoing rapid environmental change, biodiversity loss, and competing demands for
land for food production (25). Indeed, Indigenous communities in many countries tend to be
more food insecure than their non-Indigenous counterparts (26–28); for example, Indigenous
peoples in the United States including American Indians and Alaska Natives were twice as likely
to be food insecure compared to non-Indigenous peoples (29). Around 97% of Indigenous
Batwa households in Uganda were found to be food insecure, which was substantially higher
than the national Ugandan average of 20% (30). Furthermore, Indigenous peoples often have
close relationships with the environment for subsistence; as such, even subtle changes in the
environment can have large impacts on their food security (31,32), including reducing access to,
and availability of, Indigenous foods (33,34).
A food security monitoring system can help reduce climate change impacts on vulnerable food
systems (35). The information captured by the monitoring system can be used for several
purposes: to contribute to monitoring the implementation of international commitments (e.g.,
46
Sustainable Development Goals, United Nations Framework Convention on Climate Change,
Paris Agreement); to identify vulnerable areas; to serve as an early warning system for
impending food crises (36–38); and, to inform climate change adaptation3 strategies (39,40).
However, such monitoring systems currently face several major challenges. Due to the diverse
conceptualizations of food security , measurement tools, and intended uses of information, it is
often not clear what exactly is being assessed when we measure food security (41). Secondly,
food security monitoring is typically done through population-based surveys at the national
level (2,38,42). These surveys are generally not designed to provide nuanced understandings of
food security among populations, do not adequately engage with Indigenous communities (42–
44), and are not appropriate for understanding food security of certain population groups
(37,45). For example, a widely used and adapted income-based measure, the Household Food
Security Survey Module does not capture Indigenous food cultures (46,47); instead, it presents a
static snapshot of food security even though Indigenous food systems vary substantially by
household and season (16). Finally, measures of attributing food security challenges to climate
change are not well-developed (48,49); thus, it is not clear whether challenges are due to climate
change or other factors. Yet, overcoming these challenges is key to understanding current and
projected food security and climate-related food shocks (2).
3 Climate change adaptation can be defined as: “In human systems, the process of adjustment to actual or expected
climate and its effects, in order to moderate harm or exploit beneficial opportunities. In natural systems, the process
of adjustment to actual climate and its effects; human intervention may facilitate adjustment to expected climate and
its effects” (131).
47
Community-based monitoring (CBM) is often considered a promising strategy to improve
monitoring of, and local adaptation to, environmental change (50–52). CBM is an approach
whereby groups collaborate to track and respond to issues of common community concern (50).
This approach can take several forms ranging from community-directed monitoring initiatives
originating from communities’ interests and needs to initiatives that simply involve communities
in data collection (53). Similarly, the potential benefits of CBM also vary, including: improved
understanding of long-term trends; reduced cost of research by leveraging existing
infrastructure; skills development and employment opportunities for local monitors; and
provision of timely and relevant information for local decision-making (51,54,55). Moreover, as
many Indigenous communities have been monitoring the environment for centuries (56,57),
there are opportunities to use both Indigenous and Western knowledge systems in CBM to
develop a deeper understanding of pressures on the environment as they arise (58). CBM can be
a key component of community-based adaptation by working with Indigenous communities and
knowledge systems to prepare for the food-related health impacts of climate change (59).
Considering the promise of CBM, the disproportionate food security challenges experienced by
Indigenous communities, and the high sensitivity of Indigenous food systems to climate change,
the goal of our review was to understand and learn from how CBM has been used globally to
track and respond to Indigenous food security and climate-related food challenges. Specifically,
our objectives were to review and synthesize the published literature on CBM of Indigenous
food security in a climate change context. In doing so, we accumulate the knowledge and
48
experiences of developing, implementing, and evaluating CBM systems worldwide. Moreover,
we highlight how Indigenous food security monitoring is different from general food security
monitoring. This is a step toward informing community-based adaptation efforts for addressing
food security challenges of Indigenous communities worldwide.
2.13 Methods
We examined the published literature using a systematic review methodology for climate
change adaptation outlined by Berrang-Ford et al. (2015) involving a stepwise process of
selection, extraction, analysis, and synthesis of the literature (60). To guide our review and
synthesis of trends and gaps, we posed the following question: What does the published
literature tell us about Indigenous community-based monitoring of food security in the context
of climate change?
2.13.1 Search strategy
The initial search for published articles was performed on 5 November 2017 and later updated
on 15 March 2018 using the following databases: AGRICOLA©, PRISMA©, MEDLINE®, CabDirect©,
and the Web of Science™ CORE Collection (Appendix 1). These databases cover agriculture,
health, sociology, and environmental disciplines, therefore providing the opportunity to capture
the broad literature as well as approach the research question from different perspectives. No
search restrictions were placed (e.g. language, date). Each database search used Boolean
operators to pair keywords (Indigenous, monitoring systems, food security / climate change,
community-based) with their synonyms (Table 2.1).
49
2.13.2 Citation management
All citations were imported into the web-based application DistillerSR© (Evidence Partners
Incorporated, Ottawa, ON, Canada). Duplicate citations were removed using the duplicate
removal function.
2.13.3 Relevance screening and eligibility
A two-step relevance screening strategy was employed by two independent reviewers (SL, CZ).
First, the titles and abstracts of articles were screened. Next, all citations deemed potentially
relevant went through a review of the full-text. We included articles that: 1) were published in
journals; 2) described a CBM system; 3) described any aspect of food security or food-related
climate change impacts; and 4) explicitly mentioned an Indigenous community (Table 2.2). The
reference lists of all relevant articles were hand-searched to identify any further relevant studies
not captured in the database search. Reviewers met throughout the screening process to resolve
conflicts and discuss any uncertainties related to study selection. The degree of agreement
between reviewers (i.e., interrater reliability) was assessed using Cohen’s kappa at both stages of
screening (61).
2.13.4 Data extraction and synthesis
A charting form was developed by the authors to capture both count data and information on
the study (62). Information extracted from the paper included the last name of the first author,
year of publication, and whether an author was affiliated with an Indigenous organization or
community (Table 2.3). Descriptions of the CBM system were charted, including the target
50
country, target community, and methods, as well as food security, Indigenous knowledge,
climate change, and gender considerations. We counted whether the CBM system explicitly
mentioned “food security” as well as explicitly mentioned one of the food security pillars (e.g.
access, availability, utilization, stability) (63). We also extracted information on the type of food
source and driver of food security challenges monitored. We counted whether Indigenous
knowledge and climate change were explicitly mentioned. Considering that climate change
impacts will not be gender-neutral (16,64), it is important that responses to climatic stresses not
be limited in the agri-food sector by gender-based constraints. As such, we assessed for gender
considerations in CBM by adapting questions based on a quantitative tool that supports
standardization of sex and gender reporting in publications (65). We also explored the nature of
gender discourse through thematic analysis techniques (66). Specifically, each of these articles
were coded to capture how gender was framed, focusing on gender perspectives and
participation. We also categorized the CBM study broadly as having described: 1) the
development of the monitoring system; or 2) the implementation/evaluation of the monitoring
system. For the latter, we examined each article for evidence to suggest that it could be
classified into one of four general CBM categories: autonomous local monitoring; collaborative
monitoring; consultative monitoring; and externally driven monitoring (67,68). Finally, we
characterized these articles as having an evaluation component if they assessed the
process/implementation, outcome, or impact of the monitoring system.
51
Table 0.1. Search strategy to identify published articles on Indigenous community-based
monitoring of Indigenous food security.
Main terms Expanded terms
Indigenous a Aasax OR Aboriginal OR "Aboriginal-Malay" OR Aborigine OR […]a
Monitoring surveillance OR tracking OR monitoring OR reporting OR "information system"
OR "early warning" OR "early detection" OR "early notification" OR "timely
warning" OR "timely detection" OR "timely notification"
Food security and
climate change
food OR agriculture OR nutrition OR livestock OR fish* OR animal OR plant OR
wildlife OR hunt* OR gather* OR environment* OR ecological OR ecosystem OR
"natural resource" OR "resource management" OR "co-management" OR
"cooperative management" OR "joint management" OR water OR sea OR ocean
OR "climate change" OR "climate variability" OR "climate hazard" OR "extreme
weather" OR "natural hazard" OR disaster OR flood OR drought OR hurricane OR
storm OR cyclone OR "sea level rise" OR "irregular rainfall" OR "intense rainfall"
OR resilience OR poverty OR livelihood OR welfare OR income
Community-based "community-based" OR "community-centred" OR "community-centered" OR
"community-engaged" OR "community-led" OR "locally-based" OR participatory
OR collaborative OR "citizen-led" OR "citizen-engaged" OR "citizen-based" OR
"citizen science" a Search terms used to identify Indigenous peoples globally were derived from Bishop-Williams et al. 2017
(69). A shortened list of expanded search terms was provided here, please see Appendix 2 for the full list
of expanded terms and Appendix 3 for the full list of included articles.
52
Table 0.2. Inclusion and exclusion criteria to identify published articles on community-based
monitoring of Indigenous food security and climate change.
Inclusion Exclusion
Original research published in a
journal
Study described community-based
monitoring or synonyms (e.g.
community-led surveillance)
Study discusses some aspect of
food security, drivers of food
security (e.g. climate change,
poverty), or synonyms (e.g. wildlife,
natural resource)
Study explicitly mentioned an
Indigenous community
Conference abstracts or proceedings, letters to the editor,
reports, news articles, dissertations
Community-based research without a monitoring component,
monitoring study without community-engagement
Study did not discuss food security in any way, study discussed
climate change with no reference to food security
Studies of Indigenous plants, animals, or knowledge without
reference to an Indigenous community, studies of non-
Indigenous communities
53
Table 0.3. Summary of information extracted from articles on community-based monitoring of
Indigenous food security.
Categories Information extracted
Information on the article - Last name of the first author
- Author affiliation with an Indigenous organization or community
(Y/N)
- Year of publication
Community-based
monitoring characteristics
- Study country
- Indigenous community
- Methods used
- Explicit mention of “food security” (Y/N)
- Discussion on food access, availability, or use
- Provided gender-disaggregated data (Y/N)
- Themes surrounding gender (e.g. gender perspectives and
participation)
- Explicit mention of climate change (Y/N)
- Explicit mention of Indigenous knowledge and/or traditional
knowledge (Y/N)
Community-based
monitoring focused on
implementation
- Monitoring approach (autonomous, collaborative, consultative,
externally-driven)
- Evaluation component (Y/N)
2.14 Results
2.14.1 Overview of relevant articles
There were 3986 articles identified through the database search (Figure 2.1). After removal of
duplicates and non-relevant articles, and the addition of articles from reference lists, a total of
86 articles were included. See supplementary file for the full list of included articles and selected
article characteristics. The inter-rater reliability for title/abstract article screening and full-text
article screening was 0.80 and 0.71, respectively, indicating “substantial agreement” (61).
54
Figure 0.1. Flow chart of the selection of studies that explored community-based monitoring of
Indigenous food security.
2.14.2 Diversity of approaches to community-based monitoring
The reviewed studies used several methodological approaches including quantitative (n=8),
qualitative (n=13), mixed methods (n=42), review (n=21), and other (n=42) methodologies4.
Other methodologies included the use of technology, including GIS mapping, GPS tracking, or
use of drones and aerial surveys. Around 42% of articles (n=36) had at least one author with an
affiliation to an Indigenous organization or community with no clear trend on Indigenous co-
4 Numbers do not add up to 86 studies as more than one approach was used in some studies.
55
authorship over time. The majority of articles (n=74) explicitly mentioned “Indigenous
knowledge” or “traditional knowledge” in the context of the study.
2.14.3 Most studies were published in the past decade
Initially, there was a limited the number of articles published following 2001, when articles were
first found. The number of publications increased more than two-fold from 2013 onwards. The
majority of reviewed articles reported on CBM in North America (n=32), followed by South
America (n=24), Australia and Oceania (n=11), Africa (n=6), and Asia (n=4) (Figure 2.2). Several
studies were conducted across multiple continents (n=9). Among the 31 studies published in
North America, most were from Canada (n=28). There was also a varied distribution of
Indigenous group representation, with most studies focused on Indigenous groups in Canada
(n=28), specifically First Nations (n=12), Inuit (n=10), and multiple groups (n=6). Studies also
focused on Indigenous groups of South America (n=25), specifically multiple groups (n=10),
Kaxinawa (n=3), Isoseno-Guarani (n=3), Waiwai (n=1), Xerente (n=1), Makushi (n=1), Matsigenka
(n=1), Kitchwa (n=2), Purepecha (n=1), Andean (n=1), and Amerindian (n=1). There were some
articles on multiple groups from different continents (n=9), Indigenous groups of Oceania
(n=11), Indigenous groups of Africa (n=6), Indigenous groups of Asia (n=4), and Native
Americans (n=3).
56
Figure 0.2. Geographic distribution of articles on community-based monitoring of Indigenous
food security.
2.14.4 Wildlife was a prominent type of food monitored
A wide variety of food sources were monitored, including wildlife (n=25), natural resources
(n=14), and fisheries (n=11). Drivers of food security were also explored, including
environmental change (n=13) and climate change (n=8). Several articles explored a combination
of food sources and food security drivers (n=15). The term “food security” was explicitly
mentioned in 33 articles (38%), while the specific food security pillars reported included food
availability (n=16), food access (n=11), and food utilization (n=6), or some combination of these
pillars (n=17). No articles explored food stability.
57
2.14.5 More articles focused on men’s participation in monitoring
A total of 29 articles (34%) provided gender-disaggregated data. Although 2017 had the highest
proportion of articles that provided gender-disaggregated data, there was no distinguishable
trend suggesting an increase or decrease in active gender considerations over time. For articles
that provided gender-disaggregated data and described the development of the monitoring
system (n=14), all articles considered perspectives of both men and women. The importance of
considering the perspectives of women was emphasized in a study in Nunavik, Canada, aimed to
understand Elders’ and hunters’ observations and knowledge: “Women were included upon the
recommendation of their expert knowledge of ice conditions or their frequency of travel” (64, p.
30). For the articles that provided gender-disaggregated data and described the implementation
of the monitoring system (n=15), all articles mentioned the participation of men, while some of
these articles (n=9) also mentioned the participation of women. One article actively encouraged
the participation of women in monitoring “as a way to acknowledge internally marginalized
groups and alleviate social inequality” (65, p. 22). Although some studies aimed to include equal
participation of men and women, they noted that participants tended to be men (70,72,73).
2.14.6 Climate change indicators were rarely assessed
More than half (56%) of studies (n=48) explicitly mentioned climate change in the context of
food security (Figure 2.3). There was no clear trend over time to suggest a general movement
towards increasing consideration of climate change in articles. While the majority of articles
58
mentioned climate change as justification for pursuing CBM, a limited number of articles (n=8)
described the implementation/evaluation of a community-based food security monitoring
system including climate change indicators. The studies that did consider climate change were
mostly situated in Arctic regions (70,74–77); for example, a study in Alaska employed a
participatory CBM system to explore environmental conditions likely associated with climate
change, health, and food security outcomes (75). In another study, an integrated CBM system in
Nunavik, Canada tracked climatic and ice conditions to support safe access to land and other
resources (70).
Figure 0.3. Articles on community-based monitoring of Indigenous food security with and
without climate change considerations, over time.
0
2
4
6
8
10
12
14
2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
Nu
mb
er o
f ar
ticl
es
Climate change focus Non-climate change focus
59
2.14.7 Varying degrees of Indigenous community engagement in monitoring
Of the 86 included articles, 37 articles (43%) described work that recommended or informed the
development of CBM approaches. For example, Wesche et al. (2011) stated that their research
project “led to the process of establishing a community-based environmental monitoring
program” (68, p. 403). Most articles (n=49) described the implementation of an existing CBM
system. Among these 49 articles, we documented whether the authors evaluated the monitoring
system. We found that less than half of articles (n=22) reported an evaluation process or
outcome (78–80). These 49 articles were also classified according to the type of CBM approach
adopted (Table 2.4). No clear trend on the type of monitoring approach over time was
discernable.
Only one article reporting on autonomous local monitoring was identified (81). The objective of
this article was to highlight the existence of local monitoring practices based on observations in
three communities in Papua, Indonesia (81). The authors found that communities would monitor
and control their environment and resources, and found evidence of autonomous monitoring
activities in each of the three communities.
More than half of the reviewed articles (51%) were classified as collaborative monitoring. This
approach encouraged project co-creation, transfer of ownership, and knowledge sharing with
community members. An example of this approach was illustrated by Cummings et al. (2017).
The authors described steps taken to introduce drones in two Indigenous villages in Southern
60
Guyana and portrayed the extent of community engagement through the following statement:
“When the participants collect data on their missions, they have the first access to these, and can
decide whether to share the data with the non-indigenous [sic] project team members” (73, p.
13). In another example, the authors described: “A CyberTracker was developed and used by the
Naskapi Nation to verify forest-dwelling caribou presence in collaboration with New Millennium
Capital Corporation” (74, p. 39).
Compared to the other monitoring approaches discussed by articles, a relatively low number of
articles describing consultative monitoring were identified (10%). This CBM approach involved
some degree of decision-making by the community. An example of this approach was described
by Shaffer et al. (2017) where the authors “established a hunter self-monitoring program in
consultation with the Waiwai in July of 2014” (75, p. 1121). In the context of participatory
monitoring in the Amazon, the authors “provided technical support for the development of a
hunting monitoring system aimed at informing adaptive management processes for
sustainability” (76, p. 55). In both cases, community members were involved in identifying
monitoring goals and collecting data, while analysis was performed entirely by the researchers.
The proportion of publications classified as externally driven monitoring schemes was 37%. This
CBM approach involved community members only in data collection with no decision-making
by community members. For example, in one study: “All of the data in the study were collected
by locally recruited and trained indigenous [sic] technicians and most of the data-collection
61
process was established a priori, without local input” (77, p. 771). The same authors also
emphasized the importance of knowledge sharing back to the community: “However, we sought
substantial input regarding the format and content of research results to be returned to
collaborating communities” (p. 772). In another study categorized as externally-driven, hunters
were invited to participate on a voluntary basis, and those who accepted were trained to fill
monitoring data sheets (87).
Table 0.4. Types of community-based monitoring approaches described in articles that reported
the implementation of a monitoring system. Adapted primarily from Kouril et al. 2015, Danielsen et al.
2009.
Category Characteristics Number
of
articles
Example
references
Autonomous local
monitoring
Community members are involved in the whole
monitoring process and make all decisions
1 (81)
Collaborative
monitoring
Community members are involved in data collection,
analysis, and interpretation; researchers and
community members make decisions together
25 (70,88–91)
Consultative
monitoring
Community members are involved in data collection
and are consulted; researchers make decisions
5 (84,85,92)
Externally driven
monitoring
Community members are involved only in data
collection or as research assistants, researchers make
all decisions
18 (75,86,87)
2.15 Discussion
This paper systematically examines trends in peer-reviewed publications on CBM. We found the
number of articles on CBM of Indigenous food security were limited but growing. Since 2013,
there has been an increase in the number of publications on this topic by more than two-fold.
62
This trend is consistent with previous studies on broader environmental CBM (50,67) and reflects
the need to include different sources of knowledge and different knowledge users in monitoring
efforts, instead of conventional monitoring approaches which are typically externally driven
(53,93). Furthermore, the increasing interest in CBM is part of a broader trend around the need
to address the disproportionate food security and climate change impacts often experienced by
Indigenous communities globally (17,26–28). The articles captured in this review mainly reported
on CBM in North America, and specifically Canada. This finding could be explained by the
current and projected severity of climate change impacts in the Arctic (such as Arctic Canada)
compared to the rest of world (94), and the need to understand and address such impacts
through CBM. Indeed, there has been a significant growth in research on, and funding for
climate change in Canada’s North (32). This geographic distribution of articles also highlights
gaps in CBM research in many regions where community-level vulnerability is thought to be
significant, particularly in low- and middle-income regions such as Africa and Asia. Gaps could
point to a lack of resources or priority for CBM research in these regions; however, it is possible
that CBM is occurring in these regions, but do not engage with Indigenous communities
specifically, nor include research components to CBM, and thus may not be documented in
published journals. Nevertheless, climate change impacts on Indigenous food systems is a
global phenomenon (16), and this review highlights potential gaps where CBM could be further
developed and implemented.
63
Approximately one third of the reviewed articles provided gender-disaggregated data; this
research gap results in an incomplete understanding of how Indigenous women, men, and
gender-diverse people may differentially participate in, and experience CBM. We encourage
authors and journal editors to integrate assessment of gender into all manuscripts (65). Among
the articles that provided gender-disaggregated data, all articles highlighted the perspectives of
both women and men in the design of the monitoring system. When examining the
implementation of the monitoring system, articles generally emphasized participation of men
only or both men and women. The quality of participation or reasons for participation were
unclear due to lack of reporting in these articles; however, the predominant focus on men in
monitoring efforts could be due to the role of many Indigenous men in hunting, fishing, and
natural resource management in several contexts (84,87,90). Challenges to women’s
participation in, and benefit from, monitoring efforts could be due to failure to recognize and
address gendered power imbalances in project objectives or inadequate considerations of
gender in the program design (95,96). A better understanding of the gendered nature of CBM,
as well as the relationship between climate, food, and gender, are important for planning and
designing a CBM system that reflect gender equity. Critically, an examination of the underlying
social-cultural-political processes that determine differential gender exposure and sensitivity to
climate change and adaptive capacity, as well as gender equity in monitoring is recommended
(97).
64
Considering the history of unethical research conducted on and not with Indigenous
communities (98,99), there is increasing demand for the recognition of Indigenous peoples’
contributions and knowledge in the context of research (100), including climate change research
(101,102). Prioritizing collaborative publications is an emerging avenue for acknowledging
Indigenous peoples’ contributions to research (103). While Indigenous peoples’ observations of
climate change are increasingly reported in the published literature (104), we found the inclusion
of Indigenous peoples as co-authors did not appear to follow the same trend. We found less
than half (42%) of reviewed articles on CBM had a co-author with an affiliation to an Indigenous
organization or community. If a monitoring system is community-led and community-based,
there is a need to consider whether co-authorship is appropriate, recognizing that some
community members may request not to be authors for a variety of reasons. Nevertheless, steps
need to be taken to avoid risks including tokenistic inclusion, implied support for findings, and
misappropriation of knowledge (103,105). Multiple evidence-based (MEB) approaches offer a
way forward for Indigenous and non-Indigenous collaborators to work together in developing
CBM systems that respects and reflects different contributions (106). MEB approaches view
Indigenous, local, and scientific knowledge systems as generating different manifestations of
knowledge, that when viewed as complementary, can generate new insights to support
decision-making and action (57). MEB approaches are recognized under the Convention for
Biodiversity as a way of ensuring equitable participation of diverse knowledge systems in
monitoring biodiversity targets (57), as well as the Intergovernmental Science-Policy Platform on
65
Biodiversity and Ecosystem Services to enhance understanding of governance of biodiversity
and ecosystems (107). The Paris Agreement article 7.5 also supports MEB approaches by
acknowledging adaptation action “should be based on and guided by the best available science
and, as appropriate, traditional knowledge, knowledge of indigenous [sic] peoples and local
knowledge systems, with a view to integrating adaptation into relevant socioeconomic and
environmental policies and actions, where appropriate” (97, p. 25).
Variations in the level of Indigenous community engagement in CBM were observed. Articles on
CBM of Indigenous food security were mostly classified as either collaborative monitoring (51%)
or externally driven monitoring (37%). This finding is consistent with a review of environmental
CBM in the Arctic region which found that 47% of systems involved community members in
collaboration while 30% involved community members in data collection only (51). The higher
number of articles describing collaborative monitoring could be explained by the broader trend
of engaging Indigenous communities in research and monitoring practices, whereas externally-
driven monitoring is likely influenced by conventional, government or researcher implemented
monitoring approaches (109), as well as requests by Indigenous communities for external
monitoring support (51,93). However, successful CBM generally occurs when communities
monitor things they personally connect with and care about, rather than for externally-driven
needs (110,111). Moreover, in the context of climate research, studies initiated with and by
Indigenous community members tend to indicate more responsible community engagement
than studies initiative by outside researchers alone (112). Community engagement is also
66
important for generating local ownership and understandings of environmental change, and to
facilitate the development of local climate change adaptation responses (113).
We reviewed the articles that described the implementation of CBM and found less than half of
these articles reported evaluation findings. The limited focus on evaluation could be explained
by challenges of evaluating climate change adaptation programs and policies including
assessing attribution, creating baselines, absence of measurable outcomes, and monitoring over
long time horizons (49,114). Climate change adaptation is also a relatively new field with little
consensus on what constitutes effective adaptation, and how the success of adaptation efforts
should best be measured (115). Nevertheless, in the context of CBM, it is important that
communities and researchers work together towards a consensus to ensure that CBM is
translated into responses to address climate change impacts and enhance food systems
resilience. Evaluations of CBM can also help identify monitoring gaps, inform governance
systems on monitoring progress, and justify continued support for CBM (116,117). Moreover, a
better understanding of CBM processes and outcomes can serve to develop an evidence base
that will inform future climate change adaptation efforts (118).
Only one formal autonomous monitoring scheme was reported in the published literature (81).
This finding is consistent with a recent review of environmental CBM approaches which found
no formal autonomous monitoring schemes (67). It is likely that monitoring is carried out
without documentation in the published literature, as many Indigenous communities often
67
monitor environmental changes and the health of their land (51). Further, communities
participating in monitoring may not be interested in research related to CBM nor publishing in
the literature, but instead interested in mobilizing CBM to support local decision-making (67).
There may also be challenges in understanding autonomous monitoring stemming from
disciplinary obstacles and difficulties describing and characterizing such systems (81,119).
Nevertheless, greater research attention to autonomous climate change monitoring and
adaptation – case studies outlining the process and outcomes of such activities – could provide
important insights to inform climate change adaptation strategies along with a strong
foundation for further documentation of CBM efforts (81).
Four pillars are often referred to when explaining the concept of food security: availability,
access, utilization, and stability (63). For most of articles on CBM of Indigenous food security
reviewed in this paper the specific pillar of food security was not explicitly mentioned, and food
stability was not mentioned at all. The absence of focus on specific food security pillars could be
due to articles focusing on proxies or indicators of food security (e.g., wildlife, climate change).
Moreover, the objectives of CBM may be to address conservation or natural resource
management goals rather than addressing food security challenges directly (50,89,90,120).
This finding highlights the need to clarify what is being assessed when food security is being
monitored (e.g., which food security pillar is being addressed), so that monitoring approaches
can be coordinated and allow comparisons to be made across studies and contexts. Where the
68
food security pillar was specified, we found that CBM typically focuses on food availability and
access, consistent with studies on impacts of climate change on food security (11,121). Less is
known about the role of climate change on food stability and utilization. Considering and
specifying broader determinants of food security in monitoring systems, together with climate
change indicators (e.g. seasonality, rain variability), could support more accurate, transparent,
and consistent monitoring of food system resilience to climate change.
Conventional food security monitoring tends to fail to probe for information that may be
relevant to Indigenous food security, such as the status of Indigenous food systems or
environmental change (2,36,37). This review found articles focused on food security themes that
extended beyond common metrics of food security (e.g. market foods, poverty) including
wildlife, natural resources, environmental change, and fisheries. However, not all aspects of food
security considered important to Indigenous peoples were captured in CBM systems such as
land and knowledge of local food practices (20,21). This review, then, begs the following
questions: who is CBM of Indigenous food security serving? Does a conventional definition of
food security (i.e. four pillars including availability, access, utilization, and stability) adequately
capture the dynamic nature of Indigenous food systems? If a conventional definition of food
security is used to guide the development of a CBM system with Indigenous communities, does
this definition lead to the creation of systems that overlook critical components of Indigenous
food systems? We encourage researchers to grapple with these questions when developing
food security monitoring systems with Indigenous communities. Where food security was
69
defined broadly, we argue that definitions and metrics should consider components highlighted
above that are important for Indigenous communities. Furthermore, given the diverse
Indigenous groups, food systems, and food practices, it is necessary to incorporate race,
ethnicity, and culture when exploring food security (122). Indeed, Indigenous food security can
be better understood by considering the social and economic benefits that Indigenous food
systems provide (123). The importance of monitoring Indigenous food security, and having an
inclusive definition of food security, is only increasing given the current and projected changes
in climate and its disproportionate impact on Indigenous food systems (34). Supporting
Indigenous communities in defining and monitoring food security, and linking monitoring
responses to decision-making and action, will be crucial for community-based adaptation.
Climate change was explicitly mentioned in more than half of the reviewed articles with climatic
indicators explored in only a few articles. The limited consideration of climatic indicators could
be explained by the lack of consensus on how food systems resilience to climate change should
be assessed (124,125). Food system resilience can be defined as the “capacity over time of a
food system and its units at multiple levels, to provide sufficient, appropriate and accessible
food to all, in the face of various and even unforeseen disturbances” (115, p. 19). Measuring
resilience in this context is challenging because the concept of food system resilience has not
been well defined for climate change (127) and the links between resilience of food systems and
climate change are not straight forward (124). Moreover, limited conceptual tools and
frameworks are available to guide such assessments (127,128). However, many Indigenous
70
communities have knowledge of climate and weather, and have developed adaptation strategies
for ensuring food security. For example, Indigenous knowledge systems such as sky and
astronomical observations, animal behaviours, and wind direction played a key role in
determining when farmers prepare the fields and the nature of crops they plant in a particular
season (129). As such, incorporating Indigenous knowledge in the design of food security and
climate change metrics could help improve our understanding of how climate change impacts
on food systems can be assessed over time.
Our review and synthesis highlight examples of how CBM is being used worldwide to address
food-related climate change impacts. The analysis of trends shows a growing interest in CBM
over time, with steady interest in considerations of climate change. The gaps identified in this
study might be useful for communities, researchers, and decision-makers in developing, refining,
or evaluating similar monitoring efforts. While our paper is comprehensive and systematic, we
note several important limitations. First, to examine research that has been conducted on CBM,
we included only published articles. We acknowledge that a substantial body of work may be
found in the grey literature (67). Analyzing the grey literature is recommended for future
research to better understand the full spectrum of CBM practice occurring within Indigenous
communities, especially of autonomous monitoring systems which may be investigated or
evaluated outside of research and the published literature. Secondly, the categorization of the
monitoring approach of articles, along with considerations of gender, were based on the
information presented in each article. We acknowledge the articles themselves may not fully
71
elaborate on the extent to which community engagement was considered. Greater elaboration
on these processes of community engagement in articles will benefit knowledge sharing among
Indigenous and scholarly communities. Finally, reporting and synthesizing evaluative findings
can provide further insights into the effectiveness, impact, and sustainability of CBM systems.
Overall, this study provides important insights into the trends and future directions in CBM of
Indigenous food security based on experiences from around the world and as reported in the
published literature.
2.16 Conclusion
The monitoring of food security with, for, and by Indigenous communities is a growing area of
research and practice. This trend follows the recent global trend of engaging Indigenous
communities, integrating diverse sources of knowledge, and addressing the disproportionate
food security and climate change challenges experienced by Indigenous communities. The
reviewed articles highlight the importance of considering indicators of wildlife and
environmental change in food security monitoring systems, considerations that are typically
excluded in conventional food security monitoring efforts. While many articles acknowledged
the impact of climate change on food security, few articles explicitly explored climatic indicators.
This finding is reflective of the limited understanding of how food systems resilience to climate
change should be assessed. We also found many examples of collaborative CBM but limited
examples of autonomous CBM. Exploring these trends highlight key research opportunities for
supporting monitoring and adaptation of food systems. First, we encourage articles to reflect
72
and report on the processes and outcomes of CBM. Furthermore, the development and
inclusion of food security and climate change metrics in CBM are also recommended. Finally, a
greater attention to autonomous monitoring and adaptation could help inform more effective
responses to climate change. When viewed as a whole, our findings provide an evidence-base of
research on CBM worldwide to address Indigenous food security challenges, serving as a
resource to inform future community-based adaptation efforts.
73
2.17 References
1. Watts N, Adger WN, Ayeb-Karlsson S, Bai Y, Byass P, Campbell-Lendrum D, et al. The
Lancet Countdown: tracking progress on health and climate change. Vol. 389, The Lancet.
2017. p. 1151–64.
2. World Food Programme. Food Security Monitoring System (FSMS) [Internet]. 2018.
Available from: https://www.wfp.org/food-security/assessments/food-security-
monitoring-system
3. Lake IR, Hooper L, Abdelhamid A, Bentham G, Boxall ABA, Draper A, et al. Climate change
and food security: Health impacts in developed countries. Environ Health Perspect.
2012;120(11):1520–6.
4. Wheeler T, von Braun J. Climate change impacts on global food security. Science.
2013;341(6145):508–13.
5. Tai APK, Martin MV, Heald CL. Threat to future global food security from climate change
and ozone air pollution. Nat Clim Chang. 2014;4(9):817–21.
6. Gbegbelegbe S, Chung U, Shiferaw B, Msangi S, Tesfaye K. Quantifying the impact of
weather extremes on global food security: A spatial bio-economic approach. Weather
Clim Extrem. 2014;4:96–108.
7. Devereux S. The impact of droughts and floods on food security and policy options to
alleviate negative effects. In: Agricultural Economics. 2007. p. 47–58.
8. Niles MT, Brown ME. A multi-country assessment of factors related to smallholder food
security in varying rainfall conditions. Sci Rep. 2017;7(1).
9. Myers SS, Smith MR, Guth S, Golden CD, Vaitla B, Mueller ND, et al. Climate Change and
Global Food Systems: Potential Impacts on Food Security and Undernutrition. Annu Rev
Public Health [Internet]. 2017;38(1):259–77. Available from:
http://www.annualreviews.org/doi/10.1146/annurev-publhealth-031816-044356
10. Chen CC, McCarl B, Chang CC. Climate change, sea level rise and rice: Global market
implications. Clim Change. 2012;110(3–4):543–60.
11. Schmidhuber J, Tubiello FN. Global food security under climate change. Proc Natl Acad
Sci U S A [Internet]. 2007;104(50):19703–8. Available from:
http://www.pnas.org/content/104/50/19703.short
74
12. Hales S, Kovats S, Lloyd S, Campbell-Lendrum D. Quantitative risk assessment of the
effects of climate change on selected causes of death , 2030s and 2050s [Internet]. Risk
Assessment. I.World Health Organization. 2014. Available from:
www.who.int/about/licensing/copyright_form/en/index.html%5Cnhttp://who.int/globalch
ange/publications/quantitative-risk-assessment/en/
13. Vermeulen SJ, Campbell BM, Ingram JSI. Climate Change and Food Systems. Annu Rev
Environ Resour [Internet]. 2012;37(1):195–222. Available from:
http://www.annualreviews.org/doi/10.1146/annurev-environ-020411-130608
14. Conevska A, Ford J, Lesnikowski A, Harper S. Adaptation financing for projects focused on
food systems through the UNFCCC. Clim Policy [Internet]. 2018;0(0):1–16. Available from:
https://doi.org/10.1080/14693062.2018.1466682
15. Dennig F, Budolfson MB, Fleurbaey M, Siebert A, Socolow RH. Inequality, climate impacts
on the future poor, and carbon prices. Proc Natl Acad Sci [Internet]. 2015;112(52):15827–
32. Available from: http://www.pnas.org/lookup/doi/10.1073/pnas.1513967112
16. FAO. Indigenous Peoples’ food systems & well-being: Interventions & polices for healthy
communities [Internet]. 2013. Available from:
http://www.fao.org/docrep/018/i3144e/i3144e.pdf
17. Ford JD. Indigenous health and climate change. Am J Public Health. 2012;102(7):1260–6.
18. Keleman Saxena A, Cadima Fuentes X, Gonzales Herbas R, Humphries DL. Indigenous
Food Systems and Climate Change: Impacts of Climatic Shifts on the Production and
Processing of Native and Traditional Crops in the Bolivian Andes. Front Public Heal
[Internet]. 2016;4. Available from:
http://journal.frontiersin.org/Article/10.3389/fpubh.2016.00020/abstract
19. FAO. INDIGENOUS PEOPLES’ FOOD SYSTEMS: the many dimensions of culture, diversity
and environment for nutrition and health. 2009; Available from:
ftp://ftp.fao.org/docrep/fao/012/i0370e/i0370e.pdf
20. Reading CL, Wein F. Health Inequalities and Social Determinants of Aboriginal Peoples’
Health. Natl Collab Cent Aborig Heal [Internet]. 2009;1–47. Available from:
http://www.nccah-ccnsa.ca/docs/social determinates/nccah-loppie-wien_report.pdf
21. Coté C. “Indigenizing” Food Sovereignty. Revitalizing Indigenous Food Practices and
Ecological Knowledges in Canada and the United States. Humanities [Internet].
2016;5(3):57. Available from: http://www.mdpi.com/2076-0787/5/3/57
75
22. Skinner K, Hanning RM, Desjardins E, Tsuji LJS. Giving voice to food insecurity in a remote
indigenous community in subarctic Ontario, Canada: Traditional ways, ways to cope, ways
forward. BMC Public Health. 2013;13(1).
23. Elliott B, Jayatilaka D, Brown C, Varley L, Corbett KK. “We are not being heard”: Aboriginal
perspectives on traditional foods access and food security. Vol. 2012, Journal of
Environmental and Public Health. 2012.
24. Richmond CAM, Ross NA. The determinants of First Nation and Inuit health: A critical
population health approach. Heal Place. 2009;15(2):403–11.
25. Lemke S, Delormier T. Indigenous Peoples’ food systems, nutrition, and gender:
Conceptual and methodological considerations. Matern Child Nutr. 2017;13.
26. Hoover E, Cook K, Plain R, Sanchez K, Waghiyi V, Miller P, et al. Indigenous Peoples of
North America : Environmental Exposures and Reproductive Justice. Environ Health
Perspect. 2015;120(12):1645–9.
27. Ford JD, Macdonald JP, Huet C, Statham S, MacRury A. Food policy in the Canadian North:
Is there a role for country food markets? Soc Sci Med. 2016;152:35–40.
28. Council of Canadian Academies. Aboriginal Food Security in Northern Canada: An
Assessment of the State of Knowledge. 2014. 296 p.
29. Jernigan VBB, Huyser KR, Valdes J, Simonds VW. Food Insecurity Among American Indians
and Alaska Natives: A National Profile Using the Current Population Survey–Food Security
Supplement. J Hunger Environ Nutr [Internet]. 2017;12(1):1–10. Available from:
https://www.tandfonline.com/doi/full/10.1080/19320248.2016.1227750
30. Patterson K, Berrang-Ford L, Lwasa S, Namanya DB, Ford J, Twebaze F, et al. Seasonal
variation of food security among the Batwa of Kanungu, Uganda. Public Health Nutr.
2017;20(1):1–11.
31. Barber M, Jackson S, Dambacher J, Finn M. The persistence of subsistence: Qualitative
social-ecological modeling of indigenous aquatic hunting and gathering in tropical
Australia. Ecol Soc. 2015;20(1).
32. Furgal C, Seguin J. Climate change, health, and vulnerability in Canadian northern
Aboriginal communities. Environ Health Perspect. 2006;114(12):1964–70.
33. Lynn K, Daigle J, Hoffman J, Lake F, Michelle N, Ranco D, et al. The impacts of climate
change on tribal traditional foods. Clim Change. 2013;120(3):545–56.
76
34. Furgal C, Seguin J. Climate change, health, and vulnerability in Canadian northern
Aboriginal communities. Environ Health Perspect. 2006;114(12):1964–70.
35. Beddington JR, Asaduzzaman M, Clark ME, Bremauntz AF, Guillou MD, Jahn MM, et al.
The role for scientists in tackling food insecurity and climate change. Agric Food Secur.
2012;1(1).
36. FAO, IFAD, WFP. Monitoring Food Security and Nutrition in Support of the 2030 Agenda
for Sustainable Development: Taking stock and looking ahead. Rome; 2016.
37. Babu SC. Evidence-Informed Policymaking: Lessons from Food Security and Nutrition
Monitoring Systems during Food Crises. In: Sahn DE, editor. The Fight Against Hunger
and Malnutrition: The Role of Food, Agriculture, and Targeted Policies. OUP Oxford; 2015.
38. World Health Organization. Food and nutrition surveillance systems: Technical guide for
the development of a food and nutrition surveillance system. WHO Geneva; 2013.
39. Canadian Polar Comission. State of Environmental Monitoring in Northern Canada
[Internet]. 2015. Available from: https://www.canada.ca/en/polar-
knowledge/publications/cpc-stateofenv.html
40. High Level Task Force on the Global Food Security Crisis. Updated Comprehensive
Framework for Action. 2010; Available from:
http://www.fao.org/fileadmin/user_upload/ISFP/UCFA_Final.pdf
41. Jones AD, Ngure FM, Pelto G, Young SL. What Are We Assessing When We Measure Food
Security? A Compendium and Review of Current Metrics. Adv Nutr An Int Rev J [Internet].
2013;4(5):481–505. Available from:
http://advances.nutrition.org/cgi/doi/10.3945/an.113.004119
42. Ford JD, Berrang-Ford L, King M, Furgal C. Vulnerability of Aboriginal health systems in
Canada to climate change. Glob Environ Chang [Internet]. 2010;20(4):668–80. Available
from: http://dx.doi.org/10.1016/j.gloenvcha.2010.05.003
43. Smylie J, Anderson I, Ratima M, Crengle S, Anderson M. Indigenous health performance
measurement systems in Canada, Australia, and New Zealand. Lancet.
2006;367(9527):2029–31.
44. Anderson MJ, Smylie JK. Health Systems Performance Measurement Systems in Canada:
How Well do They Perform in First Nations, Inuit, and Métis Contexts? Pimatisiwin
[Internet]. 2009;7(1):99–115. Available from:
http://www.ncbi.nlm.nih.gov/pubmed/23450984%5Cnhttp://www.pubmedcentral.nih.gov/
77
articlerender.fcgi?artid=PMC3582648
45. Bilukha O, Prudhon C, Moloney G, Hailey P, Doledec D. Measuring anthropometric
indicators through nutrition surveillance in humanitarian settings: Options, issues, and
ways forward. Food Nutr Bull. 2012;33(2):169–76.
46. Statistics NC for H. US Household Food Security Survey Module: Six-Item Short Form.
Security. 2008.
47. Bickel G, Nord M, Price C, Hamilton W, J. C. Guide to measuring household food security,
revised 2000. Alexandria, VA; 2000.
48. Ebi KL, Del Barrio MO. Lessons learned on health adaptation to climate variability and
change: Experiences across low- and middle-income countries. Vol. 125, Environmental
Health Perspectives. 2017.
49. Ford JD, Berrang-Ford L, Lesnikowski A, Barrera M, Jody Heymann S. How to track
adaptation to climate change: A typology of approaches for national-level application.
Ecol Soc. 2013;18(3).
50. Pollock RM, Whitelaw GS. Community-based monitoring in support of local sustainability.
Local Environ. 2005;10(3):211–28.
51. Johnson N, Behe C, Danielsen F, Nickels S, Pulsifer PL. Community-Based Monitoring and
Indigenous Knowledge in a Changing Arctic : [Internet]. Available from:
http://www.inuitcircumpolar.com/uploads/3/0/5/4/30542564/cbm_final_report.pdf
52. Kipp A, Cunsolo A, Gillis D, Sawatzky A, Harper S. The Need for Community-led,
Integrated, and Innovative Monitoring Programs when Responding to the Health Impacts
of Climate Change. Int J Circumpolar Health. 2018;
53. Danielsen F, Jensen AE, Alviola PA, Balete DS, Mendoza M, Tagtag A, et al. Does
monitoring matter? A quantitative assessment of management decisions from locally-
based monitoring of protected areas. Biodivers Conserv. 2005;14(11):2633–52.
54. Berner J, Brubaker M, Revitch B, Kreummel E, Tcheripanoff M, Bell J. Adaptation in Arctic
circumpolar communities: Food and water security in a changing climate. Int J
Circumpolar Health. 2016;75.
55. Danielsen F, Topp-Jørgensen E, Levermann N, Løvstrøm P, Schiøtz M, Enghoff M, et al.
Counting what counts: using local knowledge to improve Arctic resource management.
Polar Geogr. 2014;37(1):69–91.
78
56. Farhan Ferrari M, de Jong C, Belohrad VS. Community-based monitoring and information
systems (CBMIS) in the context of the Convention on Biological Diversity (CBD).
Biodiversity. 2015;16(2–3):57–67.
57. Tengö M, Hill R, Malmer P, Raymond CM, Spierenburg M, Danielsen F, et al. Weaving
knowledge systems in IPBES, CBD and beyond—lessons learned for sustainability. Vols.
26–27, Current Opinion in Environmental Sustainability. 2017. p. 17–25.
58. Johnson N, Alessa L, Behe C, Danielsen F, Gearheard S, Gofman-Wallingford V, et al. The
contributions of Community-Based monitoring and traditional knowledge to Arctic
observing networks: Reflections on the state of the field. Arctic. 2015;68(5):1–13.
59. Ebi K., Semenza JC. Community-based adaptation to the health impacts of climate
change. Vol. 35, Community-based adaptation to the health impacts of climate change.
2008. p. SP-501-7.
60. Berrang-Ford L, Pearce T, Ford JD. Systematic review approaches for climate change
adaptation research. Vol. 15, Regional Environmental Change. 2015. p. 755–69.
61. Landis JR, Koch GG. The measurement of observer agreement for categorical data.
Biometrics. 1977;33(1):159–74.
62. Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology.
Implement Sci. 2010;5:69.
63. FAO. The State of Food Insecurity in the World 2001. Rome; 2001.
64. Habtezion S. Gender, climate change and food security. New York, NY; 2012.
65. Heidari S, Babor TF, De Castro P, Tort S, Curno M. Sex and gender equity in research:
rationale for the SAGER guidelines and recommended use. Gaceta Sanitaria. 2018;
66. Braun V, Clarke V. Using thematic analysis in psychology Using thematic analysis in
psychology. Qual Res Psychol. 2006;3(2):77–101.
67. Kouril D, Furgal C, Whillans T. Trends and key elements in community-based monitoring:
a systematic review of the literature with an emphasis on Arctic and Subarctic regions.
Environ Rev [Internet]. 2016;24(2):151–63. Available from:
http://www.nrcresearchpress.com/doi/10.1139/er-2015-0041
68. Danielsen F, Burgess ND, Balmford A, Donald PF, Funder M, Jones JPG, et al. Local
participation in natural resource monitoring: A characterization of approaches. Conserv
79
Biol. 2009;23(1):31–42.
69. Bishop-Williams K, Sargeant JM, Berrang-Ford L, Edge V, IHACC Research Team, Harper
SL. A protocol for a systematic narrative synthesis review: The global impact of
environment on acute respiratory infections in Indigenous populations. BMC Syst Rev
[Internet]. 2017;In preparation. Available from: http://dx.doi.org/10.1186/s13643-016-
0399-x
70. Tremblay M, Furgal C, Larrivee C, Annanack T, Tookalook P, Qiisik M, et al. Climate
Change in Northern Quebec: Adaptation Strategies from Community-Based Research.
Arctic. 2008;61:27–34.
71. Constantino P de AICarlos HSA, Ramalho EE, Rostant L, Marinelli CE, Teles D, et al.
Empowering local people through community-based resource monitoring: A comparison
of Brazil and Namibia. Vol. 17, Ecology and Society. 2012.
72. Parlee BL, Goddard E, First Nation ŁKD, Smith M. Tracking Change: Traditional Knowledge
and Monitoring of Wildlife Health in Northern Canada. Hum Dimens Wildl. 2014;19(1):47–
61.
73. Setty RS, Bawa K, Ticktin T, Gowda CM. Evaluation of a participatory resource monitoring
system for nontimber forest products: The case of Amla (Phyllanthus spp.) fruit harvest by
Soligas in South India. Ecol Soc. 2008;13(2).
74. Wesche S, Schuster RC, Tobin P, Dickson C, Matthiessen D, Graupe S, et al. Community-
based health research led by the Vuntut Gwitchin First Nation. Int J Circumpolar Health
[Internet]. 2011;70(4):396–406. Available from:
https://www.tandfonline.com/doi/full/10.3402/ijch.v70i4.17846
75. Driscoll DL, Mitchell E, Barker R, Johnston JM, Renes S. Assessing the health effects of
climate change in Alaska with community-based surveillance. Clim Change. 2016;137(3–
4):455–66.
76. Driscoll DL, Sunbury T, Johnston J, Renes S. Initial findings from the implementation of a
community-based sentinel surveillance system to assess the health effects of climate
change in Alaska. Circumpolar Heal Suppl [Internet]. 2013;72:946–54. Available from:
http://10.0.13.74/ijch.v72i0.21405%5Cnhttp://search.ebscohost.com/login.aspx?direct=tru
e&AuthType=ip,url,cookie,uid&db=aph&AN=109175649&site=ehost-live&scope=site
77. McClymont Peace D, Myers E. Community-based participatory process--climate change
and health adaptation program for Northern First Nations and Inuit in Canada. Int J
Circumpolar Health. 2012;71(0):1–8.
80
78. Ens EJ, Towler GM, Daniels C. Looking back to move forward: Collaborative ecological
monitoring in remote Arnhem Land. Ecol Manag Restor. 2012;13(1):26–35.
79. Kofinas G, Lyver POB, Russell D, White R, Nelson A, Flanders N. Towards a protocol for
community monitoring of caribou body condition. Rangifer. 2003;Special Is:29–38.
80. Oviedo AFP, Bursztyn M. Community-based monitoring of small-scale fisheries with
digital devices in Brazilian Amazon. Fish Manag Ecol. 2017;24(4):320–9.
81. Sheil D, Boissière M, Beaudoin G. Unseen sentinels: local monitoring and control in
conservation’s blind spots. Ecol Soc [Internet]. 2015;20(2). Available from:
http://www.ecologyandsociety.org/vol20/iss2/art39/
82. Cummings A, Cummings G, Hamer E, Moses P, Norman Z, Captain V, et al. Developing a
UAV-Based Monitoring Program with Indigenous Peoples. J Unmanned Veh Syst.
2017;5(4):115–25.
83. Herrmann TM, Sandström P, Granqvist K, D’Astous N, Vannar J, Asselin H, et al. Effects of
mining on reindeer/caribou populations and indigenous livelihoods: Community-based
monitoring by Sami reindeer herders in Sweden and First Nations in Canada. Polar J.
2014;4(1):28–51.
84. Shaffer CA, Milstein MS, Yukuma C, Marawanaru E, Suse P. Sustainability and
comanagement of subsistence hunting in an indigenous reserve in Guyana. Conserv Biol.
2017;31(5):1119–31.
85. Van Vliet N, Sandrin F, Vanegas L, L’haridon L, Fa JE, Nasi R. High-tech participatory
monitoring in aid of adaptive hunting management in the Amazon. Unasylva.
2017;68(249):53–62.
86. Luzar JB, Silvius KM, Overman H, Giery ST, Read JM, Fragoso JM V. Large-scale
Environmental Monitoring by Indigenous Peoples. Bioscience [Internet]. 2011;61(10):771–
81. Available from: https://academic.oup.com/bioscience/article-
lookup/doi/10.1525/bio.2011.61.10.7
87. de Paula M, Xerente V, Pezzuti J. Hunting and monitoring: Community-based research in
xerente indigenous land, Brazilian Cerrado. Hum Ecol Rev [Internet]. 2017;23(1):23–44.
Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-
85038815827&doi=10.22459%2FHER.23.01.2017.02&partnerID=40&md5=f35ffbb57e5cf9
6acc97f18422006b8f
88. Mateo-Vega J, Potvin C, Monteza J, Bacorizo J, Barrigón J, Barrigón R, et al. Full and
81
effective participation of indigenous peoples in forest monitoring for reducing emissions
from deforestation and forest degradation (REDD+): Trial in Panama’s Darién. Ecosphere.
2017;8(2).
89. Gill H, Lantz T. A Community-Based Approach to Mapping Gwich’in Observations of
Environmental Changes in the Lower Peel River Watershed, NT. J Ethnobiol [Internet].
2014;34(3):294–314. Available from: http://www.bioone.org/doi/abs/10.2993/0278-0771-
34.3.294
90. Constantino P de AL, Fortini LB, Kaxinawa FRS, Kaxinawa AM, Kaxinawa ES, Kaxinawa AP,
et al. Indigenous collaborative research for wildlife management in Amazonia: The case of
the Kaxinawá, Acre, Brazil. Biol Conserv. 2008;141(11):2718–29.
91. Brook RK, Kutz SJ, Veitch AM, Popko RA, Elkin BT, Guthrie G. Fostering community-based
wildlife health monitoring and research in the Canadian North. Ecohealth. 2009;6(2):266–
78.
92. Van Rijsoort J, Jinfeng Z. Participatory resource monitoring as a means for promoting
social change in Yunnan, China. Biodivers Conserv. 2005;14(11):2543–73.
93. Danielsen F, Pirhofer-Walzl K, Adrian TP, Kapijimpanga DR, Burgess ND, Jensen PM, et al.
Linking public participation in scientific research to the indicators and needs of
international environmental agreements. Conserv Lett. 2014;7(1):12–24.
94. Anisimov OA, Vaughan DG, Callaghan TV, Furgal C, Marchant H, Prowse TD, et al. Polar
regions (Arctic and Antarctic). In: Intergovernmental Panel on Climate Change Working
Group. 2007. p. 653–85.
95. Nordic Development Fund. Gender equality study: A study on addressing gender aspects
in projects co-financed by NDF in developing countries. Helsinki; 2004.
96. Fisher S. Tracking Adaptation and Measuring Development through a gender lens.
London, UK; 2014.
97. Bunce A, Ford J. How is adaptation, resilience, and vulnerability research engaging with
gender? Vol. 10, Environmental Research Letters. 2015.
98. Brant Castellano M. Ethics of Aboriginal Research.Pdf. J Aborig Heal. 2004;1(January):98–
114.
99. Smith LT. Decolonizing Methodologies. New York. 1999. 28-74 p.
82
100. Inuit Tapiriit Kanatami. National Inuit Strategy on Research [Internet]. 2018. Available
from: www.itk.ca
101. Ford JD, Cameron L, Rubis J, Maillet M, Nakashima D, Willox AC, et al. Including
indigenous knowledge and experience in IPCC assessment reports. Vol. 6, Nature Climate
Change. 2016. p. 349–53.
102. Ford JD, Vanderbilt W, Berrang-Ford L. Authorship in IPCC AR5 and its implications for
content: Climate change and Indigenous populations in WGII. Clim Change.
2012;113(2):201–13.
103. Castleden H, Morgan VS, Neimanis A. Researchers’ Perspectives on Collective/Community
Co-Authorship in Community-Based Participatory Indigenous Research. J Empir Res Hum
Res Ethics [Internet]. 2010;5(4):23–32. Available from:
http://journals.sagepub.com/doi/10.1525/jer.2010.5.4.23
104. Green D, Raygorodetsky G. Indigenous knowledge of a changing climate. Clim Change.
2010;100(2):239–42.
105. Castleden HE, Martin D, Cunsolo A, Harper S, Hart C, Sylvestre P, et al. Implementing
Indigenous and Western Knowledge Systems (Part 2): “You Have to Take a Backseat” and
Abandon the Arrogance of Expertise. Int Indig Policy J [Internet]. 2017;8`(4). Available
from: https://ir.lib.uwo.ca/iipj/vol8/iss4/8/
106. Gratani M, Royee F, Butler JRA. A research process and criteria indicators framework for
developing indigenous freshwater ecosystem health monitoring. Cogent Environ Sci
[Internet]. 2016;2(1):1–14. Available from:
https://www.cogentoa.com/article/10.1080/23311843.2016.1214228
107. Tengö M, Brondizio ES, Elmqvist T, Malmer P, Spierenburg M. Connecting diverse
knowledge systems for enhanced ecosystem governance: The multiple evidence base
approach. Vol. 43, Ambio. 2014. p. 579–91.
108. UNFCCC. Adoption of the Paris Agreement [Internet]. Paris; 2015. Available from:
http://www.un.org/ga/search/view_doc.asp?symbol=FCCC/CP/2015/L.9/Rev.1&Lang=E
109. Chandler M, See L, Copas K, Bonde AMZ, López BC, Danielsen F, et al. Contribution of
citizen science towards international biodiversity monitoring. Biol Conserv. 2017;213:280–
94.
110. Wiseman ND, Bardsley DK. Monitoring to Learn, Learning to Monitor: A Critical Analysis of
Opportunities for Indigenous Community-Based Monitoring of Environmental Change in
83
Australian Rangelands. Geogr Res. 2016;54(1):52–71.
111. Staddon SC, Nightingale A, Shrestha SK. The Social Nature of Participatory Ecological
Monitoring. Soc Nat Resour. 2014;27(9):899–914.
112. David-Chavez DM, Gavin MC. A global assessment of Indigenous community engagement
in climate research. Environ Res Lett. 2018;13:123005.
113. Bardsley D, Rogers G. Prioritizing Engagement for Sustainable Adaptation to Climate
Change: An Example from Natural Resource Management in South Australia. Soc Nat
Resour. 2011;24(1):1–17.
114. Fisher S, Dinshaw A, Mcgray H, Rai N, Schaar J. Evaluating Climate Change Adaptation:
Learning From Methods in International Development. New Dir Eval. 2015;2015(147):13–
35.
115. Hedger M, Greeley M, Leavy J. Evaluating Climate Change : Pro-Poor Perspectives. Inst
Dev Stud. 2012;39(4):75–80.
116. Brooks N, Anderson S, Ayers J, Burton I, Tellam I. Tracking adaptation and measuring
development [Internet]. Climate Change. 2011. Available from:
http://pubs.iied.org/pdfs/10031IIED.pdf
117. Ford JD, Berrang-Ford L. The 4Cs of adaptation tracking: consistency, comparability,
comprehensiveness, coherency. Vol. 21, Mitigation and Adaptation Strategies for Global
Change. 2016. p. 839–59.
118. Ford JD, Sherman M, Berrang-Ford L, Llanos A, Carcamo C, Harper S, et al. Preparing for
the health impacts of climate change in Indigenous communities: The role of community-
based adaptation. Glob Environ Chang. 2018;49.
119. Sheil D, Lawrence A. Tropical biologists, local people and conservation: New opportunities
for collaboration. Vol. 19, Trends in Ecology and Evolution. 2004. p. 634–8.
120. Constantino PAL. Dynamics of hunting territories and prey distribution in Amazonian
Indigenous Lands. Appl Geogr. 2015;56:222–31.
121. Campbell BM, Vermeulen SJ, Aggarwal PK, Corner-Dolloff C, Girvetz E, Loboguerrero AM,
et al. Reducing risks to food security from climate change. Glob Food Sec. 2016;11:0–1.
122. Skinner K, Pratley E, Burnett K. Eating in the City: A Review of the Literature on Food
Insecurity and Indigenous People Living in Urban Spaces. Societies [Internet]. 2016;6(2):7.
84
Available from: http://www.mdpi.com/2075-4698/6/2/7
123. Neufeld HT, Richmond CAM, Health Access Centre SOA. Impacts of place and social
spaces on traditional food systems in southwestern Ontario. Int J Indig Heal [Internet].
2017;12(1):93. Available from: https://journals.uvic.ca/index.php/ijih/article/view/16903
124. Douxchamps S, Debevec L, Giordano M, Barron J. Monitoring and evaluation of climate
resilience for agricultural development – A review of currently available tools. World Dev
Perspect. 2017;5:10–23.
125. Schipper ELF, Langston L. A comparative overview of resilience measurement frameworks
analysing indicators and approaches. Overseas Dev Inst - Work Pap 422 [Internet].
2015;(July):30. Available from:
www.odi.org%5Cnwww.odi.org/facebook%5Cnwww.odi.org/twitter
126. Tendall DM, Joerin J, Kopainsky B, Edwards P, Shreck A, Le QB, et al. Food system
resilience: Defining the concept. Vol. 6, Global Food Security. 2015. p. 17–23.
127. IISD. Climate Resilience and Food Security A framework for planning and monitoring.
2013;(June):29.
128. Sturgess P. Measuring Resilience. UK; 2016.
129. Ebhuoma EE, Simatele DM. “We know our Terrain”: indigenous knowledge preferred to
scientific systems of weather forecasting in the Delta State of Nigeria. Climate and
Development. 2005;1–12.
130. Smith LT. On tricky ground: Researching the native in the age of uncertainty. In: The SAGE
handbook of qualitative research. 3rd ed. Thousand Oaks, CA: Sage; 2005. p. 85–107.
131. IPCC. Summary for Policymakers. In: V. Masson-Delmotte, P. Zhai, H. O. Pörtner, D.
Roberts, J. Skea PRS, A. Pirani, W. Moufouma-Okia, C. Péan, R. Pidcock, S. Connors, J. B. R.
Matthews, Y. Chen, X. Zhou MIG, E. Lonnoy, T. Maycock, M. Tignor TW, editors. Global
warming of 15°C An IPCC Special Report on the impacts of global warming of 15°C above
pre-industrial levels and related global greenhouse gas emission pathways, in the context
of strengthening the global response to the threat of climate change,. Geneva,
Switzerland: World Meteorological Organization; 2018. p. 32.
85
CHAPTER 3: HOW ARE CLIMATE ACTIONS EVALUATED? A REVIEW OF UNITED NATIONS FOOD SECURITY EVALUATIONS
Abstract
Established in Article 14 of the Paris Agreement, countries must evaluate progress toward
climate adaptation, mitigation, and financial goals via the 2023 global stocktake. Yet, the extent
of climate integration into evaluation in response to the rapidly approaching stocktake mandate
remains unclear. Many United Nations agencies recognize the importance of mainstreaming
climate action into food security and broader development programming. Our goal is to
examine how United Nations food security programs are evaluating climate action. Using a
systematic search and screening process, we included evaluations that described any aspect of
food security, evaluated completed programs, and were published between 2014 and 2019. We
examined the extent of climate change mainstreaming into food security evaluation by
developing and applying an assessment rubric. We also identified, compared, and characterized
evaluation challenges and opportunities using thematic analysis. Of the 136 relevant evaluations,
69% (n = 94) assessed food security programs that integrated climate action. While many
evaluations reported adaptation outcomes, considerations of climate action in the evaluation
approach were often insufficient. Based on our analysis, challenges to climate evaluation
included inadequate resource allocation, weak monitoring and evaluation systems, dependency
on food security evaluation, and limited climate focus in programs and evaluations. The
86
assessment rubric provides a tool for understanding and encouraging climate change
integration into evaluation, and through this, helps countries prepare for the global stocktake.
3.1 Introduction
The Paris Agreement’s global goal on climate change adaptation (UNFCCC, 2015) raises a
fundamental question: How do we monitor and report on adaptation globally? The Agreement
has established that a global stocktake will be conducted in 2023, and every five years thereafter
to assess the collective progress in achieving the purpose of the Agreement (Article 14). To do
so, the global stocktake will consider “mitigation, adaptation, and the means of implementation
and support, and in the light of equity and the best available science” (UNFCCC, 2015, p. 18).
Although countries are asked to provide information on progress toward adaptation targets,
how countries track, monitor, and evaluate such progress is not prescribed by the Agreement.
The Agreement also does not provide a definition of adaptation, and continued debates about
the definition of adaptation create difficulties in monitoring and evaluating climate actions (i.e.
climate change mitigation and/or adaptation) (Tompkins et al., 2018). Furthermore, while
accepted methods to monitor migration progress exist (e.g. through national greenhouse gas
inventories), a comparable accepted method to document adaptation baselines does not yet
exist.
Assessing how people and countries are adapting is particularly important for programs aiming
to enhance food security (Conevska et al., 2019). Changes in temperature, increased extreme
87
weather events, and increased rainfall variability are already threatening food production, food
access, food safety, food stability, nutrition, and environmental sustainability (Fujimori et al.,
2018; Mbow et al., 2019; Ray et al., 2019). Yet, evaluating the success of climate-resilient food
security programs is challenging due to trade-offs between mitigation, adaptation, and food
security (Denton et al., 2015; Sherman & Ford, 2014). For example, some forms of agricultural
intensification (e.g. monoculture systems) aiming to maintain supply under climate change may
also increase vulnerabilities to climate extremes (Mbow et al., 2019). Furthermore, standard food
security monitoring and evaluation (M&E) is typically not designed to track the resilience of
food systems, defined as “the capacity over time of a food system and its units at multiple levels,
to provide sufficient, appropriate and accessible food to all, in the face of various and even
unforeseen disturbances” (Tendall et al., 2015, p. 19). Similar challenges have been noted more
broadly for tracking climate change adaptation progress (Berrang-Ford et al., 2019; Ford et al.,
2013; Tompkins et al., 2018).
Navigating trade-offs between climate change mitigation, climate change adaptation, and food
security measures in programming require processes for learning from integrated climate and
food security efforts (Fisher et al., 2015; Mbow et al., 2019). Investing in climate change M&E is
important for learning from climate change adaptation efforts in support of food security and
help countries meet the reporting requirements of the Paris Agreement (FAO et al., 2018; Lam et
al., 2019; Uitto et al., 2016). Evaluators are increasingly recognizing the threat climate change
poses to sustainable development efforts and thus the need to consider climate change in
88
evaluation (Rowe, 2019; Ssekamatte, 2018; Uitto 2019). Communities of practice have been
formed to foster evaluative thinking around climate (e.g. Earth-Eval) and food security (e.g.
EvalForward). However, there is a gap in conceptualizing the intersection of climate change and
food security in evaluation.
Because the climate crisis transcends national boundaries, intergovernmental organizations play
a key role in coordinating climate actions across countries. The United Nations (UN), for
example, promotes system-wide platforms for climate action (e.g. United Nations Framework
Convention on Climate Change; UNFCCC), food security (e.g. Committee on World Food
Security), and evaluation (e.g. United Nations Evaluation Group; UNEG). Many UN agencies have
recognized the importance of climate action mainstreaming, broadly referring to processes for
integrating climate change considerations into development policies, programs, and/or
individual actions (Somanathan et al., 2014; UNDP, 2011; USAID, 2015). The operationalization of
climate action mainstreaming, particularly adaptation, tends to be diverse, taking place across
levels (e.g. national, local) and sectors (e.g. environmental, risk reduction planning) (Runhaar et
al., 2018). Decisions surrounding the extent to which mainstreaming should be conducted is also
complex, and influenced by factors including the development policy objective, scale, and
duration. For example, a “development-first” approach that focuses on integrating climate
considerations into existing planning processes might be more appropriate for projects with a
short planning cycle (Kim et al., 2017); conversely, a “climate-first” approach focused on
comprehensive risk management of vulnerabilities to climate variability and change might be
89
more appropriate for programs implemented across a longer time horizon. Learning from
practices and experiences documented in UN program evaluation reports may provide insights
into assessing and improving the climate-resilience of food security programs and evaluations.
Given the pressing global stocktake mandate, the impacts of climate change on food security
programming, and the climate responsibilities of UN agencies, this study asks: how are climate
actions evaluated in UN food security contexts? The objectives are: 1) to assess the extent to
which climate action is mainstreamed into UN food security evaluation reports including
variations across the food security topic (e.g. access, availability), climate change focus (e.g.
adaptation, mitigation), geographic region, and time; and, 2) to identify, compare, and
characterize the challenges and opportunities for evaluating climate action embedded within
food security programs.
3.2 Methods
3.2.1 Search strategy
We adopted a systematic evaluation approach to examine publicly available information on UN
food security evaluations. Specifically, we searched the UN Evaluation Group (UNEG) database
for evaluations using a systematic search and screening process (Colquhoun et al., 2014). The
UNEG database contains publicly available evaluation reports of its 47 UNEG members, totaling
over 10,000 reports (UNEG, 2016). UNEG members are encouraged to submit UN evaluations
into the database, and in doing so, support the use of evaluation reports beyond their primary
audience in the UN agencies that commissioned them. As UNEG members are encouraged to
90
meet UNEG Norms and Standards5 for evaluation; this helps to ensure that a reasonable level of
quality and consistency among evaluation reports is achieved.
We searched the UNEG database using the term ‘food’. Evaluation reports that were completed
between 2014 and 2019 were selected. We selected reports published in the past six years to
allow for an in-depth analysis of present-day evaluations completed following the 2014
Intergovernmental Panel on Climate Change (IPCC) Fifth Assessment Report which called for
M&E strategies for managing climate impacts and reducing risks (Denton et al., 2015). To
capture UN evaluation reports that may not be indexed in the database, we used the same
database search protocol and searched the online evaluation repositories of UN entities selected
based on their membership in the High-Level Task Force on Global Food and Nutrition Security6.
Initial searches were conducted on 10 January 2019 and updated on 17 February 2020.
3.2.2 Relevance screening
A two-step screening strategy was employed. First, the titles and summaries of evaluation
reports were screened for relevance. Next, all reports deemed potentially relevant went through
a review of the full-text to confirm relevance. In each step, reports were screened according to a
5 UNEG has 14 norms (e.g. utility, credibility, independence) and five standards (e.g. institutional framework
management of the evaluation function, evaluation competencies) for evaluation. 6 UN entities searched included World Food Programme (WFP), Food and Agriculture Organization (FAO),
International Fund for Agricultural Development (IFAD), World Meteorological Organization (WMO), United Nations
Children’s Fund (UNICEF), United Nations Industrial Development Organization (UNIDO), International Labour
Organization (ILO), United Nations Environment Programme (UNEP), United Nations High Comissioner for Refugees
(UNHCR), and United Nations Development Programme (UNDP) (United Nations, 2015).
91
priori inclusion criteria. Reports were included if they generally referred to a food security pillar7,
pillar synonym, or aspect of food security8. Moreover, only final evaluations were included to
ensure sufficient information was available for synthesis purposes; midterm evaluations, country
portfolio evaluations, evaluation summaries, monitoring exercises, and reviews were excluded.
All citations were imported into a spreadsheet (Excel 2013, Microsoft Corporation, Redmond,
WA). To ensure rigour and reduce bias in the screening process, all screening steps were
conducted by two independent reviewers; any conflicts were discussed and reconciled regularly
throughout the review process. The level of reviewer agreement was measured using a kappa
score (McHugh, 2012).
3.2.3 Data extraction, classification, and analysis
A charting form was developed to collect key characteristics of each evaluation report including
the year of evaluation completion, UN entity, budget, evaluation methodology, and program
budget, scale, duration, and geographic region.
We classified evaluation reports into different categories of pathways toward food security
based on program descriptions and outcomes (Table 3.1). We developed these categories of
pathways by an inductive thematic analysis of evaluation reports captured in this review (Table
7 Four pillars are commonly referred to when explaining the concept of food security: availability, access, utilization,
and stability (FAO, 2001). 8 Food security is defined as “a situation that exists when all people, at all times, have physical, social and economic
access to sufficient, safe and nutritious food that meets their dietary needs and food preferences for an active and
healthy life” (FAO, 2001, p. 49).
92
1). We also classified evaluation reports into one of six climate change categories based on
program descriptions and outcomes. Climate change categories were adapted from the five
climate change categories developed by Donner et al. (2016) by replacing “interdeterminate
based on the information available in the database” with “climate-change impacts,” as well as
adding “both adaptation and mitigation”. Because climate change has multi-faceted and
temporal impacts on all dimensions of food security, articles discussing food security pathways
such as food production, food access, and infrastructure could be categorized as having used
climate action measures. The urgency of climate change might place a focus of climate change
in food security programs. For example, against the backdrop of Myanmar’s vulnerability to
climate-related hazards (e.g. flash floods, drought, and erratic rainfall), a food assistance
program aimed to improve the preparedness for and mitigation of disasters and climate change
(WFP, 2016). The program prioritized emergency relief (e.g. cash, food) and asset creation (e.g.
soil conservation, watershed management) in disaster-prone areas.
93
Table 3.1. Food security and climate change classification system applied to United Nations
evaluation reports.
Categories Description of pathways or actions
Food security*
Food production Pathway that directly supports food production and productivity, such as the
provision of seeds, fertilizer, and land.
Food access Pathway that meets immediate food needs, such as school feeding, food
assistance, and food relief programs.
Infrastructure Pathway to improve infrastructure including roads, flood management,
irrigation, and/or storage facilities.
Market-based Pathway that enhances markets for farmers, link consumers to markets, provide
access to financial services, and/or support business development.
Capacity building Pathway that trains extension workers, farmers, researchers, and policymakers in
food production and/or managing risks to food security.
Food security
monitoring system
Pathway to track the food security situation on an ongoing basis over time.
Nutrition Pathway addressing malnutrition, including overnutrition, undernutrition, and
associated conditions such as stunting.
Climate change
Explicit adaptation Actions to reduce vulnerability to the effects of climate change. To be
considered under this category, the report had to explicitly mention
“adaptation.”
Implicit adaptation A wide range of actions that can reduce societal vulnerability to external
stresses like climate events (e.g. disaster preparedness), or adapt to a particular
range of projected climate outcomes (e.g. coastal protective measures).
Evaluations categorized as ‘implicit’ adaptation generally described, but did not
explicitly mention, adaptation.
Mitigation Actions taken to reduce greenhouse gas emissions or greenhouse gas
concentrations.
Both adaptation and
mitigation
Actions taken to adapt to (explicitly or implicitly) and mitigate climate change.
Climate change
impacts
Actions to capture how climate change is affecting food security and/or the
programming context without referencing adaptation or mitigation.
Climate absent The climate change relevance of the program was not articulated.
* Categories are not mutually exclusive and do overlap; for example, programs aiming to improve food
access can relate to nutrition by focusing on access to nutritious foods.
We examined the extent to which climate change was integrated into evaluations using an
assessment rubric (Table 3.2). The rubric was intended to provide a high-level assessment of
94
climate action considerations. The rubric could be modified to capture greater detail, depending
on the needs and goals of the assessment. For example, the rating scale in evaluation scope
(question b) could have a maximum score of three to account for studies mentioning all three
climate change considerations (e.g. adaptation, mitigation, impact). We developed the rubric
based on a series of questions organized around three general components of evaluation:
• Evaluation scope: using a two-point (0-2) rating scale, we assessed climate change
considerations in the evaluation report introduction and questions. We assessed how the
evaluation report described the context of climate risks, vulnerabilities, and impacts.
Furthermore, we assessed whether the report described how climate change might
impact (or be impacted by) the program.
• Evaluation approach: using a one-point (0-1) rating scale, we assessed whether climate
change was mentioned in the evaluation theory, methodology, methods, and data
analysis, as well as whether the report described how it intended to assess the context-
specific climate change challenges. We used lower weighting for the evaluation
approach considering frameworks for evaluating outcomes of climate action only
recently emerged (Uitto et al., 2016).
• Evaluation results: using a three-point (0-3) rating scale, we assessed climate change
considerations in the evaluation findings, conclusions, and recommendations. We
assessed whether the reports demonstrated a link between the identified climate change
risks, vulnerabilities, and impacts and the program activities, outcomes, and impacts.
95
A climate action integration index was then developed and calculated by summing scores for
component indices using equal weighting to reflect the equal importance of each component in
the evaluation framework (Lamhauge et al. 2012). This index was calculated on a six-point scale,
building on similar evaluation methodologies in the climate change field (Bunce & Ford, 2015;
Sherman & Ford, 2014). Based on this six-point scale, we categorized the 94 reports (69% of all
evaluations) integrating climate action as having high, moderate, or low levels of climate change
considerations in evaluation. To be classified as high level, reports had to score between five to
six points; moderate levels scored between three to four points; and low levels scored between
one and two points. We used descriptive statistics to present the characteristics of evaluations
and examine the extent to which climate action was integrated into evaluation.
96
Table 3.2. Assessment rubric for climate action integration into United Nations evaluation
reports.
Criteria and guiding questions Scoring system
Evaluation scope Total possible score: 2
a. Does the introduction of the evaluation acknowledge a climate change
issue(s)?
Yes: score of 1
No: score of 0
b. Does the evaluation include an objective/question/criterion specific to
the assessment of climate change adaptation, mitigation, and/or impacts?
Yes: score of 1
No: score of 0
Evaluation approach Total possible score: 1
a. Is climate change adaptation, mitigation, and/or impacts mentioned in
the evaluation theory, methodology, methods, and/or analysis?
Yes: score of 1
No: score of 0
Evaluation results Total possible score: 3
a. Does the findings section provide information on climate change
adaptation, mitigation, and/or impacts?
Yes: score of 1
No: score of 0
b. Does the conclusion provide information on climate change adaptation,
mitigation, and/or impacts?
Yes: score of 1
No: score of 0
c. Are there specific recommendations to address climate change
adaptation, mitigation, and/or impacts?
Yes: score of 1
No: score of 0
3.2.4 Thematic analysis: examining evaluation challenges and opportunities
We used a hybrid deductive-inductive approach (Fereday & Muir-Cochrane, 2006) to conduct
thematic analysis (Braun et al., 2018). Thematic analysis involved identifying patterns or themes
in evaluation reports that captured challenges and opportunities for evaluating climate action in
food security programs. We deductively coded segments of text using an analytical tool that we
developed (Appendix 4). In this tool, we defined the following three themes: context, approach,
and opportunities/challenges. These themes were derived from literature on M&E of climate
action and food security (Bours et al., 2014; Fisher et al., 2015; Ford et al., 2013). During this
initial stage, further themes were inductively developed to accommodate data that could not be
coded into one of the predetermined themes. To ensure validity, weekly discussions among
97
team members (SL, WD, SLH) were held to co-develop, define, expand, collapse, and finalize the
themes. The final codebook, comprising all codes along with their descriptions and examples,
was applied to all the evaluation reports (Appendix 5 and 6). To better understand the
relationship between themes of climate-responsive evaluation challenges and opportunities, we
developed a thematic map (Appendix 7). NVivo© qualitative analysis software was used to
facilitate the coding of text segments, development of themes, and organization of quotations
(QSR International, Version 12).
3.3 Results
After the removal of duplicates and non-relevant reports, and the addition of reports from the
targeted search of UN websites, a total of 133 relevant reports were identified (Appendix 8).
Several reports documented more than one evaluation study, resulting in 136 unique
evaluations that were reviewed in this paper (Figure 3.1). The kappa inter-rater reliability score
for the title/summary screening was 0.70, indicating “good agreement” (McHugh, 2012). A
summary of the descriptive characteristics of these evaluations is shown in Table 3.3.
98
Figure 3.1. Flow chart illustrating the selection of United Nations food security evaluations (n =
136 unique evaluations within 133 reports) published from 2014 to 2019.
99
Table 3.3. Descriptive characteristics of United Nations food security evaluations (n = 136
evaluations) published from 2014 to 2019.
Characteristics No. %
Year of publication:
2014 23 16.9
2015 13 9.6
2016 27 19.9
2017 27 19.9
2018 29 21.3
2019 17 12.5
Geographic region: a
Africa 71 52.2
Asia 61 44.9
North America 8 5.9
South America 7 5.1
Europe 3 2.2
Australia and Oceania 3 2.2
Programming scale:
National 79 58.1
Provincial/regional 33 24.3
Local 24 17.6
United Nations agency:
Food and Agriculture Organization 44 32.4
International Fund for Agricultural Development 41 30.1
World Food Programme 27 19.8
United Nations Children’s Fund 11 8.1
United Nations Development Programme 9 6.6
United Nations Environment Programme 4 2.9
Evaluation methodology:
Qualitative 90 66.2
Mixed qualitative and quantitative 45 33.1
Quantitative 1 0.7
Food security pathway:
Capacity building 136 100
Food production 86 63.2
Nutrition 42 30.9
Market access 41 30.1
Infrastructure 32 16.9
Food access 23 16.9
Information systems 17 12.5
Climate change focus:
Implicit adaptation 46 33.8
Climate absent 42 30.9
100
Explicit adaptation 24 17.6
Climate change impacts 16 11.8
Both adaptation and mitigation 7 5.1
Mitigation 1 0.7 a Numbers may not sum to 136 as some evaluations were conducted across multiple geographic regions.
3.3.1 Climate action mainstreaming into food security evaluation
From 2014 to 2019, six UN agencies published evaluations of completed food security
programs. In general, evaluations were conducted for accountability (i.e. assessing performance
and results of the operation) and learning (i.e. identifying the reasons why certain results
occurred) purposes. The average budget of programs was 36.8 million USD (range: 0.7 – 435
million USD) and the median duration was four years (range: 1 – 24 years). Programs were
conducted in 83 countries, primarily in African and Asian regions. Most UN food security
evaluations (69%; n = 94) integrated climate action with no clear trend over time (Figure 3.2).
Among the evaluations that integrated climate action, most focused primarily on adaptation
either implicitly (34%; n = 46) or explicitly (18%, n = 24). The proportion of evaluations focused
on implicit adaptation and climate change impacts increased over time while explicit adaptation
and mitigation decreased (Figure 3.3). The proportion of evaluations focused on both
adaptation and mitigation increased until 2019, when the proportion dropped back down to
2014 levels.
101
Figure 3.2. Number of United Nations food security evaluations (n = 136) with a climate focus
published from 2014 to 2019.
0
10
20
30
40
50
60
70
80
90
2014 2015 2016 2017 2018 2019
Num
ber
of
eval
uat
ions
Year
Climate present Climate absent
102
Figure 3.3. Proportion of United Nations food security evaluations from 2014 to 2019 focused
on implicit adaptation (a), explicit adaptation (b), mitigation (c), both adaptation and mitigation
(d), and climate change impacts (e).
103
3.3.2 Climate considerations in food security evaluation was superficial
Within food security pathways, programs most likely to score medium or high climate action
integration were those focused on improving food access, followed by market access,
infrastructure, food productivity, capacity building, information systems, and nutrition. In terms
of climate change foci, programs focused on mitigation were more likely to score medium or
high climate action integration followed by both adaptation and mitigation, explicit adaptation,
implicit adaptation, climate change impacts, and climate absent. The median duration of
programs with a mitigation component was five years which is slightly higher than the median
program duration of four years, suggesting mitigation might be more relevant in programs with
longer time horizons. Geographically, programs most often scoring medium or high climate
action integration were those conducted in North America, followed by South America,
Australia/Oceania, Africa, Asia, and Europe.
Of the evaluations that considered climate action, the integration of climate differed within
components of an evaluation (Appendix 9). For example, one-quarter of evaluations (25%, n =
34) scored highest for the results component of the assessment tool by reporting climate-
related findings, conclusions, and recommendations. However, most evaluation studies did not
account for climate action in the evaluation scope or approach. For instance, 126 evaluations
(93%) did not elaborate on how climate action was evaluated. Over 66% (n = 90) of evaluations
were classified as low level of climate action integration, followed by 24% (n=32) as medium
104
integration, and 10% (n=14) as high integration. There was a slight decrease in low and high
climate integration with a slight increase in medium climate integration over the six years (2014-
2019) (Figure 3.4).
Figure 3.4. Extent of climate action integration into United Nations food security evaluations
from 2014 to 2019.
3.3.3 High scoring food security evaluations integrated climate considerations into the evaluation scope, approach, and results
The extent of climate action integration in UN evaluations differed by UN agency, with UNDP
scoring highest on average (3.3/6), followed by WFP (2.4), FAO (2.3/6), IFAD (1.9/6), UNEP
(1.25/6), and UNICEF (0.7/6). The different climate action mandates of UN agencies (Appendix
10) could help explain the varying extents of climate action integration in UN evaluations. For
0
10
20
30
40
50
60
70
80
90
100
2014 2015 2016 2017 2018 2019
Pro
po
rtio
n o
f ev
aluat
ions
(%)
Year
Low climate action integration Medium climate action integration High climate action integration
105
example, UNICEF aims to integrate environmental sustainability as a cross-cutting issue without
specifying climate action. In contrast, UNDP ensures all its programming considers climate and
disaster risk. As such, explicit commitments to mainstream climate action into all programs
could contribute to higher average climate action integration scores.
There does not appear to be a clear relationship between median program duration and
average climate change integration score of individual UN agencies (Appendix 11). The median
program duration across all levels of climate action integration was four years, suggesting no
relationship between the different levels of integration and program durations. Articles
published between 2017 and 2019 tended to have slightly higher average climate change
integration scores (2.7) compared to those published between 2014 and 2016 (2.5), suggesting
an increase in climate change integration over time.
Except for UNICEF, all UN agencies had at least one report scoring high in climate action
integration. For example, one UNDP evaluation report covering a resilience capacity building
program in the Democratic Republic of the Congo described the risks of climate change to food
security, conducted a literature review focused on climate change, and reported changes to
climate resilience of food production systems (UNDP, 2015). In a WFP evaluation report covering
a relief and recovery program in Zimbabwe, the context introduced the impacts of climate
change on agricultural production and evaluation questions examined evidence that
interventions were designed to address unpredictable drought (WFP, 2014g). Promising climate
106
evaluations included a climate risk assessment, an explicit strategy for measuring climate action
outcomes, and a dedicated overview of climate action outcomes, while climate-absent
evaluations did not mention climate in the context or methods (Appendix 12). A key finding of
our study was a weakness in climate assessment methodologies, casting doubt on the
robustness of the climate-related outcomes presented.
3.3.4 Low scoring food security evaluations integrated climate considerations into the evaluation results only
All UN agencies had at least one evaluation scoring in both the medium and low categories in
climate action integration. Evaluations scoring medium often reported risks posed by climate
change to the program and presented results related to climate change; however, they tended
not to elaborate on the climate assessment methodology. For example, one evaluation report
described Guatemala’s climate vulnerability and context and assessed changes in resilience to
climate hazards from a food-for-assets program; however, the report did not describe the
methodological approach for linking adaptation results with adaptation efforts (WFP, 2014a).
Low scoring evaluations generally either: 1) reported risks climate change posed to the program
without presenting climate-related findings, or 2) presented climate-related findings without
providing an overview of how climate change influenced the program context. Like medium
scoring evaluations, low scoring evaluations did not elaborate on how climate action was
assessed.
107
3.3.5 Challenges and opportunities for evaluating climate action
Evaluations across all UN agencies reported challenges of existing M&E systems and
implications of these challenges on evaluation resources, food security assessment, and climate
action assessment (Figure 3.5 and Supplementary file Figure 1). M&E systems were often
reported as “weak,” “underperforming,” or “unsatisfactory” (FAO, 2016; IFAD, 2014b, 2015b),
resulting in the “absence of a well-articulated theory of change,” “lack of outcome and even
output level data,” or “poor selection of indicators” (IFAD, 2016b, 2017c, 2018a). Without a clear
theory of change, for example, it was difficult to accurately assess the impact pathways leading
to adaptation and food security results (UNEP, 2016). When data were available, in many cases
they were “inconsistent” or “poor” to meaningfully inform evaluative judgments on the success
of food security programs (FAO, 2017b; WFP, 2016).
108
Figure 3.5. Challenges and opportunities for assessing climate action as reported in United
Nations food security evaluations from 2014 to 2019.
109
Challenge 1: Weak M&E systems with implications for climate evaluation
According to evaluations, a weak M&E system often did not allow changes to climate resilience
to be measured or attributed to the food security program (WFP, 2014c, p. 43). Typically, food
security indicators such as food consumption, dietary diversity, and coping strategies were
collected without taking into account “the occasional contextual events that may have affected
the regularity of the interventions or the households’ food security (i.e. floods)” (WFP, 2018, p.
26). Furthermore, the absence of “well-defined,” “quantifiable,” and “operational” indicators
(IFAD, 2016c, 2018a, 2018d) often prevented climate-related assessments. For example, one
evaluation assessing an agricultural greenhouse gas mitigation program across six countries
(South Africa, Indonesia, Columbia, Ecuador, Costa Rica, and the Democratic Republic of the
Congo) reported, “Through the information that the Evaluation Team gathered, it is not known if
the training was delivered effectively and that it strengthened the capacity of national compilers
to gather and analyze climate change data” (FAO, 2014b, p. 16). This quotation underscores the
importance of designing climate adaptation and mitigation indicators to capture climate-related
outcomes of food security programs.
Challenge 2: Inadequate resources for climate evaluation
Because evaluation often draws on existing data from M&E systems, the quality of an evaluation
can depend on the quality of M&E systems. As one evaluation report covering a livestock
development program in Laos explained, “The joint evaluation faced a number of challenges in
the assessment of both effectiveness and rural poverty impact which were mainly due to the
110
weaknesses of project-level M&E system” (IFAD, 2018a, p. 6), highlighting the role of M&E
systems in facilitating evaluation. Including climate action in the evaluation scope was also
constrained by limited resources such as human, capacity, time, funding, and logistics (FAO,
2017c; WFP, 2018). In low resource contexts, extreme weather events posed further challenges
to the evaluation process. For example, one evaluation report describing a relief and recovery
program highlighted:
It needs to be mentioned that Madagascar is not only a large country but also that it is
highly complex, with difficult political conditions, security constraints, an enormous agro-
ecological variety, and both quick-onset (cyclones) and slow-onset (droughts, harvest
failure due to locusts) disasters (WFP, 2014d, p. 2, emphasis added).
Challenge 3: Limited food security programming focused on climate action
When climate action was not an objective of the food security program, climate action was often
overlooked in the M&E stage. The lack of climate action in food security programming
contributed to limited climate-related activities, outputs, and outcomes to support evaluative
judgement. As one evaluation report covering a program in Vietnam explained, “There was also
no indication of contribution to climate change adaptation over the life of the Pro-Poor
Partnerships for Agroforestry Development (3PAD) project, even if the awareness and reporting
on severe weather events and the negative effects on crops and livestock increased over the
3PAD project lifetime” (IFAD, 2018b, p. 32). As such, an evaluation will not uncover meaningful
climate-related outcomes without climate action being integrated into the program. Several
111
evaluations reported avoiding assessing climate action entirely, demonstrating a broader
implication of a lack of climate action integration in programming. For example, one evaluation
reporting on a livestock development program in Laos stated,
This [adaptation] evaluation criterion concerns the contribution of the project to increase
climate resilience and beneficiaries’ capacity to manage short- and long-term climate
risks. The project did not specifically contribute to this objective. As a consequence, this
criterion is not rated (IFAD, 2018a, p. 36).
Challenge 4: Dependency on food security evaluation
In several evaluations, extreme weather events linked to climate change impacted food security
evaluation. In Algeria, endline surveys were delayed “to avoid biased outcomes” due to the
impact of unexpected, flash floods “on key determinants of malnutrition” (WFP, 2018b, p. 29),
highlighting the influence of severe floods on food security measurement tools. Some climate
evaluations were also influenced by the multi-faceted nature of food security programs. For
example, in a multi-sectoral project with multiple interventions that aimed to promote the use of
sustainable land management across 17 countries, attributing progress in sustainable
agriculture, biodiversity conservation, and climate change adaptation to any one particular
intervention was reported to be challenging (UNEP, 2015). Furthermore, food security evaluation
was often constrained by the “absence of monitoring data”, “the issue of attribution”, and “food
security and agricultural production not [being] the explicit focuses of the program” (IFAD,
2016a, 2017b; WFP, 2014b), posing challenges to climate evaluation. Climate action was difficult
112
to assess when data surrounding indicators (e.g. changes to food security status) were not
adequately captured.
Opportunity 1: Build a foundation for climate evaluation by investing in M&E systems
Many evaluations emphasized the importance of building a strong M&E foundation by
strengthening capacity, specifically analytical, project management, and M&E skills (IFAD, 2015a,
2017a; UNICEF, 2018); these foundational skills lay the groundwork for evaluations to integrate
climate change considerations. Some evaluations also suggested M&E systems focus on
adaptation outcomes (e.g. reducing food insecurity) over outputs and activities (e.g. number of
trainings held) to capture observed changes to adaptive capacity (FAO, 2014a). To invest in M&E
systems, reports recommended to budget for an explicit climate change evaluation criterion in
the evaluation design (FAO, 2018c) and to allocate funds to evaluations generally (IFAD, 2019).
Opportunity 2: Prioritize climate action in food security evaluation
Although a program may not have explicitly included climate change adaptation in the program
design, some activities may have contributed to climate change adaptation (IFAD, 2018c, p. 24).
An evaluation of a rural development program in Morocco reported, “Nothing was specifically
done to improve resilience to climate change” (IFAD, 2014c, p. 19). The authors added, “it is
likely that the activities to protect natural resources, especially water, soil and vegetation cover,
directly or indirectly contributed to resilience” (IFAD, 2014c, p. 19), highlighting the promise of
climate evaluation in capturing climate co-benefits. In an evaluation of an agroforestry
113
development project, the report identified climate action trade-offs, stating, “Climate change
adaptation is rated less positively: attention was focused on climate change mitigation through
the forestry sector, not climate change adaptation to improve the resilience of poor farmers and
communities” (IFAD, 2018d, p. 33). While some UN agencies have developed approaches for
measuring resilience (e.g. WFP 2014d), evaluation reports generally considered climate action as
a cross-cutting theme or stand-alone evaluation question. Furthermore, they often assessed the
ways the program was aligned with climate-related frameworks, policies, or strategies including
the Sustainable Development Goal 13 (climate action) (FAO, 2018b), national climate change
strategies (FAO, 2018c; IFAD, 2018c; WFP, 2014c), UN Development Assistance Frameworks
(UNDP, 2016; WFP, 2014d), UN country strategies (FAO, 2018a; IFAD, 2018c; UNEP, 2015; WFP,
2014f), and the UNFCCC (FAO, 2017a).
Opportunity 3: Prioritize climate action in food security programming
All UN agencies explored in this study had at least one evaluation emphasizing the importance
of integrating climate action into food security programming. In the context of a relief and
recovery program in Zimbabwe, increasing focus on resilience programming has been
recommended despite “a perception… that this kind of programming is somewhat outside
WFP’s core mandate” (WFP, 2014c, p. 48). Several evaluations also highlighted the need to
integrate climate into program theory. For example, one report covering a program in Kenya
recommended “a more robust design process that sets out an appropriate long-term theory of
change for target communities” (WFP, 2014c, p. 47), emphasizing the importance of showing the
114
theory behind how the program will contribute to the overall goal of food security resilience to
climate change.
3.4 Discussion
The rapidly approaching 2023 global climate change stocktake makes it important to clarify the
ways of assessing progress toward collective climate change goals. Considering the support of
many UN agencies in climate action mainstreaming into food security and development
planning, this paper systematically examines UN food security program evaluation reports.
While many evaluations focused on climate change adaptation programs, few assessed
programs focused on mitigation, which is a missed opportunity considering agriculture, forestry,
and other types of land use account for 23% of human greenhouse gas emissions (Mbow et al.,
2019). Importantly, evaluations with a mitigation focus were more likely to score medium to
high climate action integration. Both mitigation and adaptation – as well as the mitigation-
adaptation nexus – are critical for achieving food security (Babiker et al., 2018; Richardson et al.,
2018).
Although changing, the extent of climate action integration into food security evaluation
remained low over the six years. Progress toward climate action mainstreaming into evaluation
is slow despite global calls (e.g. the Paris Agreement, the Sendai Framework for Disaster Risk
Reduction, the 2030 Agenda for Sustainable Development) for international cooperation on
adaptation and mainstreaming of climate action into development planning. While many
evaluations integrated climate action into the evaluation findings, conclusions, and
115
recommendations, the methodological dimensions were often neglected, thus preventing
thorough considerations of climate action in evaluation data collection and analysis. Our
assessment rubric offers a tool to illuminate the extent of climate action mainstreaming into
food security evaluation and the possibilities in applying climate action mainstreaming to help
countries prepare for the global adaptation stocktake.
Mandating the assessment of program contributions to climate resilience was encouraged by
many evaluations, highlighting opportunities to integrate climate action into the evaluation
scope. For example, IFAD defined a new adaptation criterion in an updated evaluation manual
as: “The contribution of the project to reducing the negative impacts of climate change through
dedicated adaptation or risk reduction measures” (IFAD, 2016d, p. 38). IFAD also offered core
questions to guide the evaluation such as: “To what extent did the program demonstrate
awareness and analysis of current risks?” (pg. 42). Commonly used criteria for evaluating climate
policy included goal achievement, efficiency, and cost-effectiveness (Huitema et al., 2011). IFAD
was the first to mandate an explicit criterion, paving a way forward for encouraging climate
evaluation of all programs. As investments in preparing for climate change increase, developing
and applying quality assurance systems can help support climate-responsiveness of evaluation
(Carbon, 2017).
While our 6-question assessment tool did not specify the collection of qualitative data, we found
the qualitative analysis of evaluation challenges and opportunities to be useful for
contextualizing assessment findings. We recommend having a comments section beside the
116
scoring section to accommodate the extraction of relevant qualitative information, helping to
explain why a particular rating was given. Furthermore, the tool did not specify the collection of
quantitative data. Additional information related to the program, such as duration, is important
to support the analysis of the scoring. Our assessment also covered UN evaluation reports
conducted over the past six years, which was a time-intensive undertaking. However, this
approach does not require primary data collection and makes effective use of evaluation data
already systematically collected by UN agencies. Practitioners might consider applying this tool
on a more periodic basis to understand the current status of climate action integration; then,
compare findings every five years to understand trends in climate action integration over time.
3.4.1 Implications for research and practice
There was a slight decrease in low and high climate integration in evaluations, with a slight
increase in medium climate integration from 2014 to 2019. In response to a growing and
pressing climate change stocktake mandate, further integration of climate change into
evaluation is needed. As evidenced by the reviewed evaluation reports, the integration of
climate change can involve adding climate action as an explicit evaluation objective, question, or
criterion. Such framing showed value by highlighting the implications of climate change,
revealing trade-offs and co-benefits, and encouraging climate action in future programming.
Several assessment frameworks, guidebooks, and reporting formats have also been developed
over the last decade to support this integration. For example, adaptation tracking, a
subcomponent of M&E, seeks to systematically identify, characterize, and compare adaptation
117
across nations over time (Ford & Berrang-Ford, 2016). The Tracking Adaptation and Measuring
Development is a conceptual framework to monitor and evaluate climate change adaptation
within and across sectors and levels of programming (Brooks, et al., 2011). The German
Cooperation for International Cooperation and partners have developed an impact evaluation
guidebook for climate change adaptation projects (GIZ, 2015). To guide the tracking of policies
that address mitigation, the World Resources Institute developed a framework (Barua et al.,
2014). Several scholars have synthesized tools, frameworks, and approaches for evaluating
adaptation (Bours et al., 2014; Fisher et al., 2015; Ford et al., 2013). More engagement from
evaluators with these theoretical developments is important for informing climate evaluation;
reviewing the peer-reviewed literature and grey literature on research on evaluation related to
climate evaluation might support this process. Most evaluations mentioned challenges of M&E
more generally rather than specific challenges of climate evaluation; yet, climate evaluation
challenges were extensively discussed in the published literature. We expanded the existing
literature by identifying practical challenges of adaptation assessment including limited
prioritization of climate action in programming and dependency on food security evaluation.
Based on our review, a gap in evaluation practice is the reporting on mitigation measures and
outcomes. Additionally, future research is needed to further develop adaptation indicators,
criteria, and frameworks, particularly those that are grounded in the realities and experiences of
organizations that commissioned the evaluation. Evaluations attuned to the needs, priorities,
and constraints of organizations are important for developing food security programs and
118
evaluations responsive to the impacts of climate change (Bours et al., 2014).
3.4.2 Limitations
We acknowledge several limitations of our approach. Firstly, the extent of climate action
integration into evaluation was assessed based on the information presented in the evaluation
report. As with any report, many topics might have been considered during the evaluation
process but not reported on; as such, climate change might have been considered in the
evaluation but not documented. Reflecting on and sharing the decision-making process is
important for providing insights into evaluating climate action. Secondly, the different
terminology for climate action used in evaluation reports made it challenging to classify the
climate considerations of evaluations; clarity in the definition of climate action would support
future climate assessments. Nevertheless, the review is comprehensive and provides in-depth
insights into how UN food security programs are responding to climate change from 2014 to
2019.
3.5 Conclusion
Climate evaluation is a growing area of research and practice in food security and broader
development contexts. This interest is in response to the growing impacts of climate change on
food security globally, public investments made in climate action, and the need to demonstrate
progress toward collective climate change goals. We show through our assessment of UN food
security evaluation reports published over the past six years (2014-2019) that progress toward
mainstreaming climate action in evaluation is slow, particularly for climate change mitigation.
119
The limited quality and quantity of data from existing M&E systems create barriers to producing
evaluative judgments around climate action. The integration of climate into the evaluation scope
and approach is an important opportunity for contextualizing, prioritizing, and guiding the
assessment of climate action. Our assessment rubric offers a tool to understand and encourage
climate action mainstreaming into food security evaluation. In doing so, the rubric can help
strengthen the evidence-base of climate-related food security programming, informing
processes such as the upcoming 2023 global climate change stocktake.
120
3.6 References
Babiker, M., Bertoldi, P., Buckeridge, M., Cartwright, A., Araos Maldives, M., Bakker, S., Bazaz, A.,
Belfer, E., Benton, T., de Coninck, H., Revi, A., Babiker, M., Bertoldi, P., Buckeridge, M.,
Cartwright, A., Dong, W., Ford, J., Fuss, S., Hourcade, J., … Waterfield, T. (2018). Chapter 4 -
Strengthening and Implementing the Global Response. Global Warming of 1.5°C. An IPCC
Special Report on the Impacts of Global Warming of 1.5°C above Pre-Industrial Levels and
Related Global Greenhouse Gas Emission Pathways, in the Context of Strengthening the
Global Response to the Threat of Climate Change. Cambridge University Press. Cambridge.
Barua, P., Fransen, T., Wood, D. (2014). CLIMATE POLICY IMPLEMENTATION TRACKING
FRAMEWORK. World Resources Institute. Washington, DC.
Berrang-Ford, L., Biesbroek, R., Ford, J. D., Lesnikowski, A., Tanabe, A., Wang, F. M., Chen, C., Hsu,
A., Hellmann, J. J., Pringle, P., Grecequet, M., Amado, J. C., Huq, S., Lwasa, S., & Heymann, S.
J. (2019). Tracking global climate change adaptation among governments. In Nature
Climate Change. (Vol. 9, pp. 440-449).
Bours, D., McGinn, C., & Pringle, P. (2014). Monitoring & evaluation for climate change
adaptation and resilience : A synthesis of tools , frameworks and approaches. SEA Change
CoP, Phnom Penh and UKCIP, May, 1–67. Oxford. Phnom Penh.
Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2018). Thematic analysis. In P. Liamputtong (Ed.),
Handbook of Research Methods in Health Social Sciences. Springer. Singapore.
Brooks, N., Anderson, S., Ayers, J., Burton, I., & Tellam, I. (2011). Tracking adaptation and
measuring development. IIED. London.
Bunce, A., & Ford, J. (2015). How is adaptation, resilience, and vulnerability research engaging
with gender? In Environmental Research Letters (Vol. 10, Issue 12:123003).
Carbon, M. (2017). An Analytical Framework for Evaluating a Diverse Climate Change Portfolio. In
Evaluating Climate Change Action for Sustainable Development. Springer Nature. Cham.
Colquhoun, H. L., Levac, D., O’Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M., & Moher,
D. (2014). Scoping reviews: Time for clarity in definition, methods, and reporting. In Journal
of Clinical Epidemiology (Vol. 67, Issue 12, pp. 1291–1294).
Conevska, A., Ford, J., Lesnikowski, A., & Harper, S. (2019). Adaptation financing for projects
focused on food systems through the UNFCCC. Climate Policy. (Vol. 19, Issue 1, pp. 43-58).
121
Denton, F., Wilbanks, T. J., Abeysinghe, A. C., Burton, I., Gao, Q., Lemos, M. C., Masui, T., O’Brien,
K. L., Warner, K., Bhadwal, S., Leal, W., Van Ypersele, J. P., & Wright, S. B. (2015). Climate-
resilient pathways: Adaptation, mitigation, and sustainable development. In Climate Change
2014 Impacts, Adaptation and Vulnerability: Part A: Global and Sectoral Aspects. Cambridge
University Press. Cambridge.
Donner, S. D., Kandlikar, M., & Webber, S. (2016). Measuring and tracking the flow of climate
change adaptation aid to the developing world. Environmental Research Letters, 11(5), 1–9.
https://doi.org/10.1088/1748-9326/11/5/054006
FAO, IFAD, UNICEF, W. and W. (2018). The state of food security and nutrition in the world.
FAO. (2001). The State of Food Insecurity in the World 2001. FAO. Rome.
FAO. (2014a). End of project evaluation report of the Programme for Improvement of Irrigation
Systems and Construction of Micro-hydro Power Facilities in Kabul and Bamyan Provinces.
FAO. Rome.
FAO. (2014b). Final Evaluation of “Monitoring and Assessment of GHG Emissions and Mitigation
Potentials in Agriculture Project” - GCP /GLO/286/GER. FAO. Rome.
FAO. (2016). Final evaluation of the Improved Global Governance for Hunger Reduction
Programme. FAO. Rome.
FAO. (2017a). Evaluation of the Benefit-sharing Fund second project cycle. FAO. Rome.
FAO. (2017b). Final evaluation of Sudan Food Security Policy and Strategy Capacity Building
programme. FAO. Rome.
FAO. (2017c). Final Evaluation of the project “Forestry and Protected Area Management in Fiji,
Samoa, Vanuatu and Niue (GEFPAS-FPAM).” FAO. Rome.
FAO. (2018a). Final Evaluation of the Project “Increased Household Food, Income and Nutrition
Security through Commercialization of an Integrated and Sustainable Smallholder Livestock
Sector in Zimbabwe.” FAO. Rome.
FAO. (2018b). Final evaluation of the project “Mainstreaming the use and conservation of
agrobiodiversity in public policy through integrated strategies and in situ implementation in
four Andean Highlands provinces.” FAO. Rome.
FAO. (2018c). Final Evaluation of the Project “Strengthening Climate Change Resilience and
Disaster Risk Reduction in Agriculture to Improve Food Security in Haiti After the
122
Earthquake.” FAO. Rome.
Fereday, J., & Muir-Cochrane, E. (2006). Demonstrating Rigor Using Thematic Analysis : A Hybrid
Approach of Inductive and Deductive Coding and Theme Development. International
Journal of Qualitative Methods, 5, 80–92. https://doi.org/10.1063/1.2011295
Fisher, S., Dinshaw, A., Mcgray, H., Rai, N., & Schaar, J. (2015). Evaluating Climate Change
Adaptation: Learning From Methods in International Development. New Directions for
Evaluation, 2015(147), 13–35. https://doi.org/10.1002/ev.20128
Ford, J. D., & Berrang-Ford, L. (2016). The 4Cs of adaptation tracking: consistency, comparability,
comprehensiveness, coherency. In Mitigation and Adaptation Strategies for Global Change
(Vol. 21, Issue 6, pp. 839–859). https://doi.org/10.1007/s11027-014-9627-7
Ford, J. D., Berrang-Ford, L., Lesnikowski, A., Barrera, M., & Jody Heymann, S. (2013). How to
track adaptation to climate change: A typology of approaches for national-level application.
Ecology and Society, 18(3):40. https://doi.org/10.5751/ES-05732-180340
Fujimori, S., Hasegawa, T., Rogelj, J., Su, X., Havlik, P., Krey, V., Takahashi, K., & Riahi, K. (2018).
Inclusive climate change mitigation and food security policy under 1.5 ◦ C climate goal.
Environmental Research Letters, 13(7): e074033. https://doi.org/10.1088/1748-9326/aad0f7
GIZ. (2015). Impact Evaluation Guidebook for Climate Change Adaptation Projects. GIZ. Bonn.
Huitema, D., Jordan, A., Massey, E., Rayner, T., van Asselt, H., Haug, C., Hildingsson, R., Monni, S.,
& Stripple, J. (2011). The evaluation of climate policy: Theory and emerging practice in
Europe. Policy Sciences, 44:179-198. https://doi.org/10.1007/s11077-011-9125-7
IFAD. (2014a). Kingdom of Morocco Rural Development Project in the Mountain Zones of Al-Haouz
Province. IFAD. Rome.
IFAD. (2014b). Republic of the Sudan Gash Sustainable Livelihoods Regeneration Project. IFAD.
Rome.
IFAD. (2015a). Lao People’s Democratic Republic Rural Livelihoods Improvement Programme in
Attapeu and Sayabouri. IFAD. Rome.
IFAD. (2015b). Republic of Turkey Sivas-Erzincan Development Project. IFAD. Rome.
IFAD. (2016a). Federal Republic of Nigeria Community-Based Agricultural and Rural Development
Programme. IFAD. Rome.
123
IFAD. (2016b). Kyrgyz Republic Agricultural Investments and Services Project. IFAD. Rome.
IFAD. (2016c). Republic of the Philippines Rural Microenterprise Promotion Programme. IFAD.
Rome.
IFAD. (2016d). Evaluation manual. IFAD. Rome.
IFAD. (2017a). Arab Republic of Egypt West Noubaria Rural Development Project. IFAD. Rome.
IFAD. (2017b). Kingdom of Lesotho Rural Financial Intermediation Programme Project
Performance Evaluation. IFAD. Rome.
IFAD. (2017c). Republic of Malawi Rural Livelihoods Support Programme. IFAD. Rome.
IFAD. (2018a). Lao People’s Democratic Republic Northern Region Sustainable Livelihoods through
Livestock Development Project. IFAD. Rome.
IFAD. (2018b). Republic of Guyana Rural Enterprise and Agricultural Development Project. IFAD.
Rome.
IFAD. (2018c). Republic of Kenya Smallholder Horticulture Marketing Programme. IFAD. Rome.
IFAD. (2018d). Socialist Republic of Viet Nam Pro-Poor Partnerships for Agroforestry Development
Project. IFAD. Rome.
IFAD. (2019). Democratic Socialist Republic of Sri Lanka Smallholder Plantations Entrepreneurship
Development Programme. IFAD. Rome.
Kim, Y., Smith, J. B., Mack, C., Cook, J., Furlow, J., Njinga, J. L., & Cote, M. (2017). A perspective on
climate-resilient development and national adaptation planning based on USAID’s
experience. Climate and Development, 9(2):141-151.
https://doi.org/10.1080/17565529.2015.1124037
Lam, S., Dodd, W., Skinner, K., Papadopoulos, A., Zivot, C., Ford, J., Garcia, P. J., & Harper, S. L.
(2019). Community-based monitoring of Indigenous food security in a changing climate:
Global trends and future directions. Environmental Research Letters, 14(7): 073002.
https://doi.org/10.1088/1748-9326/ab13e4
Lamhauge, N., Lanzi, E., & Agrawala, S. (2012). Monitoring and evaluation for adaptation: Lessons
from development cooperation agencies. OECD. Paris.
Mbow, C., Rosenzweig, C., Barioni, L. G., Benton, T. G., Herrero, M., Krishnapillai, M., Liwenga, E.,
Pradhan, P., Rivera-Ferre, M. G., Sapkota, T., Tubiello, F. N., & Xu, Y. (2019). Food Security. In
124
P. Z. P.R. Shukla, J. Skea, E. Calvo Buendia, V. Masson-Delmotte, H.-O. Pörtner, D.C. Roberts,
J. P. P. R. Slade, S. Connors, R. van Diemen, M. Ferrat, E. Haughey, S. Luz, S. Neogi, M.
Pathak, J. Petzold, & J. M. P. Vyas, E. Huntley, K. Kissick, M. Belkacemi (Eds.), Climate Change
and Land: an IPCC special report on climate change, desertification, land degradation,
sustainable land management, food security, and greenhouse gas fluxes in terrestrial
ecosystems. Cambridge University Press. Cambridge.
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276–282.
https://doi.org/10.11613/BM.2012.031
Ray, D. K., West, P. C., Clark, M., Gerber, J. S., Prishchepov, A. V., & Chatterjee, S. (2019). Climate
change has likely already affected global food production. PLoS ONE, 14(5): e0217148.
https://doi.org/10.1371/journal.pone.0217148
Richardson, K. J., Lewis, K. H., Krishnamurthy, P. K., Kent, C., Wiltshire, A. J., & Hanlon, H. M.
(2018). Food security outcomes under a changing climate: impacts of mitigation and
adaptation on vulnerability to food insecurity. Climatic Change, 147(1–2), 327–341.
https://doi.org/10.1007/s10584-018-2137-y
Rowe, A. (2019). Sustainability-Ready Evaluation: A Call to Action. New Directions for Evaluation,
162: 29–48. https://doi.org/10.1002/ev.20365
Runhaar, H., Wilk, B., Persson, Å., Uittenbroek, C., & Wamsler, C. (2018). Mainstreaming climate
adaptation: taking stock about “what works” from empirical research worldwide. Regional
Environmental Change. 18:1201-1210. https://doi.org/10.1007/s10113-017-1259-5
Sherman, M. H., & Ford, J. (2014). Stakeholder engagement in adaptation interventions: an
evaluation of projects in developing nations. Climate Policy, 14(3):417–441.
https://doi.org/10.1080/14693062.2014.859501
Somanathan, E., Sterner, T., Sugiyama, D., Chimanikire, N., Dubash, J., Essandoh-Yeddu, S., Fifita,
L., Goulder, A., Labanndeira, S., Managi, C., Mitchell, J., Montero, P., Teng, F., & Zylicz, T.
(2014). National and Sub-national Policies and Institutions. Cambridge University Press.
Cambridge.
Ssekamatte, D. (2018). The role of monitoring and evaluation in climate change mitigation and
adaptation interventions in developing countries. African Evaluation Journal, 6(1): a254.
https://doi.org/10.4102/aej.v6i1.254
Tendall, D. M., Joerin, J., Kopainsky, B., Edwards, P., Shreck, A., Le, Q. B., Kruetli, P., Grant, M., &
Six, J. (2015). Food system resilience: Defining the concept. In Global Food Security (Vol. 6,
pp. 17–23). https://doi.org/10.1016/j.gfs.2015.08.001
125
Tompkins, E. L., Vincent, K., Nicholls, R. J., & Suckall, N. (2018). Documenting the state of
adaptation for the global stocktake of the Paris Agreement. Wiley Interdisciplinary Reviews:
Climate Change, 9(5), 1–9. https://doi.org/10.1002/wcc.545
Uitto, J.I., Puri, J., & van den Berg, R. D. (2016). Evaluating Climate Change Action for Sustainable
Development. Springer. Cham.
Uitto, Juha I. (2019). Sustainable Development Evaluation: Understanding the Nexus of Natural
and Human Systems. New Directions for Evaluation, 2019(162), 49–67.
https://doi.org/10.1002/ev.20364
UNDP. (2011). Mainstreaming Climate Change in National Development Processes and UN
Country Programming: A guide to assist UN Country Teams in integrating climate change
risks and opportunities. UNDP. New York.
UNDP. (2015). FINAL EVALUATION OF THE PROJECT “BUILDING THE CAPACITY OF THE
AGRICULTURE SECTOR IN DR CONGO TO PLAN FOR AND RESPOND TO THE ADDITIONAL
THREATS POSED BY CLIMATE CHANGE ON FOOD PRODUCTION AND SECURITY” OR
PANA-ASA PROJECT DR CONGO. UNDP. New York.
UNDP. (2016). TERMINAL EVALUATION REPORT: ENHANCING ADOPTION OF CLIMATE SMART
AGRICULTURE PRACTICES IN UGANDA‘S FARMING SYSTEMS. UNDP. New York.
UNEP. (2015). Terminal Evaluation of the UNEP/GEF Project “A Global Initiative on Landscapes for
People, Food and Nature.”UNEP. Kenya.
UNEP. (2016). Terminal Evaluation of the UNDA 7th Tranche funded UNEP project “Capacity
Building in National Planning for Food Security.” UNEP. Kenya.
UNFCCC. (2015). Paris Agreement. Accessed 6 June 2020. Available from
https://unfccc.int/sites/default/files/english_paris_agreement.pdf
UNICEF. (2018). Evaluation of Community-based Management of Acute Malnutrition Programme
supported by UNICEF in DPR Korea. UNICEF. New York.
United Nations. (2015). High level task force on global food and nutrition security (HLTF).
Accessed 15 June 2020. Available from http://www.un.org/en/issues/food/taskforce/
United Nations Evaluation Group. (2016). Norms and standards for evaluation. UNEG. New York.
USAID. (2015). INTEGRATING CLIMATE CHANGE INTO USAID ACTIVITIES. USAID. Arlington.
126
WFP. (2014a). Evaluation of the Impact of Food for Assets on Livelihood Resilience in Guatemala.
WFP. Rome.
WFP. (2014b). Evaluation of the Impact of Food for Assets on Livelihood Resilience in Uganda.
WFP. Rome.
WFP. (2014c). Evaluation of the Impact of Food for Assets on Livelihood Resilience in Senegal
(2005 – 2010). WFP. Rome.
WFP. (2014d). Kenya, PRRO 200174, Food Assistance to Refugees: An Evaluation of WFP’s
Operation (2011-2013). WFP. Rome.
WFP. (2014e). MADAGASCAR, PROTRACTED RELIEF AND RECOVERY (PRRO) 200065 “RESPONSE
TO RECURRENT NATURAL DISASTERS AND SEASONAL FOOD INSECURITY IN
MADAGASCAR.” WFP. Rome.
WFP. (2014f). Philippines, Protracted Relief and Recovery Operation PRRO 200296. WFP. Rome.
WFP. (2014g). Zimbabwe, Protracted Relief and Recovery Operation 200453 “Responding to
Humanitarian Needs and Strengthening Resilience to Food Insecurity” (September 2012–
March 2014). WFP. Rome.
WFP. (2016). Myanmar Protracted Relief and Recovery Operation 200299: “Supporting Transition
by Reducing Food Insecurity and Undernutrition Among the Most Vulnerable.” WFP. Rome.
WFP. (2018). Evaluation of the Nutrition Components of the Algeria PRRO 200301. WFP. Rome.
127
CHAPTER 4: HOW AND WHY ARE THEORY OF CHANGE AND REALIST EVALUATION USED IN FOOD SECURITY CONTEXTS? A SCOPING REVIEW
Abstract
The complex ways in which food security actions lead to nutrition and other health outcomes
make it important to clarify what programs work and how, with theory-driven evaluation
emerging as a promising approach to evaluate complex programs. However, it is unclear how
and why theory-driven evaluation is applied in food security contexts. Our objective is to
examine the development and use of Theory of Change and Realist Evaluation to support food
security programs globally. Using a systematic search and screening process, we included
studies that described a food security program, used a Theory of Change or Realist Evaluation,
and presented original research or evaluations. We found a total of 59 relevant Theory of
Change studies and eight Realist Evaluation studies. Based on our analysis, Theories of Change
arose in response to three main problems: 1) the need to evaluate under complexity; 2)
challenges with evaluation; and, 3) information gaps surrounding a program. In contrast, Realist
Evaluation was reported to be developed primarily to understand a program’s outcomes.
Reflecting on the problem to be addressed in the evaluation might help improve
understandings of the evaluation context, which might then inform the choice and design of an
evaluation approach.
128
4.1 Introduction
Food security programs play a crucial role in improving nutrition, reducing poverty, and
strengthening resilience to climate change, particularly for rural communities in low resource
settings (Mbow et al., 2019). Such programs often combine a variety of components, including
food assistance, nutritional education, and agricultural technology and training (Grace et al.,
2019). While progress is being made in understanding the impact of food security programs
(Bird et al., 2019; Fiorella et al., 2016), the causal pathways leading to impact are often unclear
(Thornton et al., 2017; Lam et al., 2021). This limitation reflects not only the lack of evaluation
studies that go beyond assessing a program’s results, but also the challenges in attributing
observed outcomes to the food security program. Yet, unraveling the complexities of how food
security actions transform into nutrition and other health outcomes is important for describing,
replicating, and scaling-up effective strategies (Jones et al., 2013).
An important tool in international development and agricultural research programs is theory-
driven evaluation (Johnson et al., 2015), which aims to develop a “program theory” of how a
program works and how the various components interact to produce outcomes (Salter &
Kothari, 2014). Two approaches to theory-driven evaluation gaining popularity in evaluating
complex programs are Theory of Change (ToC) and Realist Evaluation (RE) (Blamey & Mackenzie,
2007; Rolfe, 2019). ToC encourages stakeholders to describe the causal pathways and the
underlying assumptions of programs implemented in the context of social change (Mayne &
Johnson, 2015). RE explores the mechanisms that are likely to operate in social programs, the
129
contexts in which they might operate, and the outcomes that will be observed if they operate as
expected (Westhorp 2014). ToCs are increasingly mainstreamed into evaluations of food security
programs (Maru et al., 2018), whereas the use of RE is relatively nascent (Marchal et al., 2012).
Theory-driven evaluation has proven in many cases to be useful (Breuer et al., 2016). However,
the uncritical adoption of theory-driven evaluations opens for unanticipated consequences that
might undermine their value. The main concerns over the use of ToCs are under-contributing
theoretical knowledge and overlooking complexity (Lam, 2020). For REs, there are risks of
suboptimal analyses due to the lack of clarity on what exactly must be done to conduct an RE
(Jagosh, 2019). Despite these challenges and given the value of ToCs and REs, many agricultural
research and development agencies and their funders are continuing to apply theory-driven
evaluation to examine the pathways linking food security actions to nutrition and health
outcomes (Johnson et al., 2015). Learning from the practices and experiences documented in
evaluation studies may provide important insights into strategies for the effective use of theory-
driven evaluation, particularly ToCs and REs, in assessing food security programs.
Given the multiple pathways toward food security, the need to understand how food security
programs lead to nutrition and other health outcomes, and the rising popularity of theory-
driven evaluation, this study asks: how and why are ToCs and RE used? Our objectives are to: 1)
examine the nature, extent, and range of published and grey literature on ToCs and REs used in
the context of food security; and 2) examine the assumptions, conditions, and problem framings
130
that allow ToCs and REs to emerge as appropriate tools to guide the evaluation of food security
programs. In doing so, we become better equipped to make methodological choices that align
with the evaluation setting and the attributes of the food security program.
4.2 Methods
We examined the published and grey literature using a scoping review methodology involving a
multi-stage process of search, selection, extraction, and synthesis of the literature (Colquhoun et
al., 2014). We explored the published literature to provide insights into the theoretical
understanding of ToCs and REs whereas the grey literature offered practical knowledge.
4.2.1 Search strategy
In consultation with a university librarian, we developed search terms and strings (Appendix
13). We searched for studies using the following databases: AGRICOLA©, CabDirect©, Web of
Science™ CORE Collection, Medline®, Scopus®, and EconLit©. These databases were selected to
ensure coverage of food security topics from a wide variety of disciplines. The following search
string was used: (food OR agriculture OR nutrition OR livestock OR fish* OR animal OR plant OR
wildlife) AND ((“theor* of change”) OR (realist AND evaluation) OR (realism AND evaluation)).
We also searched Google™ and websites of relevant organizations (Theory of Change
community, the Consultative Group on International Agricultural Research) to capture studies
from the grey literature. No search restrictions were placed. Searches were performed on 10
March 2020. The reference lists of all included studies were hand-searched to identify relevant
studies not captured in the searches.
131
4.2.2 Relevance screening and eligibility
All citations were imported into the web-based application DistillerSR© (Evidence Partners
Incorporated, Ottawa, ON, Canada, v2) for duplicates removal and relevance screening. A two-
step relevance screening strategy was employed by two independent reviewers. First, the titles
and abstracts of articles and reports were screened; next, all citations deemed potentially
relevant went through a review of the full-text (Appendix 14). Studies were considered to be
relevant if: they described the development, implementation, and/or evaluation of a food
security9 or nutrition security10 program; they described a ToC and/or RE approach; and, they
were original research or evaluations published in a journal or report (Table 4.1). Reviewers met
regularly throughout the screening process to resolve conflicts and discuss any uncertainties
related to study selection. The degree of agreement between reviewers was assessed using
Cohen’s Kappa (Landis & Koch, 1977).
9 Food security is “a situation that exists when all people, at all times, have physical, social and economic access to
sufficient, safe and nutritious food that meets their dietary needs and food preferences for an active and healthy life”
(FAO 2001, p. 49). 10 Nutrition security is “a situation that exists when secure access to an appropriately nutritious diet is coupled with a
sanitary environment, adequate health services and care, in order to ensure a healthy and active life for all household
members” ((FAO, 2013, p. 50). Nutrition is an integral part of all four dimensions of food security (availability,
accessibility, utilization, and stability) (Hwalla et al., 2016).
132
Table 4.1. Criteria to select food-related Theory of Change and Realist Evaluation studies.
Inclusion Exclusion
Study describes the development,
implementation, or evaluation of a
food security program.
Study focused on a topic not related to food security.
Study explicitly mentions using a
Theory of Change and/or Realist
Evaluation approach.
Theory of Change and/or Realist Evaluation approach were
mentioned but were not the primary methodological approach
used in the study.
Individual research study published
in a journal or report.
Conference abstracts, letters to the editor, news articles,
dissertations, and reviews.
4.2.3 Data extraction and synthesis
To address objective one (to examine the nature, extent, and range of published and grey
literature on ToCs and REs used in the context of food security), a data charting form was
created to extract data from the studies (Appendix 15). We captured general characteristics of
the article/report, including authorship details, year of publication, type of publication, study
country, study scale, and food topic. Furthermore, we extracted data relevant to the scoping
review objective including: the evaluation purpose; the evaluation methodology; how climate
change was considered11; how studies described context12; and, how population characteristics
11 Climate change is increasingly impacting how programs are developed, implemented, and evaluated, particularly
those aiming to improve food security. Integrating climate considerations into program theory is essential for
understanding how food security programs will respond to climate change. 12 Making explicit the circumstances in which a program is developed, implemented, and evaluated is important for
understanding why programs work and why their impacts might vary in different settings.
133
were reported13. Results were exported into Excel® 2016 (Microsoft, Redmond, WA, USA, v16)
for descriptive analysis.
To address objective two (to examine the assumptions, conditions, and problem framings that
allow ToCs and REs to emerge as appropriate tools to guide the evaluation of food security
programs), all studies were thematically analyzed to identify patterns in the studies using an
inductive approach (Braun et al., 2018). Several sequential phases of inductive coding were
conducted, with results from one level informing the focus of coding in the subsequent phase.
In phase 1, we examined the differences in process, outcomes, and use of theory-driven
evaluation. Building from these findings, phase 2 involved understanding why differences
emerged by examining problem framings. Problem framing is a crucial element of program
theory that defines the challenge which a program will address (Archibald, 2020; Bacchi, 2009).
The multiple experiences, causes, and consequences of food security challenges make it
especially important to define the problem in this context. In phase 3, we examined the
responses to and implications of problem framings. Finally, in phase 4 we examined whether the
responses to problem framings were appropriate to the program needs (see Appendix 17 for
an overview of the coding process). Regular discussions among the authors were held to co-
develop the themes. NVivo© qualitative analysis software (QSR International, Burlington, MA,
13 Differentiating food security outcomes by social identity (e.g. population, age, gender) is helpful for ensuring that
program theories are responding to the needs and priorities of different groups of people.
134
USA, v12) was used to facilitate the coding of text segments, develop themes, and organize
quotations.
4.3 Results
4.3.1 Characteristics of studies
After the removal of duplicates and non-relevant studies, and the addition of studies from the
hand-search, a total of 63 studies were identified (Figure 4.1; see Table S4 for a list of all
included studies). Five pairs of studies reported on evaluations of the same food security
program; these pairs were counted once and analyzed together. The kappa scores for the
title/abstract screening and full-text screening were 0.71 and 0.61 respectively, indicating “good
agreement” for both screening levels (McHugh, 2012). Studies were published between 2011
and 2020 (see Appendix 18 for the number of publications over time). The number of studies
utilizing ToCs and RE in food security contexts has increased, with the majority of studies
published within the past five years (2015-2019) (71%, n = 45). Studies were predominately from
countries in Africa or Asia (90%; n = 57) (Table 4.2). A diversity of topics was covered in the
reviewed studies, with a strong focus on nutrition security and agricultural productivity (Figure
4.2). Food security programs operated primarily at the national (59%; n = 37) or international
scale (29%; n = 18). ToCs were used in nearly all studies (94%; n = 59) whereas RE was used in
eight studies (13%). Both ToCs and RE were used in four studies, which were included in the
counts above.
135
Figure 4.1. Flow chart showing the selection of food-related Theory of Change and Realist
Evaluation studies.
136
Table 4.2. Characteristics of food-related Theory of Change and Realist Evaluation studies.
Theory of Change
studies
Realist Evaluation
studies
All reviewed studies a
Characteristics No. % No. % No. %
Type of publication:
Grey literature 36 61.0 3 37.5 38 60.3
Published literature 23 39.0 5 62.5 25 39.7
Geographic region: b
Africa 41 69.5 4 50.0 42 66.7
Asia 14 23.7 3 37.5 15 23.8
North America 4 6.8 1 12.5 4 6.3
South America 2 3.4 0 0 2 6.3
Europe 2 3.4 2 25.0 2 3.2
Australia and Oceania 2 3.4 0 0 2 3.2
Not applicable c 7 11.9 0 0 7 11.1
Programming scale: b
International 18 30.5 0 0 18 28.6
National 35 59.3 4 50 37 58.7
Provincial 10 16.9 2 25 11 17.5
Local 16 27.1 3 37.5 17 27.0
Not applicable c 7 11.9 0 0 7 11.1
Evaluation purpose b
Evaluation 49 83.0 8 100 53 84.1
Planning 26 44.1 0 0 26 41.3
Learning 12 20.4 0 0 12 19.0
Implementation 8 13.6 0 0 8 12.7
Evaluation approach:
Qualitative 29 49.2 8 100 33 52.4
Quantitative 0 0 0 0 0 0
Mixed qualitative-
quantitative
0 0 0 0 0 0
Not applicable c 7 11.9 0 0 7 11.1
Not specified d 23 39.0 0 0 23 36.5
a Numbers do not add up to the total as studies may apply both Theories of Change and Realist
Evaluation.
b Categories are not mutually exclusive. c Study did not specify characteristics of case studies (e.g. conceptual/theoretical papers). d Study did not provide sufficient detail to categorize the evaluation approach.
137
Figure 4.2. Food topics of reviewed Theory of Change and Realist Evaluation studies.
4.3.2 ToCs and RE were developed primarily using qualitative approaches
The development process varied across ToC studies (94%; n = 59), with methods ranging from
participatory workshops (n = 20) which encouraged stakeholder participation, to evaluator-led
approaches such as interviews (n = 2). For example, a participatory workshop in Tanzania
engaged program implementers, dairy value chain scientists, government representatives, and
private sector actors in the creation of a ToC for a dairy value chain development program
(Kidoido & Korir 2013). Another common approach was to draft an initial ToC by the evaluation
team and then circulate the draft to program implementers and/or wider stakeholders for
revisions (Johnson et al. 2015). Of note, 23 studies (36%) did not provide enough detail to
categorize the methodology. REs (13%; n = 8) were also conducted using qualitative approaches
using either interviews (n = 6) or workshops (n = 2). For example, RE was reported to be used
for identifying the relationships between context, mechanisms, and outcomes of a farm
0 2 4 6 8 10 12 14
Food loss
Food security risk management
Community food security
Food safety
Natural resource management
Agricultural innovation systems
Climate change adaptation of food systems
Market-based approaches to food security
Agricultural productivity
Nutrition security
Number of studies
Fo
od
to
pic
Theory of Change Realist Evaluation
138
biosecurity project in the United Kingdom (Maye et al., 2019). The authors conducted interviews
with policymakers in addition to literature reviews to develop initial context, mechanism, and
outcome relationships, followed by validation of the relationships through surveys and
interviews with farmers.
4.3.3 ToCs served a variety of purposes whereas RE supported evaluation purposes only
While ToCs had multiple reported uses in a food security program, ToCs primarily served as a
basis for program monitoring and evaluation (83%; n = 49). For example, in Nimpagaritse et al.
(2020) a proposed ToC was developed for a nutrition program in Burundi to identify starting
points for monitoring and evaluation. Once ToCs were developed, many studies collected data
on indicators by complementing the ToC with other evaluation tools, including contribution
analysis (Obodai et al., 2018), outcome harvesting (Douthwaite & Getnet 2019), or qualitative
and/or quantitative monitoring of change pathways identified by the ToC (Cole et al., 2016). Of
note, few studies provided a citation for the specific ToC process or approach used (Arriola et al.,
2020; Michelini et al., 2020; Omore et al., 2019), contributing to uncertainty surrounding the
assumptions underpinning the design of these ToCs. The reviewed studies typically used ToCs to
set the stage for evaluation, suggesting ToCs were considered an evaluation planning tool rather
than an evaluation approach.
Many studies also described developing ToCs for program planning purposes (44%, n = 26) (e.g.
Arriola et al., 2020) or learning, deriving good practices, and generating new research questions
139
(20%; n = 12) (e.g. Sally & Merrey 2019). ToCs were rarely reported to be used for
implementation purposes (e.g. Catholic Relief Services 2017) (14%, n = 8). In the few cases
where guiding implementation was a stated purpose, studies supported the practice of
reflection on outcomes to “become part of the culture of implementation” (Apgar et al. 2017, p.
29). In many instances (36%; n = 23), particularly in the grey literature, we were unable to
categorize the evaluation approach because details were missing on how the ToC was
developed.
In contrast to ToC studies, no RE studies reported using the evaluation approach to support
program planning, implementation, or learning. Rather, RE played an important role in
evaluating food security programs to assess and report on results. For example, a study in
Vietnam aimed to identify what contribution the project made to reducing poverty and food
insecurity (McDonald, 2011). The study drew on RE as an evaluation framework to identify what
changed, what mechanisms brought about these changes, and how the context informed the
change. While REs generally provided rich insights into the context and mechanisms that
created desired outcomes of programs, the volume of data generated through REs can be
difficult to manage (Maye et al., 2019).
4.3.4 The use of multiple theory-driven evaluation approaches together
Four studies reported on the use of ToC together with RE. For example, interviews with health
sector staff and reviews of nutritional guidelines helped to develop a retrospective ToC for a
food and beverage sales environment intervention in Canada (Levay et al., 2018). The ToC was
140
then reported to be used as a basis for an RE aiming to provide an in-depth explanation as to
why the intervention worked. Another study described using a logical framework, ToC, and RE to
construct a program theory in the context of a child feeding program in Tanzania (Floate et al.,
2019). According to the authors, the logical framework and ToC helped provide theories of how
the program would achieve change through a series of outcomes and planned activities;
however, “critical untested assumptions” were made in using these two approaches (p. 12). Thus,
the authors explained how RE enabled them to tease out how participants may interact with
program resources in a given context to produce outcomes, providing a more nuanced
understanding of key mechanisms and contextual factors that were not provided by the logical
framework or ToC.
4.3.5 Considerations of gender, context, and climate change
The variety of food security topics, methods, and purposes of ToCs made it difficult to draw
broad conclusions on patterns in gender, climate change, and contextual considerations. Also,
there did not appear to be a clear pattern in gender, climate change, and contextual
considerations among RE studies, though this lack of a pattern was likely due to the small
sample of included RE studies. While this review explored how different social groups (e.g.
population, age, gender) were reported on, only gender implications were explicitly mentioned
in studies. Both ToC and RE studies considered gender in nearly half of studies (Figure 4.3),
primarily by outlining outcome pathways for women as the main program beneficiaries. In
several studies, gender concerns were described vaguely. For example, in a ToC of an aquatic
141
agricultural systems program, the assumption “issues of gender explicitly addressed” was
presented without an elaboration of the nature of these gendered issues (Johnson et al. 2019,
pg. 423). All RE studies accounted for contextual influences on program implementation and
outcomes, primarily social and/or political influences. Fewer ToC studies integrated context
compared to RE studies, as context was historically not considered in the conceptualization of
ToCs. In a ToC study focused on strengthening smallholder dairy value chains, contextual issues
around the program were “internalized into the change pathway” (Omore et al., 2019, p. 902),
suggesting context might sometimes be implicit. Very few ToC and RE studies noted climate
change considerations.
142
Figure 4.3. Considerations of context, gender, and climate change in food-related Theory of
Change and Realist Evaluation studies.
143
4.3.6 Problem representations offered by studies
There was considerably more heterogeneity in methods and purposes for ToC studies compared
to RE. To identify factors that might explain these differences, we explored the issues authors
defined for their evaluations. In examining the problem representations offered in studies, three
themes emerged: complex food security programs, systems, and issues; evaluation challenges;
and information gaps in food security strategies, process, and outcomes (see Appendix 19 for
the codebook). Of note, some studies used multiple framings; for example, a study may have
focused primarily on complexity but also discussed limitations of evaluation approaches.
Complex food security programs, systems, and issues. Many ToC studies framed evaluation
problems around the complexity of food security programs, which were characterized by
multidimensions and an integrated and multisectoral approach (Maye et al., 2019; Omore et al.,
2019). Such programs often required long timeframes, multiple causal strands, and multiple
feedback loops (Omore et al., 2019; Wesley et al., 2019), leading to emergence, unpredictability,
non-linearity, and uncertainty of outcomes (Douthwaite et al., 2017; Pradel, et al., 2013).
Furthermore, food security programs were reported to be complex due to the social systems
within which food security actions occur and the interactions of diverse groups whose behaviour
food security programs aimed to influence (Apgar et al., 2017). And in complex social systems,
multiple factors contributed to food security outcomes, making it difficult to identify cause-and-
effect linkages (Child et al., 2015). Food security concepts (e.g. nutrition, food system resilience)
144
were also complex themselves due to their lack of clear solutions, a plurality of perspectives, and
evolving nature (Béné et al., 2020; Epp and Garside, 2019).
Evaluation challenges. Another common problem representation reported in ToC studies was
the limitations of existing approaches to evaluation. Traditional approaches to evaluation (e.g.
those comparing a baseline and endline) were described as neither able to capture how and why
programs work (Ton 2014), nor to assess a complex program’s contribution to food security due
to multiple factors that influence change (Bene et al. 2020). Tools conceptually similar to ToC
that aimed to map change pathways (e.g. logic model, logical framework, results chain) were
critiqued for oversimplifying food security programs (Levay et al. 2019). ToCs for agricultural
research have also been critiqued for assuming impact will be created through a predictable
linear adoption pathway, thus ignoring complex systems dynamics and missing alternative
pathways (Douthwaite & Hoffecker, 2017). Furthermore, ToCs were reported to have decreased
effectiveness when program theories were not adequately spelled out or did not account for
assumptions, context, or unexpected outcomes (Cole et al., 2016; Douthwaite et al., 2017). The
limited research on how to evaluate certain food security concepts also posed challenges for
evaluation. For instance, Bene et al. (2019) argued the resilience of food systems to climate
change is a latent variable and thus cannot be measured directly.
Information gaps in food security processes, outcomes, and strategies. Most ToC studies
and all RE studies centered problems around the need for explaining how and why programs
145
work. Focusing on program process was considered particularly important in ToC studies. Maru
et al. (2018) explained that many food security programs, especially agricultural research for
development interventions, tended to have “lofty food security and/or agricultural development
goals” without articulating “the theories and pathways for how and why the particular
intervention would contribute to or achieve impact” (p. 345). The need for a framework to
navigate towards impact was also reported in ToC studies, particularly in settings characterized
by a “multiplicity of views from different actors on issues of joint concern” (Vellema et al., 2017,
p. 2). Donors have sometimes influenced the use of ToCs in program planning since many
donors require projects to present a clear plan showing how activities lead to outcomes
(Catholic Relief Services 2017). RE studies appeared motivated simply by the fact that a food
security program’s outcomes were understudied, underexplored, or unknown (Michelini et al.,
2020; Ohly et al., 2019). Among these RE studies, few also discussed how programs were
complex due to contextual factors (e.g. social, cultural, economic) that influenced a program’s
outcomes (Owusu-Addo et al., 2019).
4.3.7 Responses to problem framings
To deepen insights into problem framings articulated by authors, we examined the response of
authors to these problem framings and conditions under which responses occurred (see
Appendix 20 for the codebook). Then, we assessed whether responses were appropriate for the
problem framing. We found ToCs to be the main solution for all problem framings, suggesting
ToCs might be appropriate in addressing multi-dimensional needs. All reviewed ToC studies
146
included a narrative describing change pathways toward food security, with most narratives also
accompanied by a figure depicting such pathways. The figure typically included arrows
representing presumed causal linkages and boxes representing activities and outcomes. This
boxes-and-arrows model was popular among each problem framing, but the level of specificity
varied, with complexity-framed studies reaching an impressive level of detail. All RE studies
presented context, mechanism, and outcome configurations. The relationship between problem
framings and responses to problem framings is shown in Figure 4.4.
Figure 4.4. Concept map showing the relationship between problem framings, responses to
problem framings, and conditions under which responses occur in theory-driven evaluations.
147
There were two main pathways through which “complexity-aware” ToCs were developed: (1) the
study emphasized complex issues or systems (Douthwaite and Hoffecker 2017; Epp et al. 2019;
Maru et al. 2018), and (2) the study focused on evaluation challenges or information gaps while
also emphasizing complexity (Douthwaite et al., 2017). Studies reported accounting for
complexity through feedback loops (i.e. two-way arrows linking outcomes) and backward
mapping (i.e. asking what are the necessary preconditions for outcomes to occur). The concept
of emergence was considered in some studies through revisions made to the ToC as new
information was gained during program planning or implementation. For example, in Apgar et
al. (2017), the initial broad ToC for an aquatic agricultural systems program was revised after a
planning workshop to further detail the ToC, leading to the integration of gender equity,
nutrition, and climate change resilience needs. According to some authors, ToCs of complex
programs were purposively simplified by presenting a variety of ToCs for actors, projects, or
sites (i.e. nested ToCs). For instance, Douthwaite et al. (2013) developed a broad ToC at the
program-level and several ToCs at the project-level.
Studies describing a program’s complexity, an evaluation challenge, or an information gap
typically developed ToCs that appeared simple. Simple ToCs were often characterized by
linearity, expressed through the use of uni-directional arrows between activities and outcomes
(Pound 2015). In many studies, ToCs were described as following an “if/then” logic (i.e. if these
activities are conducted, then these outcomes will be achieved) (Álvarez-Mingote et al., 2020;
Baker et al., 2013; Levay et al., 2018), suggesting a stepwise pathway toward food security
148
outcomes. Some studies included assumptions in the linear pathway, which resembled an
adapted logic model (e.g. Levay et al., 2019; Obodai et al., 2018). While program theories in RE
studies were static, ToCs might likely be revised following implementation. For example, in
Nimpagaritse et al. (2020), the ToC initially resembled a logic model but upon revision, it
included assumptions such as socio-political stability. All RE studies were reported to be
conducted in response to information gaps around a program’s outcomes, with little variation
observed in the outputs between studies.
We found ToCs to be an appropriate solution to issues of complexity by unpacking the change
pathways and assumptions made. However, when issues were framed solely in terms of
evaluation challenges or information gaps, there was potential for ToC misuse. Some of these
studies presented an overly simplified ToC, characterized by linearity or an absence of
assumptions, leading to a reduced ability of ToCs to guide program planning and evaluation as
ToCs were originally intended. As cautioned by Apgar et al. (2017), care needs to be taken
during ToC development otherwise ToC risks becoming “nothing more than a complex log
frame” (Apgar et al., 2017, p. 29). In ToC studies where pathways toward food security were
relatively predictable and understanding cause-and-effect assumptions was not an immediate
goal, alternative approaches to evaluation might be appropriate. For example, in the context of a
therapeutic feeding program focused on addressing malnutrition, Floate et al. (2018) created a
logic model to develop an initial program theory and then stated using a ToC to provide a more
in-depth depiction of the program theory. We found RE to be appropriate for addressing
149
information gaps by providing a nuanced understanding of how, why, and for whom changes
occurred. However, for RE studies that also emphasized complexity, it would be helpful to clarify
how the dynamics of non-linear outcomes (e.g. poverty reduction) and multiple interacting
program components could be accounted for in context, mechanism, and outcome
configurations (Owusu-Addo et al., 2019).
4.4 Discussion
This review examined how and why theory-driven evaluation has been developed and used to
evaluate food security programs globally, drawing on ToC and RE as examples. The emphasis of
included studies was on the topic of food production, likely due to early conceptualizations of
food security when it was believed that producing more food led to food security (Hendriks,
2015). More recently, as awareness of the nature of malnutrition (undernutrition, overnutrition,
and micronutrient deficiencies) improved, studies equally focused on nutrition. Studies were
geographically concentrated in Africa and Asia, areas where the prevalence of malnutrition
remains the highest (FAO 2020). Indeed, we would expect studies to prioritize food security
programs in areas with high food security challenges.
While we found variations in the process of ToC development, in many studies little detail was
provided about the process. This finding was consistent with sentiments in the literature, which
highlighted the limited guidance, training, and agreement on how to develop a ToC (Maye,
2015; Maru et al., 2018). It is important to elaborate on the ToC development process as the
150
absence of detail gave the impression that the ToC might have been developed superficially and
applied in a cursory manner. Opportunities exist for authors to describe and reflect on the
process, providing clarity for others in developing ToCs for food security programs. In contrast,
reporting on RE was consistent, likely due to the presence of standards to guide practice. For
example, the Realist and Meta-narrative Evidence Syntheses: Evolving Standards (RAMESES) II
Project developed resources, training materials, and reporting standards for REs (Wong et al.,
2017).
ToCs were used substantially more than RE, although the reason for choosing the evaluation
approach was rarely made explicit. Providing rationale is important for understanding when and
why a particular evaluation approach might be suitable for capturing the complexities of a food
security program, issue, or context. Moreover, as ToCs had a variety of stated purposes, being
explicit about what the ToC hopes to achieve is critical for establishing the foundation for
starting the process of developing program theories. Interestingly, we found few studies
reported using ToCs to support implementation purposes, similar to a review of ToCs used in
public health interventions (Breur et al., 2016). Revising ToCs as new information is learned is a
potential strategy to help guide food security practitioners and evaluators toward desired
outcomes during implementation (Ghate, 2018).
We identified several studies applying a ToC followed by an RE, supporting the hypothesis that
ToC and RE might be productively combined (Blamey & Mackenzie, 2007). In comparing these
151
studies, we found many RE studies typically concluded with a presentation of various context,
mechanism, and outcome configurations for a food security program. However, exploring how
such configurations might refine the initial program theory could lead to the generation of
middle-range theories relevant in another context (Owusu-Addo et al., 2019, p. 16). A middle-
range theory is “not abstract to the point of being disconnected from the on-the-ground
workings of programs, yet not so specific to pertain to one program” (Jagosh, 2019, p. 364).
Importantly, integrating key elements of context into program theories might also build context-
sensitive knowledge, allowing for better predictions of how programs function in diverse
contexts. Considering we identified very few studies combining ToC and RE, there is a need for
further studies combining ToC and RE approaches in different fields and different ways to
further substantiate our findings (Rolfe 2019).
Although the achievement of nutrition and other health outcomes is contextually contingent,
few ToC studies reported on the context in which programs were situated, particularly in relation
to climate change. And while all RE studies reported on context, climate change was rarely
considered as a component of that context. Integrating climate change mitigation and
adaptation into program theories is important because environmental sustainability is critical to
sustaining food security in the long term and essential to achieving the Paris Agreement’s
emissions targets (Fanzo et al., 2020). Nearly half of ToC and RE studies incorporated gender
dimensions, which acknowledge that the experiences of food security differ based on gender
152
(Mbow et al., 2019). Continuing to question “for whom” program theories work might help
ensure pathways toward food security are socially inclusive (Forsyth, 2018; Lam et al., 2019).
The if/then logic presented in many ToC studies came from logic model thinking, which was in
contrast with some ToC thinking that acknowledges: “A ToC sets out testable hypotheses of a
change process by working back from long-term goals to identify all the conditions that
(theoretically) must be in place for the goals to occur” (Belcher et al., 2017, p. 3). This logic of
“for x to happen, y is essential” is an important tenet of ToC for focusing on how systems
change, before thinking about how programs might contribute to change. In doing so,
evaluations become outcome-based rather than objective-based. Clarifying how ToCs differ
from conceptually similar approaches might help ensure ToCs are used as originally intended
(Maru et al., 2018). One option for evaluators to consider is the checklist in Breuer et al. (2016)
which guides key aspects of ToC to include.
Based on our analysis, the inclusion of additional problem framings such as complexity might
lead to sufficiently detailed ToCs. For RE studies, it is important to consider how elements of
complexity such as emergence and non-linearity might be reflected in context, mechanism, and
outcome relationships. Potential strategies include the use of arrows showing the
interconnections between different relationships (Punton et al., 2016) or feedback loops
(Feather, 2018). In multisectoral programs with multiple food security outcomes, prioritizing key
context, mechanism, and outcome relationships were critical for ensuring the data produced are
153
manageable (Leavy et al., 2018). As problem framing sets boundaries on what aspects should be
studied and evaluated and what other aspects should be left out, reflecting on the problem
framing should be considered in theory-driven evaluation (Archibald, 2020; Bacchi, 2009).
4.4.1 Strengths and limitations
We acknowledge several limitations. First, GoogleTM was not able to capture all studies from
grey literature, especially from sources that do not allow GoogleTM to crawl their sites. We partly
addressed this limitation by complementing the search with a targeted website search of select
organizations (e.g. ToC community and CGIAR). Secondly, our ability to understand the
development of theory-driven evaluation was based on what was reported in studies. Given
word and space limits, articles and reports might have prioritized documenting evaluation
outcomes rather than process. For example, in several studies, evaluation experiences were
presented across multiple articles (e.g. Levay et al., 2018; Levay et al., 2019). Thirdly, this review
included multiple articles published by the same research team. For instance, Douthwaite et al.
published five included studies. Researchers may have used a similar approach across multiple
studies, which could have resulted in an over-emphasis of their research process in the results.
4.5 Conclusion
By synthesizing the literature on theory-driven evaluation, we showed the various ways in which
ToCs and RE were developed and used in food security contexts. Our review demonstrated that
ToCs arose in response to several issues, including the need to evaluate under complexity,
challenges with evaluation approaches, and information gaps surrounding a food security
154
process, outcome, and strategy. In contrast, studies reported conducting an RE primarily to
understand a program’s outcomes in response to information gaps. In examining these
responses, we showed that problem framings focused on complexity contributed to more
nuanced program theories, possibly through improved understandings of the evaluation context
and adaptations to the evaluation approach. We recommend that practitioners working in food
security contexts consider reflecting on the problem to be addressed in the evaluation,
potentially leading to a more informed choice and design of evaluation approaches.
4.5.1 Lessons learned
1) Make explicit the purpose, process, and rationale for conducting a theory-driven evaluation.
This clarity from the beginning provides the foundation for the development of theory-driven
evaluation. Furthermore, this clarity helps readers understand when and why a particular
approach is suitable for a specific context.
2) Consider revising ToCs when new knowledge is gained. Adapting ToC during program
planning or implementation might inform important program adaptations, while refining ToCs
after an evaluation is completed may generate program theories that apply to other settings.
3) Consider how complexity might be accounted for in RE of complex programs. Many food
security outcomes have multiple interacting influences crossing socio-ecological levels and
require multi-level strategies to achieve food security. This complexity requires evaluative
thinking on how dynamics can be creatively captured in configurations that link context,
mechanisms, and outcomes.
155
4) Question why a specific evaluation approach was adopted, for whom the approach supports,
and what might be included/excluded. This process might help evaluators discern which
evaluation approaches are suitable and what elements to include in these approaches.
156
4.6 References
Álvarez-Mingote, C., Moore, A., & McNamara, P. (2020). Assessing the role of stakeholder
platforms as drivers of resilient communities: the case of Malawi. Journal of Agricultural
Education and Extension, 26(1), 75–95. https://doi.org/10.1080/1389224X.2019.1674169
Apgar, J. M., Allen, W., Albert, J., Douthwaite, B., Paz Ybarnegaray, R., & Lunda, J. (2017). Getting
beneath the surface in program planning, monitoring and evaluation: Learning from use of
participatory action research and theory of change in the CGIAR Research Program on
Aquatic Agricultural Systems. Action Research, 15(1), 15–34.
https://doi.org/10.1177/1476750316673879
Archibald, T. (2020). What’s the Problem Represented to Be? Problem Definition Critique as a
Tool for Evaluative Thinking. American Journal of Evaluation, 41(1), 6–19.
https://doi.org/10.1177/1098214018824043
Arriola, K. R. J., Ellis, A., Webb-Girard, A., Ogutu, E. A., McClintic, E., Caruso, B., & Freeman, M. C.
(2020). Designing integrated interventions to improve nutrition and WASH behaviors in
Kenya. Pilot and Feasibility Studies, 6, 10. https://doi.org/10.1186/s40814-020-0555-x
Bacchi, C. L. (2009). Analysing Policy: What’s the problem represented to be? Frenchs Forest,
Australia: Pearson Education.
Baker, J., Sanghvi, T., Hajeebhoy, N., Martin, L., & Lapping, K. (2013). Using an evidence-based
approach to design large-scale programs to improve infant and young child feeding. Food
and Nutrition Bulletin, 34(4): S146-55. https://doi.org/10.1177/15648265130343s202
Belcher, B., Suryadarma, D., & Halimanjaya, A. (2017). Evaluating policy-relevant research: lessons
from a series of theory-based outcomes assessments. Palgrave Communications, 3(1),
17017. https://doi.org/10.1057/palcomms.2017.17
Bene, C., Riba, A., & Wilson, D. (2020). Impacts of resilience interventions - Evidence from a
quasi-experimental assessment in Niger. International Journal of Disaster Risk Reduction, 43.
https://doi.org/10.1016/j.ijdrr.2019.101390
Béné, C., Riba, A., & Wilson, D. (2020). Impacts of resilience interventions – Evidence from a
quasi-experimental assessment in Niger. International Journal of Disaster Risk Reduction, 43,
101390. https://doi.org/10.1016/j.ijdrr.2019.101390
Bird, F. A., Pradhan, A., Bhavani, R. V., & Dangour, A. D. (2019). Interventions in agriculture for
nutrition outcomes: A systematic review focused on South Asia. Food Policy, 82, 39-49.
https://doi.org/10.1016/j.foodpol.2018.10.015
157
Blamey, A., & Mackenzie, M. (2007). Theories of Change and Realistic Evaluation: Peas in a Pod
or Apples and Oranges? Evaluation, 13(4), 439–455.
https://doi.org/10.1177/1356389007082129
Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2018). Thematic analysis. In P. Liamputtong (Ed.),
Handbook of Research Methods in Health Social Sciences. Singapore: Springer.
Breuer, E., Lee, L., De Silva, M., & Lund, C. (2016). Using theory of change to design and evaluate
public health interventions: A systematic review. Implementation Science, 11(1). 63.
https://doi.org/10.1186/s13012-016-0422-6
Catholic Relief Services. (2017). Theory of Change, Metrics and Learning Agenda. Baltimore,
United States: Catholic Relief Services.
Child, K., Kidoido, M., Kanyuuru, C. (2015). Using the Theories of Change approach to monitor,
evaluate and learn in the CGIAR research program on Livestock and Fish. Nairobi, Kenya:
ILRI.
Cole, D. C., Levin, C., Loechl, C., Thiele, G., Grant, F., Girard, A. W., Sindi, K., & Low, J. (2016).
Planning an integrated agriculture and health program and designing its evaluation:
experience from Western Kenya. Evaluation and Program Planning, 56, 11–22.
https://doi.org/10.1016/j.evalprogplan.2016.03.001
Colquhoun, H. L., Levac, D., O’Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M., & Moher,
D. (2014). Scoping reviews: Time for clarity in definition, methods, and reporting. In Journal
of Clinical Epidemiology, 67(12), 1291-1294. https://doi.org/10.1016/j.jclinepi.2014.03.013
Douthwaite, B., Getnet, K. (2019). Outcome Evaluation of the work of the CGIAR Research Program
on Land, Water and Ecosystems (WLE) on soil and water management in Ethiopia. Nairobi,
Kenya: CGIAR.
Douthwaite, B., Kamp, K., Longley, C., Kruijssen F., Puskur, R., Chiuta, T., Apgar, M., Dugan, P.
(2013). Using theory of change to achieve impact in AAS. Penang, Malaysia: WorldFish.
Douthwaite, B., & Hoffecker, E. (2017). Towards a complexity-aware theory of change for
participatory research programs working within agricultural innovation systems.
Agricultural Systems, 155, 88–102. https://doi.org/10.1016/j.agsy.2017.04.002
Douthwaite, B., Mayne, J., McDougall, C., & Paz-Ybarnegaray, R. (2017). Evaluating complex
interventions: A theory-driven realist-informed approach. Evaluation, 23(3), 294–311.
https://doi.org/10.1177/1356389017714382
158
Fanzo, J., Covic, N., Dobermann, A., Henson, S., Herrero, M., Pingali, P., & Staal, S. (2020). A
research vision for food systems in the 2020s: Defying the status quo. Global Food Security,
26, 100397. https://doi.org/10.1016/j.gfs.2020.100397
FAO. (2013). The State of Food Insecurity in the World 2013: The multiple dimensions of food
security. Rome, Italy: FAO.
Feather, J. L. (2018). Developing programme theories as part of a realist evaluation of a
healthcare quality improvement programme. International Journal of Care Coordination,
21(3), 68-72. https://doi.org/10.1177/2053434518779753
Fiorella, K. J., Chen, R. L., Milner, E. M., & Fernald, L. C. H. (2016). Agricultural interventions for
improved nutrition: A review of livelihood and environmental dimensions. Global Food
Security, 8, 39-47. https://doi.org/10.1016/j.gfs.2016.03.003
Floate, H., Durham, J., & Marks, G. C. (2019). Moving on from logical frameworks to find the
“missing middle” in international development programmes. Journal of Development
Effectiveness, 11(1), 89–103. https://doi.org/10.1080/19439342.2018.1551921
Forsyth, T. (2018). Is resilience to climate change socially inclusive? Investigating theories of
change processes in Myanmar. World Development, 111, 13–26.
https://doi.org/10.1016/j.worlddev.2018.06.023
Ghate, D. (2018). Developing theories of change for social programmes: co-producing evidence-
supported quality improvement. Palgrave Communications, 4(1), 90.
https://doi.org/10.1057/s41599-018-0139-z
Grace, D., Dipeolu, M., & Alonso, S. (2019). Improving food safety in the informal sector: nine
years later. Infection Ecology and Epidemiology, 9(1), 1579613.
https://doi.org/10.1080/20008686.2019.1579613
Hendriks, S. L. (2015). The food security continuum: a novel tool for understanding food
insecurity as a range of experiences. Food Security, 7(3), 609–619.
https://doi.org/10.1007/s12571-015-0457-6
Howland, F. (2019). Individual capacity baseline report. Rome, Italy: CIAT.
Hwalla, N., El Labban, S., & Bahn, R. A. (2016). Nutrition security is an integral component of food
security. Frontiers in Life Science, 9(3), 167-172.
https://doi.org/10.1080/21553769.2016.1209133
Jagosh, J. (2019). Realist Synthesis for Public Health: Building an Ontologically Deep
159
Understanding of How Programs Work, for Whom, and in Which Contexts. Annual Review
of Public Health, 40: 361-372. https://doi.org/10.1146/annurev-publhealth-031816-044451
Johnson, N., Mayne, J., Grace, D., & Wyatt, A. (2015). How will training traders contribute to
improved food safety in informal markets for meat and milk? A theory of change analysis.
Washington, DC: International Food Policy Research Institute.
Jones, A. D., Ngure, F. M., Pelto, G., & Young, S. L. (2013). What Are We Assessing When We
Measure Food Security? A Compendium and Review of Current Metrics. Advances in
Nutrition: An International Review Journal, 4(5), 481–505.
https://doi.org/10.3945/an.113.004119
Kidoido, M., Korir, L. (2013). Report of the Tanzania Dairy Value Chain Impact Pathways
Workshop, Dar-es-Salaam, Tanzania, 7-8 May 2013. Nairobi, Kenya: ILRI.
Lam, S, Dodd, W., Berrang-Ford, L., Ford, J., Skinner, K., Papadopoulos, A., & Harper, S. L. (2021).
How are climate actions evaluated? A review of United Nations food security evaluations.
Global Food Security, 28, 100509. https://doi.org/10.1016/j.gfs.2021.100509
Lam, S., Dodd, W., Whynot, J., & Skinner, K. (2019). How is gender being addressed in the
international development evaluation literature? A meta-evaluation. Research Evaluation,
28(2), 158-168. https://doi.org/10.1093/reseval/rvy042
Lam, S. (2020). Toward Learning from Change Pathways: Reviewing Theory of Change and Its
Discontents. Canadian Journal of Program Evaluation, 35(2), 188-203.
https://doi.org/10.3138/cjpe.69535
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data.
Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310
Leavy, J., Boydell, E., McDowell, S., Sladkova, B. (2018). Resilience results BRACED final evaluation.
Washington, DC: Itad.
Levay, A. V, Chapman, G. E., Seed, B., & Wittman, H. (2018). It’s just the right thing to do:
conceptualizing a theory of change for a school food and beverage sales environment
intervention and implications for implementation evaluation. Evaluation and Program
Planning, 70, 73–82. https://doi.org/10.1016/j.evalprogplan.2018.04.011
Levay V, A., Chapman, G. E., Seed, B., & Wittman, H. (2018). It’s just the right thing to do:
Conceptualizing a theory of change for a school food and beverage sales environment
interv ention and implications for implementation evaluation. Evaluation and Program
Planning, 70, 73–82. https://doi.org/10.1016/j.evalprogplan.2018.04.011
160
Levay, A. V, Chapman, G. E., Seed, B., & Wittman, H. (2019). District-level implementation of
British Columbia’s school food and beverage sales policy: a realist evaluation exploring
intervention mechanisms in urban and rural contexts. Canadian Journal of Public Health,
110(1), 21–30. https://doi.org/10.17269/s41997-018-0159-x
Marchal, B., van Belle, S., van Olmen, J., Hoerée, T., & Kegels, G. (2012). Is realist evaluation
keeping its promise? A review of published empirical studies in the field of health systems
research. Evaluation, 18(2), 192-212. https://doi.org/10.1177/1356389012442444
Maru, Y., Sparrow, A., Stirzaker, R., & Davies, J. (2018). Integrated agricultural research for
development (IAR4D) from a theory of change perspective. Agricultural Systems, 165, 310–
320. https://doi.org/10.1016/j.agsy.2016.09.012
Maru, Y. T., Sparrow, A., Butler, J. R. A., Banerjee, O., Ison, R., Hall, A., & Carberry, P. (2018).
Towards appropriate mainstreaming of “Theory of Change” approaches into agricultural
research for development: Challenges and opportunities. Agricultural Systems, 165, 344–
353. https://doi.org/10.1016/j.agsy.2018.04.010
Maye, D., Enticott, G., & Naylor, R. (2019). Theories of Change in Rural Policy Evaluation.
Sociologia Ruralis, 60(1): 198-221. https://doi.org/10.1111/soru.12269
Mayne, J., & Johnson, N. (2015). Using theories of change in the CGIAR Research Program on
Agriculture for Nutrition and Health. Evaluation, 21(4), 407–428.
https://doi.org/10.1177/1356389015605198
Mbow, C., Rosenzweig, C., Barioni, L.G., Benton, T.G., Herrero, M., Krishnapillai, M., Liwenga, E.,
Pradhan, P., Rivera-Ferre, M.G., Sapkota, T., Tubiello, F.N., Xu, Y., 2019. Food security. In:
Shukla, P.Z.P.R., Skea, J., Calvo Buendia, E., Masson- Delmotte, V., P¨ortner, H.-O., Roberts,
D.C., Slade, J.P.P.R., Connors, S., van Diemen, R., Ferrat, M., Haughey, E., Luz, S., Neogi, S.,
Pathak, M., Petzold, J., Vyas, J.M.P., Huntley, E., Kissick, K., Belkacemi, M. (Eds.), Climate
Change and Land: an IPCC Special Report on Climate Change, Desertification, Land
Degradation, Sustainable Land Management, Food Security, and Greenhouse Gas Fluxes in
Terrestrial Ecosystems. Cambridge, UK: Cambridge University Press.
McDonald, B. (2011). Managing water and land at the interface between fresh and saline
environments An impact evaluation. Colombo, Sri Lanka: CGIAR.
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276–282.
https://doi.org/10.11613/BM.2012.031
Michelini, L., Grieco, C., Ciulli, F., & Di Leo, A. (2020). Uncovering the impact of food sharing
platform business models: a theory of change approach. British Food Journal, 122(5). 1437-
161
1462. https://doi.org/10.1108/BFJ-06-2019-0422
Nimpagaritse, M., Korachais, C., & Meessen, B. (2020). Effects in spite of tough constraints - A
theory of change based investigation of contextual and implementation factors affecting
the results of a performance based financing scheme extended to malnutrition in Burundi.
PLoS ONE, 15(1), e0226376. https://doi.org/10.1371/journal.pone.0226376
Obodai, J., Adjei, P. O. W., Hamenoo, S. V. Q., & Abaitey, A. K. A. (2018). Towards household food
security in Ghana: assessment of Ghana’s expanded forest plantation programme in Asante
Akim South District. GeoJournal, 83(2), 365–380. https://doi.org/10.1007/s10708-017-9776-
9
Ohly, H., Crossland, N., Dykes, F., Lowe, N., & Moran, V. H. (2019). A realist qualitative study to
explore how low-income pregnant women use Healthy Start food vouchers. 15, e12632.
Maternal and Child Nutrition. https://doi.org/10.1111/mcn.12632
Omore, A., Kidoido, M., Twine, E., Kurwijila, L., O’Flynn, M., & Githinji, J. (2019). Using “theory of
change” to improve agricultural research: recent experience from Tanzania. Development in
Practice, 29(7), 898–911. https://doi.org/10.1080/09614524.2019.1641182
Owusu-Addo, E., Renzaho, A. M. N., & Smith, B. J. (2019). Cash transfers and the social
determinants of health: Towards an initial realist program theory. Evaluation, 25(2), 224-
244. https://doi.org/10.1177/1356389018814868
Pradel, W., Cole, D.C., Prain, G. (2013). Mixing methods for rich and meaningful insight: Evaluating
changes in an agricultural intervention in the Central Andes. Available from:
https://www.betterevaluation.org/sites/default/files/Mixing%20Methods%20for%20Rich%2
0and%20Meaningful%20Insight.pdf. Accessed 10 Nov 2020.
Punton, M., Vogel, I., Lloyd, R. (2016). Reflections from a Realist Evaluation in Progress: Scaling
Ladders and Stitching Theory. Brighton, UK: IDS.
Rolfe, S. (2019). Combining Theories of Change and Realist Evaluation in practice: Lessons from a
research on evaluation study. Evaluation, 25(3), 294–316.
https://doi.org/10.1177/1356389019835229
Sally, H., Merrey, D. (2019). Outcome Evaluation of Research for Development Work Conducted in
Ghana and Sri Lanka under the Resource, Recovery and Reuse (RRR) Subprogram of the
CGIAR Research Program on Water, Land and Ecosystems (WLE). Colombo, Sri Lanka: CGIAR.
Salter, K. L., & Kothari, A. (2014). Using realist evaluation to open the black box of knowledge
translation: a state-of-the-art review. Implementation Science. 9(1), 115.
162
https://doi.org/10.1186/s13012-014-0115-y
Thornton, P. K., Schuetz, T., Förch, W., Cramer, L., Abreu, D., Vermeulen, S., & Campbell, B. M.
(2017). Responding to global change: A theory of change approach to making agricultural
research for development outcome-based. Agricultural Systems, 152, 145–153.
https://doi.org/10.1016/j.agsy.2017.01.005
Van Epp, M., & Garside, B. (2019). Towards an evidence base on the value of social learning-
oriented approaches in the context of climate change and food security. Environmental
Policy and Governance, 29(2), 118–131. https://doi.org/10.1002/eet.1835
Vellema, S., Maru, Y. T., Ekong, J., McNamara, P., Waters-Bayer, A., Watson, D., & Brouwers, J.
(2017). Do theories of change enable innovation platforms and partnerships to navigate
towards impact? Amsterdam, Netherlands: Royal Tropical Institute.
Wesley, A. S., De Plaen, R., Michaux, K. D., Whitfield, K. C., & Green, T. J. (2019). Integrating
nutrition outcomes into agriculture development for impact at scale: Highlights from the
Canadian International Food Security Research Fund. Maternal and Child Nutrition, 15(3),
e12812. https://doi.org/10.1111/mcn.12812
Westhorp, G. (2014). Realist Impact Evaluation. London, UK: ODI.
Wong, G., Westhorp, G., Greenhalgh, J., Manzano, A., Jagosh, J., & Greenhalgh, T. (2017). Quality
and reporting standards, resources, training materials and information for realist evaluation:
the RAMESES II project. Health Services and Delivery Research, 5, 28.
https://doi.org/10.3310/hsdr05280
163
CHAPTER 5: “WE HAVE A LOT OF THINGS TO DO:” ON PLANNING AND EVALUATING A COMPLEX FOOD SAFETY PROGRAM OPERATING UNDER A CHANGING CLIMATE
Abstract
The multifaceted nature of food safety often requires coordinated action across sectors to
ensure safe food. Yet, minimal guidance exists on how to design and measure the outcomes of
multisectoral programs that operate under climate change. Our objective is to explore the
experiences of food safety actors to consider how we might better develop and evaluate food
safety programs. To do so, we draw on a case study of an ongoing, multi-year development
program aiming to improve pork safety in informal food markets of Vietnam called SafePORK.
We thematically analyze data from remote qualitative interviews conducted with program
researchers (n=7) and program participants (n=23). Researchers found the process of reflecting
on pathways toward program goals helped them acknowledge, document, and respond to the
adaptive nature of complex programs. Program participants, particularly slaughterhouse workers
and retailers, shared how they are adapting to the impacts of climate change, an aspect not
originally considered within the design of the program. We identified three lessons: 1) develop
the program theory; 2) attune to climate change; and, 3) design with gender equity in mind.
When applied, these lessons have the potential to bring a new level of nuance and focus to
evaluations of complex food safety programs.
164
5.1 Introduction
Food safety risks are exacerbated by shocks to the food system, including environmental
degradation, market distortions, political upheavals, and now, the COVID-19 pandemic (Fanzo et
al., 2020). Increasingly coming into sharper focus is the shock of climate change, which has
multifaceted impacts on food safety. For instance, rising temperatures strengthen the survival of
foodborne pathogens (FAO et al., 2021). During flooding, food becomes unsafe to eat if the
food encounters contaminated floodwater (Mbow et al., 2019). The 2021 report on Food Security
and Nutrition in the World highlights that the number of people suffering from food insecurity is
increasing after decades of steady declines, contributed primarily by climate change and COVID-
19 (FAO et al., 2021). These impacts highlight the need for holistic action that addresses the
vulnerabilities of food safety to climate change and other drivers of food safety risk.
Recognizing the multifaceted nature of food safety, efforts are increasingly moving from
addressing a single driver of food safety to multisectoral approaches that strengthen food safety
(FAO et al., 2021; Havelaar et al., 2015). When governments, non-governmental organizations,
producers, consumers, and other actors in the agri-food sector work together, more effective
regulations, guidelines, and monitoring mechanisms can be created toward safer food. However,
these food control systems are often not feasible in low-resource settings where regulatory
enforcement capacity is weak and food production is highly fragmented and largely based on
small-scale producers (Bari & Yeasmin, 2018; Hoffmann & Jones, 2021; Vipham et al., 2020).
165
5.1.1 Complex food safety programs
Food safety is a shared responsibility among many different actors, including governments at all
levels, the agri-food sector, consumers, media, and food safety researchers and practitioners
(WHO, 2021). Alongside measures to improve food control systems, multisectoral food safety
programs often focus on health promotion, incentives for behavioural change, and/or the
enabling environment show promise in addressing food safety risks (Grace, 2017). Such
programs can be considered complex programs14, defined as programs with “several interacting
components” (Moore et al., 2014). The Medical Research Council lists several characteristics of
complex programs, including, but not limited to: the number of components involved; the
number of groups, settings, or levels targeted; and, the level of flexibility of the program
(Skivington et al., 2021).
While multi-component, multisectoral programs in addressing food safety are promising, there
exists limited guidelines supporting their development. Further insights into how complex food
safety programs can best be supported are critical, especially now in the context of climate
change. This article explores strategies complex food safety programs are adopting in order to
support the development of future related programs. To do so, we draw on a case study of an
14 Programs are complex not only because of the properties of the program but also the context into which the
program is introduced (Hawe, 2015; Skivington et al., 2021; Thirsk & Clark, 2017). Context is the pre-existing
conditions, or anything external to the program, that may affect implementation and outcomes. Complexity arises
from the interaction of program components with the context.
166
ongoing program aiming to improve food safety in Vietnam. Specifically, our objective is to
examine the experiences of researchers and participants involved in the program to characterize
how we might effectively implement and evaluate complex food safety programs.
5.1.2 Context: Food safety in Vietnam
Understanding how food safety programs in Vietnam work requires an understanding of
Vietnamese food systems. Because many Vietnamese have moved from the status of ‘having
enough food to eat’ to that of ‘needing safe food to eat’ (Nguyen-Viet et al., 2017), food safety
has become one of the most pressing issues among citizens (World Bank, 2016). Repeated
episodes of unsafe food practices receive widespread media attention in Vietnam (Nguyen-Viet
et al., 2017). A particular area of concern is pork safety, the main animal source food in
Vietnamese diets. Pork is often produced by smallholder systems and sold fresh in traditional
‘wet’ markets. Although wet markets are the primary access point for fresh vegetables, fruits,
and meat, wet markets often lack mechanisms for food safety control (Dang-Xuan et al., 2016).
Roles of smallholder pork producers in the pork value chain in Vietnam are gendered. Pig
collection and slaughtering activities are mostly done by men, whereas women are
predominately responsible for pork processing and retail (Mitchell, 2019). Because of gendered
roles, pork producers also have differential access to credit, resources, and agricultural training
(Tran et al., 2018). Research identifying opportunities for increased engagement of women in
agricultural value chains has highlighted constraints to participation, including social norms,
167
asset constraints, and time constraints (Quisumbing et al., 2021). Programs aiming to improve
food safety must be responsive to the differential needs of different value chain actors.
In Vietnam, the demand for pork products is increasing but regulatory systems have not been
able to adapt to this change in food consumption (Pham & Dinh, 2020). For example, Vietnam’s
Law on Food Safety (No. 55/2010/QH12) defines regulations for managing regular occurrences
of food safety incidences of food businesses (The National Assembly, 2010). However, these
regulations do not apply to those exempted from business registration such as smallholders
producers (Pham & Dinh, 2020). Food safety policy in Vietnam is gradually shifting from an
inspection-and-enforcement approach to a risk-based approach, a direction that many food
safety programs in low-resource settings are supporting (Nguyen-Viet et al., 2018; WHO, 2021).
Importantly, Vietnamese agriculture is vulnerable to the impacts of climate change, as Vietnam
has experienced and will continue to experience dramatic climate change impacts (Hanh et al.,
2020). Temperature changes, increased extreme weather events, and increased rainfall variability
are already threatening rice and aquaculture production in the Mekong Delta (Anh & Nghiep,
2020). While the impacts of climate change on food production and food security are well
known, knowledge gaps remain in our understanding of how climate change affects food safety
(FAO, 2020; Mbow et al., 2019). As food systems shift, food safety researchers and practitioners
must be cognizant of climate change risks and integrate climate change considerations into
their programming.
168
Since February 2019, African Swine Fever (ASF) has caused significant pig deaths across Vietnam.
Consequently, consumer demand for pork initially declined due to fears around food safety.
Once fears settled, the demand for pork rose. However, given the limited supply of pigs, pork
prices more than doubled (e.g. from 40.000 to 100.000 VNĐ per kg) (Chau, 2020). These shifts in
pork production and consumption patterns influence food safety programming in Vietnam. For
example, to maximize economic benefits, many pig farmers would raise heavier pigs, which
creates challenges in following guidelines for safe pork handling (e.g. difficulties lifting heavy pig
carcasses off the floor onto a grid for processing). Additionally, the enhanced biosecurity at pig
farms and slaughterhouses to control ASF limited access to facilities for food security
researchers and practitioners supporting existing food safety programs.
Since January 2020, Vietnam has been responding to COVID-19, which creates additional
challenges and opportunities for food safety programming. One notable challenge is the decline
in attention from local authorities engaged in food safety programming as their priorities shifted
to COVID-19 control. At the same time, pork producers are more concerned about public health
now compared to before the pandemic, improving interest in using personal protective
equipment during pork production (ILRI, 2020).
5.1.3 The SafePORK program
SafePORK is a five-year program (2017-2022) aiming to reduce the burden of foodborne disease
in informal markets of Vietnam through designing, piloting, and evaluating food safety
interventions (Lam et al., 2020). SafePORK builds on a previous five-year program called PigRISK
169
(2012-2017) (Lam et al., 2016). There are five core objectives of SafePORK: (1) generate
actionable evidence on the efficacy of approaches for improving pork safety; (2) develop and
test light-touch interventions toward food safety; (3) validate the theory of change for market-
based interventions; (4) identify recommendations for enhanced engagement of pig value chain
actors; and, (5) build capacity in food safety risk communication (Figure 5.1).
SafePORK is guided by One Health, a conceptual framework considered promising for
addressing food safety in low- and middle-income countries (LMICs) (Grace, 2017). Recognizing
that human, animal, and environmental health are inextricably linked, One Health calls for
multiple disciplines to work together on complex health challenges (Zinsstag et al., 2011).
Responding to this call, SafePORK mobilized researchers from different fields and institutions
including veterinary epidemiology (International Livestock Research Institute), public health
(Hanoi University of Public Health), agricultural economics (Vietnam National University of
Agriculture), and farming systems (National Institute of Animal Science). SafePORK also works
with program participants (local authorities, slaughterhouse workers, retailers, market managers)
to shape the direction of the research and actions that follow (Baum et al., 2006).
170
Figure 5.1. SafePORK program objectives and their interrelationships.
5.1.4 SafePORK locations
SafePORK takes place in Hung Yen (Red Mekong Delta), Nghe An (Central), and Hoa Binh
(Northwest) provinces (Figure 5.2). All SafePORK study sites were selected to build on previous
activities and relationships (e.g. from PigRISK). Hung Yen and Nghe An provinces represent rural
areas characterized by medium-scale and small-scale pork production systems, respectively. Hoa
Binh is a mountainous province inhabited mostly by ethnic minorities. Ethnic minority groups in
this province typically farm Ban pigs (lợn Bản), which are an Indigenous breed raised on a
171
traditional free scavenging feeding system. Undernutrition and stunting are particularly common
among ethnic minority women and children in Vietnam, requiring targeted efforts to reduce
health inequities (World Bank Group, 2019). SafePORK is a program comprised of multiple ‘light-
touch’ interventions addressing community priorities within the three study sites (Lam et al.,
2021).
Figure 5.2. Map showing locations of study sites (Hung Yen, Nghe An, Hoa Binh provinces) and
the capital city of Vietnam (Hanoi).
172
5.2 Methods
5.2.1 Data collection and analysis
We developed semi-structured interview guides for researchers and program participants
(Appendix 21). Interviews were conducted between August and September 2021. For SafePORK
researchers, interviews were carried out in English by the lead author (SL). For program
participants, interviews were conducted in Vietnamese by team members (SD-X, TL). Team
members had previous experience conducting qualitative interviews and received further
training from SL before data collection, as well as ongoing support throughout the study.
Participants were selected by team members based on occupation, age, and location, to ensure
a diversity of experiences. Interviews were audio-recorded with verbal informed consent and
varied in duration by group (e.g. 60-120 mins with researchers and 15-20 mins with program
participants).
We conducted a thematic analysis to identify patterns in the data using a deductive approach
(Braun et al., 2018). We paid specific attention to several elements. First, we considered how
context was reported as per guidelines for process evaluations of complex programs (Moore et
al., 2014). Secondly, we explored explicit or implicit mentions of climate change given its
growing impact on food security and other development programs (Lam, et al., 2021; Rowe,
2019; Uitto, 2021). Finally, we considered how differences and similarities in the social identities
of researchers and program participants (e.g. population, age, gender) shaped experiences with
173
SafePORK. This approach enabled us to explore how SafePORK is responding to the needs and
priorities of different groups of people.
5.2.2 Data validity and management
As a way of assuring research quality, we kept an audit trail (Appendix 22) of decisions made
and wrote reflective memos. Additionally, the authorship team regularly discussed key themes
to ensure the accuracy of results and interpretation. Coding and analyses were performed using
NVivo© software (QSR International, Burlington, MA, USA).
5.2.3 A note on COVID-19 and ethics
At the time of this study, Vietnam experienced a surge in COVID-19 cases. Due to travel
restrictions, all interviews were conducted remotely via Microsoft Teams or by phone (see
Appendix 23 for notes on the impacts of COVID-19 on data collection). This research was made
possible through collaboration with the coordinators of SafePORK (H-NV, FU) who provided
regular guidance on this work. Additionally, the first author (SL) – and the primary research
instrument – had existing rapport with the SafePORK team as he worked as an evaluation
consultant for SafePORK. The research findings around barriers and facilitators to
implementation were reported separately and used by the team to adjust ongoing activities
(Appendix 24).
Rapport was also pre-established between S-DX, TL, and program participants. As rapport
facilitated access to research participants, the authorship team continually reflected on whether
174
interviewees felt they had to participate, especially during a pandemic. All interviewees verbally
provided their informed consent to participate before the interview. Program participants also
received 50.000 VNĐ (approximately US$2.2) as compensation for their time. To protect the
identity of interviewees, while also allowing readers to see the breadth of interviews, we
identified quotations from each interviewee with a number assigned to the interviewee. This
work received research ethics approval from the University of Guelph (20-02-003) and Hanoi
University of Public Health (018-110/DD-YTCC).
5.3 Results
Seven researchers (4 men, 3 women) of the SafePORK team were interviewed. Researchers came
from public health (n=3), agricultural economics (n=3), veterinary science (n=2), and farming
systems (n=1) backgrounds. A total of 23 program participants also participated in an interview.
They included slaughterhouse workers (n=10), retailers (n=9), local authorities (n=3), and market
managers (n=1) from Hung Yen (n=13), Nghe An (n=7), and Hoa Binh provinces (n=3). The
market manager and two out of three local authorities were men. All slaughterhouse workers
interviewed were men and all but one retailer were women. There were two husband-and-wife
teams whereby the husband would process the pork for the wife to sell. Participants worked
their occupation anywhere from two to 35 years and engaged with SafePORK anywhere from
one to three years. Apart from two ethnic minority interviewees in Hoa Binh, all program
participants were the ethnic majority (n=22) (Appendix 25).
175
In reviewing factors influencing implementation, we developed three overarching themes that
speak to how complex food safety interventions might be effectively supported: using program
theory to track changes to the program, attuning to climate change, and designing with equity
in mind (Appendix 26).
5.3.1 “We imagine how it can work in practice:” Using program theory to track changes to the program
The experiences of the SafePORK team suggested interventions were strongly influenced by
context (Figure 5.3). Six months after the launch of interventions (August 2020), the team
conducted formative research in Hanoi to identify strategies for improving the uptake of
hygienic practices. The team noted the adoption of practices was quicker in urban areas than
rural areas, likely because consumers there were more aware of food safety concerns. Urban
consumers tend to have greater risk perceptions because of higher reported food safety
incidents in urban regions (Ha et al., 2021). One researcher said: “As long as consumers are
concerned about microbial contamination, slaughterhouse workers and retailers have to do it
[safe practice]” (Interviewee #6). Another researcher, who previously worked on a project
involving Ban pig production in another province (Son La), reflected on how working in Hoa
Binh was easier because local authorities were concerned about Ban pigs. The scale of pork
production operations also influenced interventions. One researcher described difficulties
working in Hung Yen because of larger-scale pig operations compared to Nghe An:
176
you can imagine from 2 am to 5 am, they have a few hours to slaughter seven or eight
pigs. So they have to do it quickly. Not like in Nghe An where there is sometimes only
one pig (Interviewee #4).
In response to these contextual factors, adaptations were made to SafePORK interventions. For
example, the program co-invested with slaughterhouse owners in Hung Yen in an additional
grid to facilitate the medium-scale pork processing. In Nghe An, only one grid was considered
necessary for the small-scale pork operations. SafePORK also adjusted engagement strategies. In
Hoa Binh, more time was dedicated to working with ethnic minority communities (often through
a translator) and adapting interventions to respond to priorities of ethnic minority communities
(e.g. focus on Ban pig production rather than slaughterhouse and market improvement). These
actions suggested complex food safety programs require tailoring of activities within specific
program components rather than a one-size-fits-all approach to the entire program. Reflecting
on the ongoing nature of program design, one researcher noted “we have a lot of things to do.”
The flexibility and adaptability of SafePORK led to emergent outcomes. For example, in response
to travel restrictions resulting from ASF control measures, researchers reported they were careful
when entering sites as they could be the vector bringing the virus to the pigs. The inability to
access certain slaughterhouse facilities led the team to focus on other activities such as raising
food safety awareness and improving pork preparation practices of consumers. Shifting the
focus from slaughterhouse workers to other food safety actors was viewed as an important
177
strategy for indirectly supporting slaughterhouse workers. The team also reported increased
engagement with local authorities during this time, resulting in the agreement of local
authorities to certify ‘model’ small-scale slaughterhouses (e.g. those that meet some but not all
conditions of the Food Safety Law 2010), which was viewed as a notable achievement.
To track learnings from context, adjustments in program activities, and the emergence of
outcomes, the team reflected on the development of a program theory. To do so, the team used
Theory of Change and Outcome Mapping, two approaches often used within evaluations of
complex health interventions (Lam et al., 2021). One researcher described developing the
program theory to “imagine how it [SafePORK] can work in practice” (Interviewee #1). The
researcher then described using Outcome Mapping to track “changes in knowledge, attitudes,
and practices of value chain actors, and even researchers” identified by the Theory of Change
process. Operationally, monitoring of outcomes – and the strategies and context influencing
outcomes –was conducted after each field visit. According to the team, developing the program
theory using information collected from Theory of Change and Outcome Mapping is helping
them understand how, why, and under what context the program is contributing to safer pork.
178
Figure 5.3. Timeline of implementation, key milestones, and barriers of SafePORK.
5.3.2 “When it rains, you have to cover the pork:” Attuning to climate change
When asked whether climate change was a factor influencing implementation, most researchers
believed climate factors had potential impacts on the program but noted evidence was lacking.
At the farm level, some researchers mentioned hearing from farmers that pigs are “more
sensitive and get sick more frequently” (Interviewee #7) from climate variation. To mitigate this
challenge, one local authority in Hung Yen reported some farmers would “do things that are not
needed” (lam những thứ không cần thiết) like using veterinary drugs, which contributed to
production costs and concerns of chemical residues in pork (Interviewee #10).
179
Researchers and slaughterhouse workers commented on the impact of heatwaves on pigs and
pork during transportation. One slaughterhouse worker in Hung Yen explained how pigs are
transported using trucks equipped with water sprays to prevent heat stress. Though important
for supporting pigs, he acknowledged such practices were water- and energy-intensive. One
researcher noted it is common for pig carcasses to be transported uncovered and un-chilled on
the backs of motorbikes, especially in mountainous areas of Vietnam (e.g. Hoa Binh).
For slaughterhouse workers, times of drought also created challenges when processing pork.
One slaughterhouse worker in Hoa Binh with 22 years of experience explained because he
sourced water from the mountains, there was sometimes limited water availability, especially
during the dry season. And during periods of extreme flooding, researchers discussed the
possibilities of rain contaminating waterways through run-off from nearby farms. Researchers
reported many slaughterhouse workers used these open water sources to process pig carcasses
and/or clean tools, resulting in increased food safety risks from using contaminated floodwater.
Both retailers and slaughterhouse workers also mentioned that during storms, rain sometimes
blocked travel and prevented pork from reaching consumers. For those without trucks, they “try
to overcome this” (cũng cố gắng để khắc phục) (Interviewee #14) by using tarps to cover the
pork and traveling more carefully. Additionally, researchers reported heavy rain often prevented
them from traveling to the field:
180
In Nghe An, Hung Yen, wherever we go, we could access these places to take samples or
discuss with stakeholders or just to do the interventions. For Hoa Binh, this was much
more difficult in the rainy season where there can landslides (Interviewee #9).
For some food safety actors, infrastructure was a factor influencing the extent of impacts from
rain. One slaughterhouse worker with 22 years of experience processing and selling pork
explained: “Rainy weather does not affect much because I work at home with a roof, only when
selling at the market there were impacts (because there are no shelters at this particular
market)” (Trời mưa gió không ảnh hưởng lắm do mình làm ở nhà có mái che, còn chỉ khi bán ở
chợ thì chỗ bán bị ảnh hưởng hơn) (Interviewee #17). Retailers were very careful to avoid getting
rain on the meat because it will “make the meat mushy” (làm nhã hết thịt) (Interviewee #15),
which detracted customers.
Retailers were particularly concerned about how heat spoiled pork faster. For example, one
retailer in Hung Yen with 15 years of experience explained “when the weather is hot, it can make
your meat worse, it is not fresh and beautiful anymore” (như thời tiết oi ả, thì có thể làm miếng
thịt của mình xấu hơn đi, nó không được tươi, đẹp bắt mắt nữa) (Interviewee #15). Retailers also
described how heat made meat ‘wet’ and would use cardboard to help absorb moisture. This
practice was observed by researchers, who emphasized placing meat on cardboard was
unsanitary. The market manager commented on how there were more flies around pork during
hot weather.
181
Many researchers also mentioned heatwaves as a possible driver of microbial replication in pork
though could not say whether this driver was from climate change or simply seasonal influences.
One researcher reflected: “In Nghe An, in the summer, we had some days that were too hot. To
keep and maintain the quality of pork it is hard for retailers, they do not have refrigerators to
store the meat” (Interviewee #6). Another researcher explained retailers do not have
refrigerators because of the market infrastructure: “It is not their place. So, they cannot do that”
(Interviewee #9).
Asking questions about climate change revealed SafePORK activities are indirectly contributing
to climate change adaptation. One researcher believed following recommended hygienic
handling practices from SafePORK is one way of adapting to climate change: “if all interventions
are working they will adapt with those [climate] changes well. If hygiene is practiced well,
whatever summer or winter, will still have a good effect on microbiological levels” (8). Recalling a
risk communication activity conducted earlier in 2021, one researcher alluded to the important
role of keeping meat chilled during heatwaves when he stated:
Some small wet markets operate during the afternoon, between 4 to 7 pm. But most wet
markets operate early in the morning. We encouraged consumers to buy the pork early
morning and put it in the refrigerator. So that in the evening they use it for cooking
(Interviewee #2).
182
5.3.3 “We always talk about gender in SafePORK:” Attuning to gender equity
Researchers emphasized the importance of designing programs with equity in mind. This view
was also reflected in objective three of SafePORK (ensuring equitable benefits among different
groups). The team stated roles within the pork value chain in Vietnam were highly gendered,
with pork processing viewed as a man’s job, whereas pork retailing was mostly done by women.
Indeed, all slaughterhouse workers interviewed were men and nearly all retailers interviewed
were women. As such, gender intersected with occupation in this context.
After participating in SafePORK, slaughterhouse workers noticed their businesses were doing
better than before, as one slaughterhouse worker in Nghe An with seven years of experience
explained “because customers see [slaughtering practices] directly, it’s clean, they like it, and
they introduce us other customers who buy more pork” (họ nhìn thấy trực tiếp sạch sẽ là họ
thích, họ ủng hộ, giới thiệu khách khác và mua nhiều hơn) (Interviewee #16). Retailers described
how meat is now cleaner, more beautiful, and had less odor than before, helping them attract
more customers. One retailer in Hung Yen shared she used to sell around sold 20 kg of pork per
day, but now she can sell 25-30 kg. Other outcomes reported by both slaughterhouse workers
and retailers included cleanliness of the work area, trust from consumers, and knowledge on
how to maintain hygiene during pork handling.
Slaughterhouse workers and retailers recalled key messages from the training sessions,
highlighting the contributions of SafePORK interventions in building food safety knowledge:
183
Prevent knives from rusting… maintain cleanliness… wash pigs before slaughtering (dao
không để chỗ cao ráo nó han rỉ ra… vệ sinh chân tay… tắm cho sạch sẽ đã rồi mới giết mổ)
(Interviewee #21).
Raw and cooked food must be kept separate, or when you are done have to wash the
table (thớt dao đồ sống đồ 183hin phải để riêng, hoặc làm xong đau đấy mình phải rửa
bàn) (Interviewee #25).
Whether knowledge was applied in practice was dependent on certain enabling factors that
differed between food safety actors. For slaughterhouse workers, the convenience of proposed
recommendations was valued. They appreciated how equipment (e.g. grid, table) helped keep
the floor clean and relieved them of “not worrying about borrowing (from others)” (không lo
phải đi mượn) (Interviewee #18). For retailers, having access to water at the market was essential
for applying learned hygienic practices: “now that there’s water I’ll wash my hands, wash the
cutting board, and clean seats, then sell pork without the smell anymore” (có nước là chị rửa tay
sạch sẽ, rửa dao thớt và chỗ ngồi sạch sẽ, sau đấy bán nó không có mùi hôi thối nữa) (Interviewee
#24).
The reasoning behind the adoption of recommended practices slightly differed between food
safety actors. Both slaughterhouse workers and retailers talked about the need to maintain the
cleanliness of facilities to attract customers and protect the quality of pork. For slaughterhouse
workers, adopting safe hygienic practices was a way to protect their health (e.g. from rusty
184
knives, dirty water) and the economic well-being of their families. Retailers more often than
slaughterhouse workers emphasized that changing habits was important for protecting
consumers’ health. As, as one retailer explained, “when we make the product clean, people’s
health will be guaranteed when they eat and drink” (mình làm sản phẩm nó sạch sẽ thì người
dân khi họ ăn uống vào sẽ đảm bảo sức khỏe) (Interviewee #16).
Food safety actors also faced unique and similar challenges, resulting from gendered differences
in occupation. Slaughterhouse workers, particularly those in medium-scale pork operations,
faced time pressures from having to process many pigs within a few hours (e.g. 2 am to 5 am).
For retailers we interviewed, time was also a scarce resource. One retailer described how she
could only clean her cutting boards when there were no customers: “I cut pork for this person
first, and when no one has come to buy pork, I bend down to wash quickly before someone
buys pork and I cut it again” (mình cắt cho người này trước, chưa có người đến mua mình cúi
xuống mình rửa lại có người ta mua mình lại cắt tiếp tục) (Interviewee #24). No matter how busy
they were, retailers said they had to practice safe hygiene or they would lose customers.
Slaughterhouse workers tended to work fewer hours compared to retailers. These findings
further suggested different genders provide much labour in pork value chains in Vietnam
(Mitchell, 2019).
Gendered roles in the pork value chain were also a factor in determining exposures to various
risks. Where men mainly worked in slaughterhouses, they reported the need to exercise caution
185
with handling knives and preventing contact with dirty water. Because of the open-air market
environment where retailers worked, retailers more frequently than slaughterhouse workers
reported challenges associated with climatic events (e.g. the impact of extreme heat on pork
quality). With ASF, both retailers and slaughterhouse workers worried about sourcing pork/pigs
unaffected by the disease. Understanding these differences in risk exposures helped the team
manage risks faced by different pork value chain actors.
We also explored intra-gender differences between social groups. For example, examining how
gender intersects with ethnicity, we considered whether ethnic majority and ethnic minority men
had differential experiences with the program. While there were only two ethnic minority men
interviewed, it is worth noting that both interviewees reported challenges from heavy rain, while
few ethnic majority men reported facing this challenge. This difference in experiences was
primarily due to geographic reasons. The ethnic minorities SafePORK worked with primarily lived
in mountainous areas where heavy rain often led to poor driving conditions and thus difficulty
transporting pork to the market. Additionally, due to differences in travel restrictions across
geography, ethnic minority men reported the COVID-19 pandemic decreased sales (up to 50%)
while few ethnic majority men reported impacts of COVID-19 on their businesses. Exploring
intra-gender differences between younger and older women, younger women referred to the
same constraints that older women face (e.g. time, such as time resources split between cleaning
and selling pork). However, younger retailers tended to adopt hygienic practices more readily
compared to older retailers likely because younger retailers were more likely to listen to
186
SafePORK researchers, many of whom are older or around the same age. Younger retailers also
might not have yet adopted ingrained practices. Identifying these patterns was a strategy to
understand progress toward achieving objective three of SafePORK (ensuring equitable benefits
among different groups).
Paying attention to gender roles also helped the team identify key individuals who can adopt
changes to food safety practices. For example, researchers shared how joint decisions were
made between husband and wife in upgrading their slaughterhouse, particularly in Hung Yen
and Nghe An. In this case, researchers emphasized the need to exchange ideas about the
intervention with both the husband and wife to improve uptake with the intervention.
When asked whether SafePORK contributed to behavioural changes in food safety practice,
slaughterhouse workers and retailers acknowledged the role of training, equipment, and
interactions with and reminders from the team. However, given the many changes to context
(e.g. ASF, COVID-19) during the duration of SafePORK programming, SafePORK was likely one of
many factors involved. One retailer in Hung Yen with 10 years of experience alluded to
economic development and shifting consumption patterns when she shared: “Before, I sold of
whatever quality is okay, but now everyone’s life is better, people have to eat clean” (tại vì trước
thì mình bán tào lao thế nào cũng được, chứ bây giờ cuộc sống của mỗi người nó một khá lên)
(24). These economic factors contributed to the adoption of safe food practices among different
food safety actors. In other ways, contextual factors affected different groups differently. For
187
example, the work in slaughterhouses remained relatively unchanged during COVID-19. In
contrast, many retailers stressed the importance of handwashing and sanitizing liquid for
protecting their health and community health since COVID-19. Both COVID-19 prevention
measures and SafePORK food safety measures focused on proper hand hygiene. SafePORK also
started communicating how handwashing was not only needed to address COVID-19 but was an
important step in preventing food-borne disease. Understanding change processes as
contextual was helpful for the team in engaging with the context-specific realities of various
food safety actors.
5.4 Discussion
The increasing complexity of food security programs due to climate change and other
contextual factors introduces challenges for evaluation (Hendriks, 2020). In this paper, we add to
the limited evidence base on assessments of complex food safety programs through an analysis
of an ongoing program in Vietnam aiming to reduce foodborne diseases in informal markets.
We show how using program theory, attuning to climate change, and attuning to gender equity
are potential strategies for evaluating programs characterized by complexity. Many of these
evaluation strategies align with solutions proposed for a more sustainable and just food systems
transformation (Queiroz et al., 2021).
Guidance for considering context within public health intervention research encourages asking
“how does the intervention interact with its context?” at each phase of the intervention research
188
(Skivington et al., 2021). However, in settings where external factors may change rapidly,
monitoring of context on an ongoing basis will be critical. Additionally, because complex
programs can have components with varying levels of complexity (Foran et al., 2014), different
program components will likely require different evaluations. In SafePORK, conventional
approaches to evaluation are being applied to assess components relatively unaffected by
context. For example, risk assessments in SafePORK aim to provide an estimate of risks to
human health at a particular point in time; because the nature and purpose of these
assessments are quite straightforward, the team is measuring changes in risk by comparing
baseline and endline risk estimates. Conversely, risk management involves identifying,
implementing, and monitoring control measures over a longer period. For risk management and
other multidimensional components, the team finds it helpful to develop a program theory,
which provides a structure for identifying and tracking influences from changing contexts on
program components in real-time. Of note, developing a program theory is one of many
strategies for considering context. Another promising strategy for assessing program context
change – though not used in our study – is to explore how context relates to constructs of
complexity theory (e.g. emergence, feedback, self-organization) (Bleecker et al., 2021).
An important contextual factor contributing to program complexity and uncertainty is climate
change. Although the integration of climate considerations into program planning and
evaluation within development contexts is increasingly encouraged (Rowe, 2019; Ssekamatte,
2018; Uitto 2019), the impacts of climate change on food security programs remain overlooked
189
(Lam et al., 2021). Our findings showed asking questions about climate change – for a food
safety program where climate action was not an explicit objective – helped to uncover climate
vulnerabilities and adaptation strategies, highlighting the importance of climate change
mainstreaming in evaluation. Furthermore, research on the impact of climate change on food
safety has typically focused on the end food product (Fanzo et al., 2021); an important
contribution of this study is how intermediary links within the food value chain (e.g. processing,
retailing) are influenced by climate change, which can support programs aiming for more
holistic action toward food safety.
The social context in which programs are implemented is another critical dimension in
evaluations. For example, climate change will continue to have a disproportionate impact on
certain groups of people (Mbow et al., 2019). As our findings revealed, retailers reported
experiencing more severe disruptions to their livelihoods from climate change compared to
slaughterhouse workers. Additionally, food safety actors can have different rationales for
participating in food security programs. Feminist evaluators have acknowledged “there is
tremendous but largely untapped potential for evaluation to deepening understanding around
gender inequities” (Hay, 2012, p. 337). Our study examined program mechanisms – understood
as the reasoning and responses of participants – that produced intended program outcomes by
asking “how did the program contribute to change?” (Dalkin et al., 2015). In doing so, we
contributed an example of how gender considerations can go beyond simply reporting
outcomes for different groups of people to include reporting on program mechanisms (Dalkin et
190
al., 2015; Lam et al., 2019). We suggest understanding program mechanisms can facilitate the
tailoring of food safety interventions toward equitable outcomes.
Importantly, food security challenges differ between ethnic majority and minority communities.
For example, extended maternity leave and restricted infant formula advertising are policies
designed to address Vietnam’s low breastfeeding rates. While seen as a major accomplishment
for the nutrition community, these policies target women working in the formal sector and thus
do not address issues faced by most ethnic minority women (Harris et al., 2021). As in Tavenner
& Crane (2019), applying intersectionality to an agricultural systems context, analyzing intra-
group differences helped identify factors that shape vulnerability. Programs should take into
account the particular needs and priorities of marginalised ethnic minority groups to address
determinants of food insecurity. Additionally, evaluations need to be sensitive to women’s and
men’s heterogeneity and pay attention to the differences in social positions.
We suggested several areas for future research. First, analyzing trade-offs and co-benefits
between economic productivity and safe food handling practices will be important for
identifying sustainable food safety pathways (Antle & Valdivia, 2021; Foran et al., 2014). For
example, food safety practices such as handwashing and cleaning work areas are being adopted
but maintaining these practices is not easy when slaughterhouse workers need to process pork
quickly or when retailers are dealing with multiple customers at once. Secondly, smallholder
producers are increasingly affected by government policies aiming to formalize, centralize, and
191
intensify pork production systems. Given the potential public health, environmental, and animal
welfare impacts of industrialization of food animal production in low- and middle-income
countries (Lam et al., 2019), there is a need to continue legitimatizing and supporting informal
markets. As this study demonstrated, food safety actors were supportive of risk management
practices. Continuing to develop alternative approaches to reduce food safety risks – e.g. via
capacity building, adoption incentives, and an enabling environment – is important for
supporting the livelihoods of food safety actors and community food safety. Finally, such
programs often call for multidisciplinarity; however, the process and outcomes of
multidisciplinary collaborations are not well examined within food security contexts (Gaihre et
al., 2019; Lam, et al., 2019; Nguyen-Viet et al., 2021). To enhance multidisciplinary collaborations
toward food security under a changing climate, future research should continue exploring
strategies for working across disciplines and sectors in this area.
5.4.1 Limitations
We note a couple of limitations of our study. First, retailers and slaughterhouse workers largely
shared positive feedback, likely due to familiarity with the interviewer. After the interviewers
made this observation, interviewers reminded participants to openly share feedback on both
what worked well and what could be done better, so that the SafePORK team can better plan
future programming activities. Furthermore, additional questions were added to the interview
guide to probe for suggestions for improvement. Secondly, because interviews were conducted
remotely, many program participants were joining the interview from their homes. Children,
192
customers, pets, and/or livestock sometimes created background noise that interrupted the flow
of the interview. Despite these limitations, this research enhanced our understanding of how
complex food safety programs could be effectively implemented and evaluated.
5.5 Conclusion
As food safety is a shared responsibility among many different actors in the agri-food sector,
complex programs – those that have multiple interacting components at multiple levels – are
often key to achieving desired food safety outcomes. However, complex food safety programs
are often challenging to evaluate and learn from, especially now given additional complexities
introduced by the changing climate. This study explored the experiences of the researchers and
program participants involved in an ongoing program aiming to reduce foodborne diseases in
Vietnam. Using program theory, attuning to climate change, and ensuring inclusivity were
identified as key strategies for implementing and evaluating complex food safety programs.
Although findings from this research are specific to Vietnam, these strategies can inform
contexts where researchers and evaluators are looking to measure the outcomes of multisector,
multicomponent programs operating under a changing climate.
193
5.6 References
Anh, D.T., Nghiep, P. C. (2020). New challenges for food security in Vietnam.
Antle, J. M., & Valdivia, R. O. (2021). Trade-off analysis of agri-food systems for sustainable
research and development. Q Open. https://doi.org/10.1093/qopen/qoaa005
Bari, M. L., & Yeasmin, S. (2018). Foodborne Diseases and Responsible Agents. In Food Safety
and Preservation. https://doi.org/10.1016/b978-0-12-814956-0.00008-1
Baum, F., MacDougall, C., & Smith, D. (2006). Participatory action research. Journal of
Epidemiology and Community Health, 60(10), 854–857.
https://doi.org/10.1136/jech.2004.028662
Bleecker, L., Sauveplane-Stirling, V., Di Ruggiero, E., & Sellen, D. (2021). Evaluating the
integration of strategic priorities within a complex research-for-development funding
program. Evaluation and Program Planning.
https://doi.org/10.1016/j.evalprogplan.2021.102009
Braun, V., Clarke, V., Hayfield, N., & Terry, G. (2018). Thematic analysis. In P. Liamputtong (Ed.),
Handbook of Research Methods in Health Social Sciences.
Chau, H. (2020). Scarcity sends Vietnam pork prices to 20-year high. VN Express.
https://e.vnexpress.net/news/business/industries/scarcity-sends-vietnam-pork-prices-to-
20-year-high-4103700.html
Craig, P., Di Ruggiero, E., Frohlich, K. L., Mykhalovskiy, E., & White, M. (2018). Taking account of
context in population health intervention research: guidance for producers, users and funders
of research.
Dalkin, S. M., Greenhalgh, J., Jones, D., Cunningham, B., & Lhussier, M. (2015). What’s in a
mechanism? Development of a key concept in realist evaluation. Implementation Science.
https://doi.org/10.1186/s13012-015-0237-x
Dang-Xuan, S., Nguyen-Viet, H., Meeyam, T., Fries, R., Nguyen-Thanh, H., Pham-Duc, P., Lam, S.,
Grace, D., & Unger, F. (2016). Food safety perceptions and practices among smallholder
pork value chain actors in Hung Yen Province, Vietnam. Journal of Food Protection.
https://doi.org/10.4315/0362-028X.JFP-15-402
Fanzo, J., Bellows, A. L., Spiker, M. L., Thorne-Lyman, A. L., & Bloem, M. W. (2021). The
importance of food systems and the environment for nutrition. American Journal of Clinical
Nutrition. https://doi.org/10.1093/ajcn/nqaa313
194
Fanzo, J., Covic, N., Dobermann, A., Henson, S., Herrero, M., Pingali, P., & Staal, S. (2020). A
research vision for food systems in the 2020s: Defying the status quo. Global Food Security.
https://doi.org/10.1016/j.gfs.2020.100397
FAO, IFAD, UNICEF, W. and W. (2021). The State of Food Security and Nutrition in the World 2021.
FAO. (2020). Climate change: Unpacking the burden on food safety.
Foran, T., Butler, J. R. A., Williams, L. J., Wanjura, W. J., Hall, A., Carter, L., & Carberry, P. S. (2014).
Taking complexity in food systems seriously: An interdisciplinary analysis. World
Development, 61, 85–101. https://doi.org/10.1016/j.worlddev.2014.03.023
Gaihre, S., Kyle, J., Semple, S., Smith, J., Marais, D., Subedi, M., & Morgan, H. (2019). Bridging
barriers to advance multisector approaches to improve food security, nutrition and
population health in Nepal: transdisciplinary perspectives. BMC Public Health.
https://doi.org/10.1186/s12889-019-7204-4
Grace, D. (2017). Food safety and the Sustainable Development Goals.
Ha, T. M., Shakur, S., & Pham Do, K. H. (2021). Food risk in consumers’ eye and their
consumption responses: evidence from Hanoi survey. Journal of Asian Business and
Economic Studies. https://doi.org/10.1108/jabes-12-2019-0126
Harris, J., Huynh, P., Nguyen, H. T., Hoang, N., Mai, L. T., Tuyen, L. D., & Nguyen, P. H. (2021).
Nobody left behind? Equity and the drivers of stunting reduction in Vietnamese ethnic
minority populations. Food Security. https://doi.org/10.1007/s12571-021-01183-7
Havelaar, A. H., Kirk, M. D., Torgerson, P. R., Gibb, H. J., Hald, T., Lake, R. J., Praet, N., Bellinger, D.
C., de Silva, N. R., Gargouri, N., Speybroeck, N., Cawthorne, A., Mathers, C., Stein, C., Angulo,
F. J., Devleesschauwer, B., Adegoke, G. O., Afshari, R., Alasfoor, D., … Zeilmaker, M. (2015).
World Health Organization Global Estimates and Regional Comparisons of the Burden of
Foodborne Disease in 2010. In PLoS Medicine.
https://doi.org/10.1371/journal.pmed.1001923
Hawe, P. (2015). Lessons from Complex Interventions to Improve Health. Annual Review of Public
Health, 36(1), 307–323. https://doi.org/10.1146/annurev-publhealth-031912-114421
Hay, K. (2012). Engendering Policies and Programmes through Feminist Evaluation:
Opportunities and Insights. Indian Journal of Gender Studies, 19(2), 321–340.
https://doi.org/10.1177/097152151201900208
Hendriks, S. (2020). Food security policy, evaluation and impact assessment. Earthscan.
195
Hoffmann, V., & Jones, K. (2021). Improving food safety on the farm: Experimental evidence from
Kenya on incentives and subsidies for technology adoption. World Development.
https://doi.org/10.1016/j.worlddev.2021.105406
ILRI. (2020). Can COVID-19 spur transformations in food safety in traditional Vietnamese pork
value chains? https://www.ilri.org/news/can-covid-19-spur-transformations-food-safety-
traditional-vietnamese-pork-value-chains
Lam, S., Nguyen, H.T.T., Nguyen-Viet, H., Unger, F. (2020). Mapping pathways toward safer pork in
Vietnam.
Lam, S., Barot, M., Nguyen-Viet, H., & Unger, F. (2016). Changes in researcher capacity in
assessing food safety risks and value chains: Insights from PigRisk team. ILRI. Hanoi.
Lam, S., Dodd, W., Whynot, J., & Skinner, K. (2019). How is gender being addressed in the
international development evaluation literature? A meta-evaluation. Research Evaluation,
28(2), 158–168. https://doi.org/10.1093/reseval/rvy042
Lam, S., Nguyen, H. T. T., Tuan, H. N. H., Nguyen, L. T., Nguyen-Viet, H., Toribio, J.-A., Thanh, H. L.
T., Pham-Van, H., Grace, D., & Unger, F. (2021). Unpacking the Theory Behind One Health
Food Safety Programs : A Vietnam Case Study. Frontiers in Veterinary Science, 8(December),
763410. https://doi.org/10.3389/fvets.2021.763410
Lam, S., Dodd, W., Berrang-Ford, L., Ford, J., Skinner, K., Papadopoulos, A., & Harper, S. L. (2021).
How are climate actions evaluated? A review of United Nations food security evaluations. In
Global Food Security. https://doi.org/10.1016/j.gfs.2021.100509
Lam, S., Dodd, W., Wyngaarden, S., Skinner, K., Papadopoulos, A., & Harper, S. L. (2021). How and
why are Theory of Change and Realist Evaluation used in food security contexts? A scoping
review. Evaluation and Program Planning, 89(September), 102008.
https://doi.org/10.1016/j.evalprogplan.2021.102008
Lam, S., Thompson, M., Johnson, K., Fioret, C., & Hargreaves, S. K. (2019). Toward community
food security through transdisciplinary action research. Action Research.
https://doi.org/10.1177/1476750319889390
Lam, Y., Fry, J. P., & Nachman, K. E. (2019). Applying an environmental public health lens to the
industrialization of food animal production in ten low- A nd middle-income countries.
Globalization and Health. https://doi.org/10.1186/s12992-019-0479-5
Mbow, C., Rosenzweig, C., Barioni, L. G., Benton, T. G., Herrero, M., Krishnapillai, M., Liwenga, E.,
Pradhan, P., Rivera-Ferre, M. G., Sapkota, T., Tubiello, F. N., & Xu, Y. (2019). Food Security. In
196
P. Z. P.R. Shukla, J. Skea, E. Calvo Buendia, V. Masson-Delmotte, H.-O. Pörtner, D.C. Roberts,
J. P. P. R. Slade, S. Connors, R. van Diemen, M. Ferrat, E. Haughey, S. Luz, S. Neogi, M.
Pathak, J. Petzold, & J. M. P. Vyas, E. Huntley, K. Kissick, M. Belkacemi (Eds.), Climate Change
and Land: an IPCC special report on climate change, desertification, land degradation,
sustainable land management, food security, and greenhouse gas fluxes in terrestrial
ecosystems.
Mitchell, M. E. V. (2019). A review of the literature: gender, food safety and the pork value-chain in
Vietnam.
Moore, G., Audrey, S., Barker, M., Bond, L., Bonell, C., Cooper, C., Hardeman, W., Moore, L.,
O’Cathain, A., Tinati, T., Wight, D., & Baird, J. (2014). Process evaluation in complex public
health intervention studies: The need for guidance. In Journal of Epidemiology and
Community Health. https://doi.org/10.1136/jech-2013-202869
Nguyen-Viet, H., Grace, D., Pham-Duc, P., Dang-Xuan, S., Luu-Quoc, T., Unger, F., de Vlieger, S.,
Pham-Thi, N., Duong-Van, N., Nguyen-Hung, L., Tran-Dinh, L., & Tuyet-Hanh, T. T. (2018).
Research and training partnership to assist policy and capacity building in improving food
safety in Vietnam. In Global Food Security. https://doi.org/10.1016/j.gfs.2018.08.009
Nguyen-Viet, H., Pham, G., Lam, S., Pham-Duc, P., Dinh-Xuan, T., Jing, F., Kittayapong, P.,
Adisasmito, W., Zinsstag, J., & Grace, D. (2021). International, Transdisciplinary, and
Ecohealth Action for Sustainable Agriculture in Asia. Frontiers in Public Health.
https://doi.org/10.3389/fpubh.2021.592311
Nguyen-Viet, H., Tuyet-Hanh, T. T., Unger, F., Dang-Xuan, S., & Grace, D. (2017). Food safety in
Vietnam: Where we are at and what we can learn from international experiences. In
Infectious Diseases of Poverty (Vol. 6, Issue 1). https://doi.org/10.1186/s40249-017-0249-7
Pham, H. V., & Dinh, T. L. (2020). The Vietnam’s food control system: Achievements and
remaining issues. Food Control. https://doi.org/10.1016/j.foodcont.2019.106862
Queiroz, C., Norström, A. V., Downing, A., Harmáčková, Z. V., De Coning, C., Adams, V., Bakarr,
M., Baedeker, T., Chitate, A., Gaffney, O., Gordon, L., Hainzelin, É., Howlett, D., Krampe, F.,
Loboguerrero, A. M., Nel, D., Okollet, C., Rebermark, M., Rockström, J., … Matthews, N.
(2021). Investment in resilient food systems in the most vulnerable and fragile regions is
critical. In Nature Food. https://doi.org/10.1038/s43016-021-00345-2
Quisumbing, A., Heckert, J., Faas, S., Ramani, G., Raghunathan, K., Malapit, H., Malapit, H.,
Heckert, J., Eissler, S., Faas, S., Martinez, E., Myers, E., Pereira, A., Quisumbing, A., Ragasa, C.,
Raghunathan, K., Rubin, D., & Seymour, G. (2021). Women’s empowerment and gender
197
equality in agricultural value chains: evidence from four countries in Asia and Africa. Food
Security. https://doi.org/10.1007/s12571-021-01193-5
Rowe, A. (2019). Sustainability-Ready Evaluation: A Call to Action. New Directions for Evaluation,
2019(162), 29–48. https://doi.org/10.1002/ev.20365
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N.,
French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A
new framework for developing and evaluating complex interventions: Update of Medical
Research Council guidance. The BMJ. https://doi.org/10.1136/bmj.n2061
Ssekamatte, D. (2018). The role of monitoring and evaluation in climate change mitigation and
adaptation interventions in developing countries. African Evaluation Journal, 6(1).
https://doi.org/10.4102/aej.v6i1.254
Tavenner, K., & Crane, T. A. (2019). Beyond “women and youth”: Applying intersectionality in
agricultural research for development. Outlook on Agriculture.
https://doi.org/10.1177/0030727019884334
The National Assembly. (2010). Law No. 55/2010/QH12. https://vanbanphapluat.co/law-no-55-
2010-qh12-of-june-17-2010-on-food-safety
The World Bank. (2016). Vietnam Food Safety Risks Management Challenges and Opportunities.
Thirsk, L. M., & Clark, A. M. (2017). Using Qualitative Research for Complex Interventions: The
Contributions of Hermeneutics. International Journal of Qualitative Methods.
https://doi.org/10.1177/1609406917721068
Tran, T. K. Van, Elahi, E., Zhang, L., Abid, M., Pham, Q. T., & Tran, T. D. (2018). Gender differences
in formal credit approaches: rural households in Vietnam. Asian-Pacific Economic Literature.
https://doi.org/10.1111/apel.12220
Tuyet Hanh, T. T., Huong, L. T. T., Huong, N. T. L., Linh, T. N. Q., Quyen, N. H., Nhung, N. T. T., Ebi,
K., Cuong, N. D., Van Nhu, H., Kien, T. M., Hales, S., Cuong, D. M., Tho, N. T. T., Toan, L. Q.,
Bich, N. N., & Van Minh, H. (2020). Vietnam Climate Change and Health Vulnerability and
Adaptation Assessment, 2018. Environmental Health Insights.
https://doi.org/10.1177/1178630220924658
Uitto, J.I. (2021). Evaluating Environment in International Development. Routledge.
Uitto, Juha I. (2019). Sustainable Development Evaluation: Understanding the Nexus of Natural
and Human Systems. New Directions for Evaluation, 2019(162), 49–67.
198
https://doi.org/10.1002/ev.20364
Vipham, J. L., Amenu, K., Alonso, S., Ndahetuye, J. B., Zereyesus, Y., Nishimwe, K., Bowers, E.,
Maier, D., Sah, K., Havelaar, A., & Grace, D. (2020). No food security without food safety:
Lessons from livestock related research. In Global Food Security.
https://doi.org/10.1016/j.gfs.2020.100382
WHO. (2021). Draft WHO global strategy for food safety 2022-2030.
World Bank Group. (2019). Persistent Malnutrition in Ethnic Minority Communities of Vietnam :
Issues and Options for Policy and Interventions.
Zinsstag, J., Schelling, E., Waltner-Toews, D., & Tanner, M. (2011). From “one medicine” to “one
health” and systemic approaches to health and well-being. Preventive Veterinary Medicine.
https://doi.org/10.1016/j.prevetmed.2010.07.003
199
CHAPTER 6: HOW THE WORLD WILL MEASURE PROGRESS ON THE PARIS CLIMATE AGREEMENT AND KEEP COUNTRIES ACCOUNTABLE
Climate change is a global problem that knows no national borders
[https://www.un.org/sustainabledevelopment/blog/2015/11/climate-change-knows-no-
national-borders-un-chief-says]. All countries will have to work together and take bold actions
to reduce greenhouse gas emissions and demonstrate that their emissions are declining if we
are to meet the Paris Agreement goal of keeping the global temperature rise to no more than
1.5 C above pre-industrial levels [https://unfccc.int/process-and-meetings/the-paris-
agreement/the-paris-agreement].
To achieve any goal, targets must be set and progress measured. When it comes to climate
change, that assessment is called the “global stocktake”
[https://unfccc.int/topics/science/workstreams/global-stocktake-referred-to-in-article-14-of-
the-paris-agreement]. The stocktake, which will occur every five years beginning in 2023, takes a
look at the collective progress the world’s nations have made on climate action.
Several reports [https://unfccc.int/news/climate-commitments-not-on-track-to-meet-paris-
agreement-goals-as-ndc-synthesis-report-is-published] have found that countries’ climate
pledges (called nationally determined contributions, or NDCs) [https://unfccc.int/process-and-
meetings/the-paris-agreement/nationally-determined-contributions-ndcs/nationally-
determined-contributions-ndcs/ndc-synthesis-report] aren’t ambitious enough to meet the
200
goals of the Paris Agreement. Even if current pledges are achieved, there are few ways to further
reduce emissions after 2030 quickly enough to limit global warming to 1.5 C. The expert
reviewers for the reports published by the Intergovernmental Panel on Climate
Change [https://www.ipcc.ch/reports] say strong, rapid and sustained emissions reductions
would limit global warming and prevent the worst climate impacts.
When country representatives gather at the United Nations climate conference (COP26) in
Glasgow, Scotland, in November 2021, they will finalize the plans for the global stocktake —
how the parties to the agreement will measure and report their climate actions — so they can
then strengthen their own country’s climate pledges.
6.1 Where are we?
Article 14 [https://unfccc.int/topics/science/workstreams/global-stocktake-referred-to-in-article-
14-of-the-paris-agreement] of the Paris Agreement offers some information about what the
global stocktake might look like, but falls short in naming specific accountability and reporting
mechanisms.
That’s a bit like working together on a community garden to grow healthy food without those
involved agreeing upon some clear ground rules, such as how plots are assigned, which crops
are appropriate and how to make the garden thrive.
201
Five-year submission cycle for Nationally Determined Contributions, and how the global stocktake influences
them. (Authors, adapted from UNFCCC 2016 [https://unfccc.int/process-and-meetings/the-paris-agreement/nationally-
determined-contributions-ndcs/nationally-determined-contributions-ndcs])
The 191 parties (190 countries plus the European Union) aren’t without some guidance,
however. At the COP24 meeting in Katowice, Poland, in 2019, they agreed on a three-phase
process [https://www.wri.org/insights/how-will-paris-agreements-global-stocktake-work] for the
stocktake.
1. Information collection and preparation.
2. Technical assessment.
3. Communicating and acting on findings.
This process does not specify what data are needed and in what format. The absence of a
harmonized approach to data collection, and the fragmented evidence that comes with that, will
constrain our ability to measure progress.
6.2 How are we doing?
Some countries [https://www.climate-chance.org/en/comprehend/global-synthesis-report-on-
local-climate-action], such as Germany and Canada, are already producing progress reports on
climate action. But it is unlikely all these reports will be in a format that allows them to be
compared, which is important for aggregating findings and answering the question: How are we
collectively doing?
202
Establishing reporting mechanisms is nothing new. Guidelines
exist [https://www.undp.org/content/dam/LECB/docs/pubs-reports/undp-wri-ndcsp-ndc-
enhancement-report-2019] to help governments structure their nationally determined
contributions, which feed into the NDC Synthesis [https://unfccc.int/process-and-meetings/the-
paris-agreement/nationally-determined-contributions-ndcs/nationally-determined-
contributions-ndcs/ndc-synthesis-report], a report that provides an overview of national climate
ambitions. Similar guidelines for reporting on national climate actions are essential to support
the global stocktake.
For efforts to reduce emissions, also called mitigation, one way of determining success is by
measuring the size of the emission reductions. But it’s unclear if a comparable method exists to
measure efforts to respond to and manage actual or expected climate and its effects, called
adaptation.
The global goal on adaptation [https://unfccc.int/news/the-adaptation-committee-considers-
methods-to-review-the-global-goal-on-adaptation], such as enhancing the capacity to adapt,
strengthening resilience and reducing vulnerability to climate change, is vague, leading to
challenges in measuring its success [ https://unfccc.int/documents/268841].
203
Green roofs, like this one in the Netherlands, can reduce heat transfer through the building roof, improving comfort
inside and reducing heat stress from heat waves. (Shutterstock)
When governments make decisions about addressing climate change, they often overlook how
they will measure climate progress. In his 2021 executive
order [https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/executive-
order-on-tackling-the-climate-crisis-at-home-and-abroad] on tackling the climate crisis, U.S.
President Joe Biden noted that agencies will develop “climate action plans and data and
information products to improve adaptation” without elaborating on how those plans will be
evaluated.
204
The limited attention to climate action measurement risks poorly understanding progress on
climate action and designing subsequent climate targets that miss the mark.
6.3 Collecting and preparing information
UN agencies often conduct evaluations to measure outcomes of their programs. We
recently reviewed UN evaluation reports published from 2014 to
2019 [https://doi.org/10.1016/j.gfs.2021.100509] to assess how climate change was accounted
for in programs intended to bolster food security — the availability of food and access to it.
Climate change — including increasing temperatures, changing rainfall patterns and a greater
risk of extreme weather — has strong negative effects on crop yields and livestock health.
Our review provided insights into how food security programs are managed and how the results
are measured in the context of climate change. We identified three things countries can do now
to prepare for the stocktake.
Build strong monitoring and evaluation systems for climate action. Credible data systems need
to be put in place, developed and sustained. Monitoring and evaluation specialists require
training. Training must also extend to decision-makers who use the information.
Measure climate action. Although a program may not have intentionally addressed climate
change, some activities may have contributed to unexpected outcomes. One study of a rural
development project in
Morocco [https://www.ifad.org/documents/38714182/39736369/pdrzmh_ppa_e.pdf/b70c0bb6-
205
dc4a-47e8-8ba2-d5249daa8da9?version=2.0] noted that while nothing was specifically done to
improve the climate resilience of farmers, efforts to protect water, soil and vegetation cover did
have that effect.
Concern for climate change may not be enough to catalyze support to implement more
ambitious climate mitigation and adaptation efforts. Documenting additional benefits for health,
ecosystems and economy, also called co-benefits [https://www.ipcc.ch/report/ar5/wg2/human-
health-impacts-adaptation-and-co-benefits], can increase interest in taking climate action.
Integrate climate measures into programs that are at risk of the impacts of climate change, such
as planning for climate impacts or incorporating climate resilience training. Unpacking the
pathway by which climate change influences programs can help identify entry points for
mitigating those climate impacts. For example, a study in El Salvador that focused on the
management of natural resources, including vegetation cover and soil quality
[http://www.fao.org/3/ca5009en/CA5009EN], spelled out how farmers could improve their
management practices and how adaptation might support this goal.
206
CHAPTER 7: CONCLUSION
7.1 Reflecting on the journey
My professional experiences in evaluation have largely centered on food security contexts,
contributing to my interest in research at the intersection of food security and evaluation.
Initially, my dissertation focused on co-developing and co-evaluating a community-based
monitoring system for Indigenous food security in Uganda in collaboration with the Indigenous
Health Adaptation to Climate Change (IHACC) team and communities. I was introduced to the
IHACC project through my advisors (Drs. Sherilee Harper and Warren Dodd) who are principal
investigators and collaborators of the program. The first research chapter (Chapter 2) reviewed
community-based food security monitoring systems globally, contributing to research objective
1 (characterize how food security is monitored under a changing climate). Learnings from this
chapter were intended to provide a scaffolding foundation for co-developing and co-evaluating
community-based monitoring systems in Chapter 3 and subsequent chapters. However, when it
came time for community collaboration in January 2019, there were Ebola cases reported in the
Democratic Republic of the Congo in areas near the Ugandan border (and near the IHACC
project location). The travel restrictions, uncertainties, and risks presented by Ebola encouraged
me to explore existing food security program evaluation efforts rather than support the
development of a new food security program. The review of food security monitoring programs
presented in Chapter 2 found there was a limited number of evaluations considered climate
change indicators. This review established a basis for exploring the ways in which evaluations
207
can better measure and encourage climate action outcomes. As such, my dissertation aimed to
enhance evaluations primarily by learning from past evaluations of food security programs
rather than developing and evaluating a food security program.
For Chapter 3, I developed and applied a tool assessing climate change integration into food
security program evaluation, contributing to research objective 2 (assess how food security
evaluations are considering climate change). Because United Nations agencies often measure the
outcomes of their development programs, I drew on United Nations food security evaluation
reports as a case study. A key finding was the supportive role of program theories in thinking
through how food security programs – and measures used to assess them – can respond to
climate change. While program theories show promise in supporting food security programs
operating under climate change, it was unclear how and under what circumstances theory-
driven evaluation should be used.
To address this knowledge gap, Chapter 4 reviewed common theory-driven approaches to
evaluation, contributing to research objective 3 (characterize the use of theory-driven evaluation
in food security contexts). This study found that theory-driven evaluations respond to different
evaluation challenges; for example, Theories of Change were often developed to address the
need to evaluate programs operating under complexity while Realist Evaluation was generally
used to understand outcomes of programs. Additionally, Chapter 4 highlighted how reflecting
on context can further shape the design of the evaluation.
208
The important role of the context in shaping the evaluation raised the question: How are food
security program evaluations responding to context? Chapter 5 presented a qualitative study
exploring the perspectives and experiences of researchers and community members engaged in
a food security program in Vietnam (SafePORK), contributing to objective 4 (capture lessons
learned from designing and evaluating complex food security programs). This chapter explored
how climate change and other contextual factors influence the implementation and evaluation
of SafePORK. My existing connection to SafePORK – as a contracted evaluation consultant for
the program during my PhD studies – was critical in getting support for this research. Because
the program is ongoing, the study findings are timely for informing ongoing programming. A
key finding was that asking about climate change – in a program where climate action was not
an explicit/implicit objective – yielded important entry points for future climate responses.
The practice-based chapter in this dissertation, Chapter 6, was written for a public audience (The
Conversation Canada) to broadly disseminate research findings from this dissertation. A key
message amplified in this piece was the importance of measuring climate action outcomes.
Importantly, it provided insights into how we can support this measurement, drawing on
findings from Chapter 3. This news article was timely considering the rapidly approaching global
2023 climate change stocktake – a mandatory step of the Paris Agreement assessing collective
progress toward long-term climate goals – and the need for countries to prepare information
for the stocktake. Additionally, the piece expanded on how developing a program theory can
help identify entry points for mitigating climate impacts, thus bridging Chapters 3 and 4.
209
7.2 Synthesis of findings
Exploring the intersection of food security, climate change, and evaluation, findings across all
four research chapters (2-5) point to the importance of 1) exploring program theory in
evaluations of complex food security programs; 2) considering gender and other social
identities; and, 3) considering climate change (Figure 7.1).
Figure 7.1. Visual representation of key findings from this dissertation.
210
7.2.1 Exploring program theory in food security program evaluation
Underscoring much of this dissertation is the role of program theory in developing and
evaluating complex food security programs. As suggested in Chapter 4, theory-driven evaluation
places a focus on the assumptions underlying the linkages between food security outcomes and
activities. In identifying the causal relationships thought to be of greatest importance to the
program’s success, program theories help to prioritize data collection and analysis. This step is
particularly helpful for programs with multiple components and thus multiple potential
pathways toward food security outcomes (Hendriks, 2020).
Program theories should also contain information about contextual factors expected to
influence causal relationships. This information provides a structure for identifying and
addressing potential challenges for implementation. For example, in Chapter 3, thinking about
the climate context can help conceptualize pathways toward climate-responsive food security
outcomes (e.g. UNEP, 2016; Howland, 2019; Béné et al., 2020). Finding ways of tracking
intermediate outcomes toward coupled food security and climate action goals – along with the
final outcomes – is of timely importance in the current context of the climate crisis (Mbow et al.,
2019).
In exploring the factors influencing the successful implementation of a multi-component food
safety program in Vietnam, Chapter 5 also considered program theory, alongside other
evaluation tools, as a key success factor. This chapter provided an empirical example of how
program theories are being used in practice, contributing to the evidence base of theory-driven
211
evaluations in food security contexts (Chapter 4). Importantly, program theories are not meant
to be static. In Apgar et al., (2017), the initial broad program theory for an aquatic agricultural
systems program was revised after a planning workshop, leading to the integration of gender
equity, nutrition, and climate change into the revised program theory. Program theories should
be regularly revisited following new information obtained from implementation and/or changes
in context (Lam, 2020; Mayne, 2015).
7.2.2 Considering gender and other social identities in evaluation processes
Global calls have long been made for action and accountability on gender equity, notably the
1995 Fourth World Conference on Women, the Millennium Development Goals, and the 2030
Agenda for Sustainable Development (Heidari et al., 2018). Many international development
institutions today acknowledge the importance of gender by implementing gender
mainstreaming policies. For example, since the release of evaluation guidelines by the United
Nations Evaluation Group in 2011, entities across the United Nations system have adopted
gender integration into all evaluations (UNEG, 2011). Evaluators have acknowledged “there is
tremendous but largely untapped potential for evaluation to deepen understanding around
gender inequities” (Hay, 2012, p. 337). And scholars are increasingly encouraging gender and
sex considerations in research and practice (Heidari et al., 2018; McLean & Sen, 2019;
Tannenbaum et al., 2016).
This dissertation responds to these calls to integrate gender and other social identities into their
work. Food insecurity and climate change impacts are most significant for underserved
212
populations, particularly women, girls, and Indigenous Peoples, who do not always have access
to the resources they need to protect themselves and recover from these shocks (Preet et al.,
2010; Ford, 2012; Saxena et al., 2016; van Daalen et al., 2020). Chapters 2 and 4 integrated
gender in research on evaluation by assessing how and to what extent gender was considered in
studies. Chapter 2 found gender was often considered by providing gender-disaggregated data
on who participated in the program and what their experiences were like. This finding is
consistent with other studies exploring gender considerations in evaluation (Fletcher, 2015; Lam
et al., 2019). In Chapter 4, theory-driven evaluations considered gender primarily by outlining
outcome pathways for women as the main program beneficiaries. These studies also identified
gaps in gender considerations. For example, in Chapter 2, only one-third of the reviewed articles
provided gender-disaggregated data, resulting in an incomplete understanding of how women,
men, and gender-diverse people may differentially participate in, and experience programs.
While this review explored how different social groups (e.g. population, age, gender) were
reported on, only gender implications were explicitly mentioned in studies. Continuing to
question “for whom” program theories work might help ensure pathways toward food security
are socially inclusive (Lam et al., 2019).
7.2.3 Integrating climate change into all stages of an evaluation
This dissertation offered strategies for considering complexities arising from climate change and
other contextual factors. In the program description of evaluations, Chapter 3 suggested
providing an overview of potential climate risks, which would facilitate the consideration of
213
climate factors throughout evaluation processes. Chapter 4 identified ways in which theory-
driven approaches to evaluation considered climate change within food security situations; for
example, Realist Evaluation studies tracked contextual influences including climate change on
program implementation and outcomes through context-mechanism-outcome configurations
(e.g. Maye et al., 2019). Chapter 5 described how the climate context alongside program
outcomes and strategies on a real-time basis through an evaluation tool called Outcome
Mapping (Earl et al., 2001). These strategies are consistent with those offered in evaluation
studies in public health settings (Craig et al., 2018; Skivington et al., 2021).
This dissertation joins calls by other evaluators to consider climate change in evaluations of food
security and other development programs (Rowe, 2019; Ssekamatte, 2018; Uitto 2019). Research
findings from this dissertation highlighted opportunities for enhanced inclusion of climate
change considerations in evaluation. In Chapter 3, evaluation studies scoring high in terms of
climate change integration included a climate risk assessment, an explicit strategy for measuring
climate action outcomes, and dedicated reporting of climate action outcomes. In Chapter 4,
evaluation studies applying theory-driven approaches to evaluation accounted for climate
change considerations by examining how climate change might influence program outcome
pathways (e.g. Béné et al., 2020) or how program operations under the context of climate
variability (e.g. Howland, 2019). In Chapter 5, asking researchers and community members
involved in a food safety program where climate change was not an explicit consideration
214
identified climate change implications, highlighting the importance of examining climate change
in programs where climate change seems far removed from the program.
The practice of integrating climate change into evaluation builds on efforts to mainstream
environmental and climatic considerations into development programming (UNDP, 2015;
Vincent & Colenbrander, 2018; World Resources Institute, 2018). For example, the Government
of Canada developed the Environmental Integration Process to guide mainstreaming of
environment and climate change considerations in all sectors of Canadian international
assistance programming (GAC, 2017). This mainstreaming process involves assessing all policy,
program, and project initiatives and integrating appropriate environmental and climatic
measures into their design, implementation, and monitoring. Mainstreaming climate change
into evaluations of food security programs might encourage programs to consider measures to
respond to climate change.
7.3 Strengths and limitations
There are several strengths and limitations of the dissertation approach worth noting. First, the
ability to discern how evaluation approaches (e.g. community-based monitoring, climate change
integration into evaluation, theory-driven evaluation) have shifted over time in the review-based
chapters was dependent on what was reported in the reviewed studies. Given word and space
limits, articles and reports might have prioritized documenting evaluation outcomes rather than
processes. For example, in several studies, evaluation experiences were presented across
215
multiple articles (e.g. Levay et al., 2018; Levay et al., 2019). Related to the above limitation is that
review-based chapters often do not provide adequate insights into context; accounting for
variations in implementation context is critical as some differences in context can determine
whether outcomes are achieved (Hammersley, 2020). Nevertheless, incorporating qualitative
analysis within review-based chapters allowed for in-depth insights into how and why
evaluations of food security programs were conducted. Furthermore, these chapters synthesized
bodies of evaluation knowledge that are often scattered across disciplines and countries.
A key strength of this work was collaborating with users of the research. The review of
community-based monitoring systems presented in Chapter 2 addressed an important research
priority of the IHACC team. In Chapter 5, the qualitative research uncovered insights that are
guiding ongoing implementation efforts for the SafePORK team. Furthermore, Chapter 5 adds
community members’ voices to the literature on how the program is impacting them and how
climate change is influencing their food safety practices. Additionally, this chapter integrated the
rich experiences of SafePORK coordinators who joined bi-weekly meetings to discuss the
research process and findings. Debrief sessions were also held with research assistants to reflect
on the research process, adjust the interview guide, and co-interpret the data. Importantly, the
existing strong relationships between research assistants and participants was key in facilitating
data collection remotely during a pandemic.
216
Another strength of the research presented in Chapter 5 was how this work was informed by my
previous experience of living in Vietnam for several years before starting my dissertation. I
became familiar with the local food safety situation which allowed me to probe this context.
Additionally, I had already worked alongside SafePORK researchers and community members in
Vietnam. This access to research participants is likely not available to individuals without pre-
existing relationships. My status as an insider-outsider also provided faster access to
participants. While going through informed consent, many members of the SafePORK team
stated they were willing to talk with me openly as they were interested in learning from the
results of the study.
7.4 Suggestions for future research
This dissertation highlighted strategies reported in research and evaluation studies for
measuring climate action outcomes within food security programs. An important area for future
research is developing guidance for integrating climate action into evaluation. For example,
should climate change be assessed in some or all food security programs? Are there different
levels of climate assessment, depending on the extent that programs impact (and are impacted
by) climate change? How should climate action outcomes be reported? We encourage
researchers to grapple with these questions and reflect on strategies used to measure climate
action outcomes. Importantly, findings from this dissertation drew primarily on studies in the
published and grey literature; engaging with evaluators and evaluation partners involved in this
work could yield additional insights not captured from the literature.
217
There are many indicators, criteria, and frameworks for measuring climate action outcomes in
development settings (Bours et al., 2014; Fisher et al., 2015; Wojewska et al., 2021). Future
research is needed to develop measurement tools that apply to agriculture and food security
settings, where climate change impacts are especially visible. Measurement tools attuned to the
needs, priorities, and operational constraints of organizations on the ground are important for
developing practical food security program evaluations. And because climate change is a global
problem, future research should consider how findings from individual evaluation studies and
settings can be aggregated to promote learning of and accountability toward collective climate
action.
7.5 Concluding thoughts
As I write the conclusion of my dissertation near the end of 2021, several high-level events took
place. The Secretary-General of the United Nations convened a Food Systems Summit as a way
of setting the stage for global food systems transformation toward achieving the 2030
Sustainable Development Goals (UN, 2021). Expert reviewers for the sixth assessment report
published by the Intergovernmental Panel on Climate Change stated strong, rapid, and
sustained emissions reductions are needed to limit global warming and prevent the worst
climate impacts (IPCC, 2021). The United Nations climate change conference (COP26) resulted in
an outcome document, known as the Glasgow Climate Pact, which calls on 197 countries to
report their progress toward more ambitious climate commitments (United Nations, 2021).
Because food systems impact (and are impacted by) climate change, food systems reform is
218
critical for addressing the climate emergency. In response to the growing impacts of climate
change on food security globally, public investments made in climate action, and the need to
demonstrate progress toward climate change goals, interest in evaluating climate action within
food security and other development contexts is growing.
Against this backdrop, I offer two key messages to take away from this dissertation. The first is
for evaluators to ask about climate change. Opportunities exist for enhanced inclusion of climate
considerations in evaluations of food security and other development programs, provided the
design is responsive to the needs, priorities, and operational constraints of evaluation. A strong
entry point into initially thinking about climate change is the risk management/context section
of program and evaluation planning. Asking how food security programs affect (and are affected
by) climate risks and whether risks differ by gender and other social identities can help identify
opportunities for addressing possible climate vulnerabilities. These considerations could
improve the climate resilience programs and avoid investments that unintentionally lead to
climate maladaptation.
Secondly, there is a need for further research on evaluation about complex food security
programs to keep up to date on shifts in evaluation priorities and tools. Although widely used,
conventional assessments of program performance against desired outcomes are often not
effective for supporting complex programs where program activities and intended outcomes
shift during implementation. The use of alternative approaches to evaluation – mixed-methods,
219
qualitative, context-sensitive, theory-based – shows promise in supporting the learning and
accountability needs of complex programs. Funders and evaluation commissioners should
encourage bold methods where experiences are still quite limited. And evaluators should keep
up to date with shifts in evaluation priorities and continue investigating evaluation processes
and products to inform food security evaluation theory and practice.
This dissertation makes substantial contributions to the global literature by providing syntheses
of evaluations that measure climate action outcomes within food security programs. From these
syntheses, promising alternative approaches to evaluation supporting climate action outcome
measurements are highlighted. Additionally, this dissertation provides a practical case study
illustrating how asking about climate change (for a food security program where climate change
impacts are not immediately obvious) can yield important insights for future climate action.
Funders, evaluation commissioners, evaluators, and practitioners working in food security
contexts should encourage climate considerations in evaluation, and in doing so, contribute to
climate-responsive food security programming.
220
7.6 References
Apgar, J. M., Allen, W., Albert, J., Douthwaite, B., Paz Ybarnegaray, R., & Lunda, J. (2017). Getting
beneath the surface in program planning, monitoring and evaluation: Learning from use of
participatory action research and theory of change in the CGIAR Research Program on
Aquatic Agricultural Systems. Action Research, 15(1), 15–34.
https://doi.org/10.1177/1476750316673879
Béné, C., Riba, A., & Wilson, D. (2020). Impacts of resilience interventions – Evidence from a
quasi-experimental assessment in Niger. International Journal of Disaster Risk Reduction.
https://doi.org/10.1016/j.ijdrr.2019.101390
Bours, D., McGinn, C., & Pringle, P. (2014). Monitoring & evaluation for climate change
adaptation and resilience : A synthesis of tools , frameworks and approaches. SEA Change
CoP, Phnom Penh and UKCIP, May, 1–67.
Craig, P., Di Ruggiero, E., Frohlich, K. L., Mykhalovskiy, E., & White, M. (2018). Taking account of
context in population health intervention research: guidance for producers, users and funders
of research.
Earl, Sarah, Carden, Fred, Smutylo, T. (2001). Outcome mapping. In Evaluation (Issue Idrc 1999).
https://doi.org/10.1007/978-981-10-0983-9_6
Fisher, S., Dinshaw, A., Mcgray, H., Rai, N., & Schaar, J. (2015). Evaluating Climate Change
Adaptation: Learning From Methods in International Development. New Directions for
Evaluation, 2015(147), 13–35. https://doi.org/10.1002/ev.20128
Fletcher. (2015). Addressing gender in impact evaluation: a methods lab publication.
Ford, J. D. (2012). Indigenous health and climate change. American Journal of Public Health,
102(7), 1260–1266. https://doi.org/10.2105/AJPH.2012.300752
GAC. (2017). Environmental Integration Process – Development Programming.
Hammersley, M. (2020). Reflections on the Methodological Approach of Systematic Reviews. In
Systematic Reviews in Educational Research. https://doi.org/10.1007/978-3-658-27602-7_2
Hay, K. (2012). Engendering Policies and Programmes through Feminist Evaluation:
Opportunities and Insights. Indian Journal of Gender Studies, 19(2), 321–340.
https://doi.org/10.1177/097152151201900208
Heidari, S., Babor, T. F., De Castro, P., Tort, S., & Curno, M. (2018). Sex and gender equity in
221
research: rationale for the SAGER guidelines and recommended use. Gaceta Sanitaria.
https://doi.org/10.1016/j.gaceta.2018.04.003
Hendriks, S. (2020). Food security policy, evaluation and impact assessment. Earthscan.
Howland, F. (2019). Individual capacity baseline report.
IPCC. (2021). AR6 Synthesis Report: Climate Change 2022. https://www.ipcc.ch/assessment-
report/ar6/
Keleman Saxena, A., Cadima Fuentes, X., Gonzales Herbas, R., & Humphries, D. L. (2016).
Indigenous Food Systems and Climate Change: Impacts of Climatic Shifts on the Production
and Processing of Native and Traditional Crops in the Bolivian Andes. Frontiers in Public
Health, 4. https://doi.org/10.3389/fpubh.2016.00020
Lam, S., Dodd, W., Whynot, J., & Skinner, K. (2019). How is gender being addressed in the
international development evaluation literature? A meta-evaluation. Research Evaluation,
28(2), 158–168. https://doi.org/10.1093/reseval/rvy042
Lam, S. (2020). Toward Learning from Change Pathways: Reviewing Theory of Change and Its
Discontents. Canadian Journal of Program Evaluation, 35(2), 188-203.
https://doi.org/10.3138/cjpe.69535
Maye, D., Enticott, G., & Naylor, R. (2019). Theories of Change in Rural Policy Evaluation.
Sociologia Ruralis. https://doi.org/10.1111/soru.12269
Mayne, J. (2015). Useful theory of change models. Canadian Journal of Program Evaluation,
30(2), 119–142. https://doi.org/10.3138/cjpe.230
Mbow, C., Rosenzweig, C., Barioni, L. G., Benton, T. G., Herrero, M., Krishnapillai, M., Liwenga, E.,
Pradhan, P., Rivera-Ferre, M. G., Sapkota, T., Tubiello, F. N., & Xu, Y. (2019). Food Security. In
P. Z. P.R. Shukla, J. Skea, E. Calvo Buendia, V. Masson-Delmotte, H.-O. Pörtner, D.C. Roberts,
J. P. P. R. Slade, S. Connors, R. van Diemen, M. Ferrat, E. Haughey, S. Luz, S. Neogi, M.
Pathak, J. Petzold, & J. M. P. Vyas, E. Huntley, K. Kissick, M. Belkacemi (Eds.), Climate Change
and Land: an IPCC special report on climate change, desertification, land degradation,
sustainable land management, food security, and greenhouse gas fluxes in terrestrial
ecosystems.
McLean, R. K. D., & Sen, K. (2019). Making a difference in the real world? A meta-analysis of the
quality of use-oriented research using the Research Quality Plus approach. Research
Evaluation, 28(2), 123–135. https://doi.org/10.1093/reseval/rvy026
222
Preet, R., Nilsson, M., Schumann, B., & Evengård, B. (2010). The gender perspective in climate
change and global health. Global Health Action. https://doi.org/10.3402/gha.v3i0.5720
Rowe, A. (2019). Sustainability-Ready Evaluation: A Call to Action. New Directions for Evaluation,
2019(162), 29–48. https://doi.org/10.1002/ev.20365
Skivington, K., Matthews, L., Simpson, S. A., Craig, P., Baird, J., Blazeby, J. M., Boyd, K. A., Craig, N.,
French, D. P., McIntosh, E., Petticrew, M., Rycroft-Malone, J., White, M., & Moore, L. (2021). A
new framework for developing and evaluating complex interventions: Update of Medical
Research Council guidance. The BMJ. https://doi.org/10.1136/bmj.n2061
Ssekamatte, D. (2018). The role of monitoring and evaluation in climate change mitigation and
adaptation interventions in developing countries. African Evaluation Journal, 6(1).
https://doi.org/10.4102/aej.v6i1.254
Tannenbaum, C., Greaves, L., & Graham, I. D. (2016). Why sex and gender matter in
implementation research. BMC Medical Research Methodology, 16(1).
https://doi.org/10.1186/s12874-016-0247-7
Uitto, J. I. (2019). Sustainable Development Evaluation: Understanding the Nexus of Natural and
Human Systems. New Directions for Evaluation, 2019(162), 49–67.
https://doi.org/10.1002/ev.20364
UN. (2021). The food systems summit. https://www.un.org/food-systems-summit
UNDP. (2015). Mainstreaming Environment and Climate for Poverty Reduction and Sustainable
Development.
UNEP. (2016). Terminal Evaluation of the UNDA 7th Tranche funded UNEP project “Capacity
Building in National Planning for Food Security.”
United Nations. (2021). COP26 closes with ‘compromise’ deal on climate, but it’s not enough, says
UN chief. https://news.un.org/en/story/2021/11/1105792
United Nations Evaluation Group. (2011). Integrating Human Rights and Gender Equality in
Evaluation - Towards UNEG Guidance.
http://www.unevaluation.org/documentdownload?doc_id=980&file_id=1294
van Daalen, K., Jung, L., Dhatt, R., & Phelan, A. L. (2020). Climate change and gender-based
health disparities. In The Lancet Planetary Health. https://doi.org/10.1016/S2542-
5196(20)30001-2
223
Vincent, K., & Colenbrander, W. (2018). Developing and applying a five step process for
mainstreaming climate change into local development plans: A case study from Zambia.
Climate Risk Management. https://doi.org/10.1016/j.crm.2018.04.005
Wojewska, A. N., Singh, C., & Hansen, C. P. (2021). A policy tool for monitoring and evaluation of
participation in adaptation projects. Climate Risk Management.
https://doi.org/10.1016/j.crm.2021.100326
World Resources Institute. (2018). FROM PLANNING TO ACTION: MAINSTREAMING CLIMATE
CHANGE ADAPTATION INTO DEVELOPMENT.
224
Appendices
Appendix 1: Search strategy to identify published articles on Indigenous community-based
monitoring of Indigenous food security and climate change in AGRICOLA©, PRISMA©,
MEDLINE®, CabDirect©, and the Web of Science™ CORE Collection databases.
Main terms Expanded terms
Indigenousa Aasax OR Aboriginal OR "Aboriginal-Malay" OR Aborigine OR Achi OR
Achuar OR Adibashi OR Adivasi OR Adivasis OR Afar OR Ainu OR Aka OR
Akatek* OR Akawai OR Akha OR Akie OR Akoula OR Akurio OR Akwoa OR
"al-Kaabneh" OR "al-Asarmeh" OR "al-Ramadin" OR "al-Rshaida" OR Aleut
OR Alutor OR Amazigh OR Ambo OR Ameridian OR Amuesha OR Anak OR
"Andean Kichwe" OR "Andoa" OR "Andorrans" OR Angaite OR Anikhwe
OR Anu OR Arara OR Arawak OR "Arawak-Taino" OR Arwak OR Ashaninka
OR Atayal OR Austronesian OR "Ava Guarani" OR Awajun OR Awa OR
Awakateco OR Aweer OR Ayeoreo OR Aymara OR Ayoreo OR Aztec OR
Baaka OR Baantonu OR Babi OR Bahnar OR Babongo OR Bacwa OR
Bagame OR Bagombe OR Bagyeli OR BaGyeli OR Bajuni OR Baka OR
Bakgalagadi OR Bakola OR Bakongo OR Bakoya OR Balala OR Bambara
OR Bambuti OR Bantu OR Barabaig OR Bariba OR Barimba OR Basarwa OR
Bassari OR Batwa OR Bawarwa OR BaWka OR Bawn OR BaYeyi OR Bedzam
OR Benet OR Berabis OR Berawan OR Berber OR Berbers OR Bidayuh OR
Bigombe OR Biharis OR Bilma OR Bisayah OR Bobo OR Boeschs OR Bofi
OR Boni OR Bonis OR Boranna OR Boro OR Bororo OR Boruka OR
Botsarwa OR Bozo OR Brao OR Bribri OR "Bri Bri" OR Brunca OR Bugakhwe
OR Bulu OR Bumipeuteras OR Bunak OR Bunun OR Bwiti OR Cabecar OR
Cacaopera OR Campeche OR Carib OR Caribs OR "Ch’orti" OR Chachi OR
Chaima OR Chakma OR Chalchiteco OR Chamorro OR Chamorru OR
Chamorous OR "Chao-Khao" OR "Chao Ley" OR Charrua OR Chelkancy OR
Chiapas OR Chibcha OR Chibchense OR Chipaya OR Chiriguano OR
Chiquito OR Chiquitano OR Chorotega OR Chorti OR Cofan OR Chuaa OR
Chuj OR Chukchi OR Chulymcy OR Chuvancy OR Ciboney OR "Ciboney-
Taino-Arawak" OR "Cocama-Cocomilla" OR Colla OR Copts OR Cotier OR
Cree OR Cumanagoto OR Dabou OR Dagiri OR Dahalo OR Danisi OR
Daroobe OR Datoga OR Daza OR Degar OR Deti OR Diaguita OR Dinka
OR Dioula OR Ditammari OR Dogon OR Dolgan OR Doma OR Dukha OR
Dusun OR Ebrie OR "egga hodaabe" OR Elmolo OR "El Mono" OR Embera
OR Emerillon OR Ency OR Endorois OR "Enlhet Norte" OR Enxet OR "Enxet
Sur" OR Epera OR Eskimo OR "Ese Eja" OR Evenk OR Ewondo OR Fatukuku
OR "First Nation*" OR "Forest dwell*" OR Fuegian OR Fulani OR Fulbe OR
225
Galibia OR Galibi OR Garifuna OR Gaoshan OR Gio OR Guadalcanese OR
Guana OR Guaicuru OR Guarani OR "Guarani Mbya" OR Guyami OR
Guaymi OR Guerrero OR Gurani OR Guransi OR Gurung OR "G//ana" OR
"G/wi" OR "Gwich’in" OR Hadzabe OR Hadza OR Haida OR Herero OR
Hidalgo OR "Hill People" OR "Hill Person" OR Hmong OR Hoa OR
Huambisa OR Huastec OR Hui OR Huetar OR Hutu OR Iban OR Igotot OR
Ik OR Imazighn OR Imazighen OR Indigenous OR Ingarico OR Inuit OR
Inupiat OR Inuvialut OR Iroquoian OR Itelmen OR Itza OR Ixil OR Jacalteco
OR "Jahalin Bedouin" OR Jarai OR Jivi OR Jumma OR "Ju’hoansi" OR
"K’iche" OR "Ka Pei Aina" OR Kachin OR Kaiowa OR Kalanga OR Kalina OR
"Kalina-go" OR Kalinago OR "Kalinago-Taino" OR "Kali’na" OR Kamchadal
OR Kanak OR "Kanaka Maoli" OR Kanuri OR Kaqchikel OR Karamajong OR
Karenni OR Kavalan OR Kayapo OR Kawashkar OR Kayan OR Kazakh OR
Kedayan OR Kelait OR Kenyah OR Kereki OR Kety OR "Khali’nago" OR
Khamu OR Khanty OR Khengs OR "Khmer Krom" OR Khoekhoe OR "Khoe-
San" OR Khoikhoi OR Khoisan OR Khomani OR "Khudro Nrigoshthhi" OR
Khumi OR Khwe OR Khyang OR Kichwas OR Kipsigis OR Kirdi OR Koba OR
Koryak OR Krio OR Krohn OR Kua OR Kumandincy OR Kuna OR Kuy OR
Kwisi OR Lahu OR Lao OR "Laotian Tribes" OR Lenca OR Lickanantay OR
Limbu OR Lisu OR Livs OR Lobi OR Lokono OR Loma OR Lua OR Lumad
OR "Lunda-Chokwe" OR Lushai OR Maasai OR Macourai OR Macuxi OR
Macuzi OR Magar OR Makasae OR Makuxi OR Malagasy OR Malakote OR
Malay OR "Malayo-Polynesian" OR Maleku OR Mangyan OR Mani OR
Mano OR Mansi OR Maori OR Mam OR Manjo OR Marma OR "Marsh
Dwellers" OR Mapuche OR Maskoy OR "masyarakat adat" OR Mataco OR
"Mataco Matguayo" OR Matagulpa OR Maya OR "Maya Chorti" OR
Mayagna OR Mbanderu OR Mbini OR Mbororo OR Mbukushu OR
Mbundu OR Mbuti OR Mbri OR Mbya OR Mdendjele OR Melanesian OR
"Melanesian-Papauan" OR Mestico OR Mestizo OR Merina OR Metis OR
Miao OR Mien OR Mikaya OR Miskito OR Miskitu OR Misquito OR Mixte
OR Mnong OR Mogeno OR "Mon-Khmer" OR Montagnards OR Mopan
OR Moxeno OR Mozabite OR Mpukushu OR Mru OR Muong OR Murut OR
"N/oakhwe" OR "N’guigmi" OR Nagas OR Nahoa OR Nahua OR Nahuatl
OR Nama OR Nambiquara OR Nanaicy OR Nandeva OR "Nandevi Guarani"
OR Naro OR "Naso Tjerdi" OR Native OR Negidalcy OR Negeri OR Negrito
OR Nemadi OR Nenets OR Nganasan OR Ngabe OR Ngobe OR "Ngobe-
Bugle" OR Nivkhy OR Nuer OR "Nyaneka-Nkumbi" OR Oaxaca OR Ocanxiu
OR Ogiek OR Ogoni OR Ojibway OR Okinawans OR "Orang Asli" OR
Orochi OR Oroki OR Otomi OR Ovimibundu OR Oyampi OR "Pai Tavytera"
OR Paiwan OR Palenqueros OR Palikur OR Pankho OR Patamona OR Pech
226
OR Pemon OR Peul OR Peulh OR Penan OR Piaroa OR "Ping Pu" OR Pipil
OR Pocomam OR Pokot OR Poqomam OR "Poqomchi’" OR Puebla OR
Punan OR Puyuma OR "Q’anjob’al" OR "Q’eqchi" OR Qawasqar OR
Qicaque OR Quechua OR Quenchua OR "Quintano Roo" OR Qom OR Rai
OR Raisales OR Rakhine OR Rama OR Rapanui OR “Rapa Nui” OR Raute
OR Rhade OR Roraima OR Rotumans OR Rukai OR Saami OR Sabaot OR
Saharawis OR Saisyat OR Sakapulteco OR Sakizaya OR Sami OR (San AND
Africa) OR Sanapan OR Sanapana OR Sandawe OR "Santa Rosa Carib" OR
Sanya OR Saramancas OR ("Scheduled Tribes" AND India) OR Secoya OR
Sediq OR Selkup OR Semang OR Sengwer OR Senoi OR Shan OR Sherpa
OR Shipibo OR "Shipibo-Conibo" OR Shiwiar OR Shorcy OR Shua OR
Shuar OR Siona OR Sipakapense OR Soioty OR "South Sea Islander*" OR
Stieng OR "Sumu-Mayangna" OR Sutiaba OR Tachangya OR "Tai-Kadai"
OR Taino OR "Taino-Kalingo" OR Tamang OR Tampuan OR Tapeba OR
Tapebo OR Tareno OR Taurepang OR Tawahka OR Tazy OR Teda OR
Teenek OR Teko OR Tektiteko OR Telengity OR Teleuty OR Temenbe OR
Teribe OR Tesker OR Thakali OR Tharu OR Thao OR Tikuna OR Tikigaq OR
Tirio OR Toba OR "Toba Maskoy" OR Tofolar OR Tolai OR Toloupan OR
Tomarao OR Topnaars OR "Torres Straight Islander*" OR Totonac OR
Toubou OR Truku OR Tsexakhwe OR Tripura OR Tsaatan OR Tsachila OR
Tsou OR Tsumkwe OR Tshwa OR Tuareg OR Tuaregare OR Tubolar OR
Tubu OR Tugen OR Tukano OR Tupi OR Tutong OR Tutsi OR "Tuvin-
Todjin" OR Twa OR Tyua OR "Tz’utujil" OR Tzeltal OR Tzotzil OR Uchay OR
Udege OR Ulchi OR "Ureueu-Wau-Wau" OR Uru OR Uspanteko OR Vadda
OR Vadema OR Vai OR Veddhas OR Veps OR Vyadha OR "Waaniy-a-
Laato" OR Waata OR Wadoma OR Wagashi OR Wapaichana OR Waorani
OR Wapixana OR Warao OR Warrau OR "Warrau Wayana" OR Wayampi
OR Wayana OR Wayeyi OR Wayuu OR Wichi OR Wodaabe OR Wounaan
OR Xinka OR Yaaku OR Yami OR Yamana OR Yanomami OR Yukpa OR
Yvytoso OR Zamuco OR Zapara OR Zapotec OR "!Xoo" OR "//’Xauesi" OR
"/Xaisa"
Surveillance surveillance OR tracking OR monitoring OR reporting OR "information
system" OR "early warning" OR "early detection" OR "early notification"
OR "timely warning" OR "timely detection" OR "timely notification"
Food security
and climate
change
food OR agriculture OR nutrition OR livestock OR fish* OR animal OR
plant OR wildlife OR hunt* OR gather* OR environment* OR ecological OR
ecosystem OR "natural resource" OR "resource management" OR "co-
management" OR "cooperative management" OR "joint management" OR
water OR sea OR ocean OR "climate change" OR "climate variability" OR
"climate hazard" OR "extreme weather" OR "natural hazard" OR disaster
227
OR flood OR drought OR hurricane OR storm OR cyclone OR "sea level
rise" OR "irregular rainfall" OR "intense rainfall" OR resilience OR poverty
OR livelihood OR welfare OR income
Community-
based
"community-based" OR "community-centred" OR "community-centered"
OR "community-engaged" OR "community-led" OR "locally-based" OR
participatory OR collaborative OR "citizen-led" OR "citizen-engaged" OR
"citizen-based" OR "citizen science" a Search terms used to identify Indigenous peoples globally were adapted from Bishop-Williams
et al. 2017 (1). We removed several terms that were captured under other umbrella terms (e.g.
"Indigenous Palestinians" was captured by the term Indigenous and so was removed).
228
Appendix 2: Complete list of included articles and select article characteristics on Indigenous
community-based monitoring of food security and climate change.
Author Year Country Study
regiona
Indigenous
community
Food
security
theme
Wong (2) 2017 Canada Nunavut
(Arctic Bay,
Kimmirut,
and Arviat)
Inuit Wildlife
McKay (3) 2017 Canada Northcentral
British
Columbia
Takla Lake
First Nations
Natural
resource
management
Delgado-
Aguilar (4)
2016 Ecuador Sumaco
Biosphere
Reserve,
Napo,
central-
northern
Ecuador
Kichwa
community
Environmenta
l change
Dobbs (5) 2016 Australia Beagle Bay,
Dampier
Peninsula
north of
Broome,
Western
Australia
Nyul Nyul
community
Aquatic
ecosystems
Wiseman (6) 2015 Australia Alinytjara
Wilurara
Natural
Resources
Management
region,
Northwest of
South
Australia
Anangu Natural
resource
management
Harmsworth
(7)
2016 New Zealand Aotearoa Maori Aquatic
ecosystems
229
Cochran (8) 2016 Brazil Tiquie River,
Northwestern
Amazonia
Tukano and
Seana
communities
Climate
Alessa (9) 2016 Multiple
countries
Arctic Multiple
Indigenous
communities
Environmenta
l change
Johnson (10) 2015 Multiple
countries
Arctic Multiple
Indigenous
communities
Environmenta
l change
Torres (11) 2014 Mexico Chihuahua,
Chiapas,
Yucatan
Peninsula,
Oaxaca
Multiple
Indigenous
communities
Reducing
Emissions
from
Deforestation
and Forest
Degradation
(REDD+)
Holmes (12) 2014 Multiple
countries
Not specified Multiple
Indigenous
communities
REDD+
Torres (13) 2014 Multiple
countries
Not specified Multiple
Indigenous
communities
REDD+
Galabuzi (14) 2014 Uganda Mabira Forest
Reserve
Bantu ethnic
groups
Forest
resources
Gunn (15) 2014 Canada Northwest
Territories
Lutsel K’e
Dene First
Nation
Wildlife
Bell (16) 2012 Canada Canada Inuvialuit Multiple
Peace (17) 2012 Canada Northern
Canada
First Nations
and Inuit
Climate
Laidler (18) 2011 Canada Cape Dorset,
Igloolik, and
Pangnirtung,
Nunavut
Inuit Climate
Lauer (19) 2010 Solomon
Islands
Roviana
Lagoon
Pacific Island
Communities
Environmenta
l change
Valdivia (20) 2010 Bolivia Northern and
Central
Altiplano
Andean
communities
Climate
230
Heaslip (21) 2008 Canada British
Columbia
Kwakwaka'wa
kw First
Nations
Aquatic
Ntk (22) 2008 Canada Sanikiluaq,
Hudson Bay
Multiple
Indigenous
communities
(Inuit and
Cree)
Aquatic
Berkes (23) 2007 Canada Hudson and
James Bay
Multiple
Indigenous
communities
(Inuit and
Cree)
Aquatic
Robinson
(24)
2004 Australia Kakadu
National Park
Jawoyn
people
Wildlife
Wiber (25) 2004 Canada Nova Scotia
and Prince
Edward
Island
Mi’Kmaq First
Nation
Fisheries
Danby (26) 2003 Multiple
countries
Saint Elias
Mountains
(Alaska,
Yukon, British
Columbia)
First Nations
in general
Environmenta
l change
Berkes (27) 2001 Canada Arctic Multiple
Indigenous
communities
Aquatic
Kendrick (28) 2013 Canada Arctic Inuit Wildlife
Oba (29) 2012 Multiple
countries
Rangelands Multiple
Indigenous
communities
(Orma, Afar,
Karamojong)
Environmenta
l change
Dabasso (30) 2012 Kenya Central
Division of
Marsabit
Central
District
Borana Environmenta
l change
231
Kouril (31) 2016 Multiple
countries
Arctic region Multiple
Indigenous
communities
Environmenta
l change
Ens (32) 2012 Australia Arnhem Land Multiple
Indigenous
communities
Multiple
Moller (33) 2004 Multiple
countries
Not
specified
Multiple
Indigenous
communities
Wildlife
Armitage (34) 2005 Canada Bassin Island
region,
Nunavut
Inuit Fisheries
Jollands (35) 2007 New Zealand Not specified Maori Natural
resource
management
Paneque-
Galvez (36)
2017 Multiple
countries
Kukama
Kukamiria
Territory
(Peru),
Wapichana
Territory
(Guyana),
Embera-
Wounaan
Territory
(Panama)
Multiple
indigenous
communities
Environmenta
l change
Macherera
(37)
2017 Zimbabwe Gwanda
district
Gwanda
Indigenous
communities
Multiple
Vigilante (38) 2017 Australia Wunambal
Gaambera
Country,
North
Kimberley
Wunambal
Gaambera
Aboriginal
people
Forest
resources
Shaffer (39) 2017 Guyana Masakenari
Village,
Southern
Guyana
Indigenous
Waiwai
Wildlife
Oviedo (40) 2017 Brazil Kaxinawá
Nova Olinda
Indigenous
Kaxinawá
Fisheries
232
Indigenous
Territory,
State of Acre
Mateo-Vega
(41)
2017 Panama Tierras
Colectivas,
Darien
Embera and
Wounaan
people
REDD+
de Paula (42) 2017 Brazil XIL, Brazilian
Cerrado
Xerente
community
Wildlife
Ferguson (43) 2016 United States Colorado
Plateau,
northeastern
Arizona
Hobi tribe Climate
Fragoso (44) 2016 Guyana Rupununi
region
Makushi and
Wapishana
people
Wildlife
Iverson (45) 2016 Canada Nunavik,
Quebec
Inuit Wildlife
Constantino
(46)
2015 Brazil Acre State,
western Brazil
Brazilian
Amazon
Kaxinawa Wildlife
Bellfield (47) 2015 Guyana Annai
District,
North
Rupunini
Makushi
communitites
communities
REDD+
Gill (48) 2014 Canada Fort
McPherson,
Northwest
Territories
Teetł’it
Gwich’in
Environment
change
Lagasse (49) 2014 Canada Great Bear
Rainforest of
British
Columbia
Coastal First
Nations
(Heiltsuk,
Metlakatla,
Wuikinuxw,
Haisla,
Gwa'sala'Nak
waxda'xw,
Nuxalk)
Fisheries
233
Housty (50) 2014 Canada Coastal
British
Columbia
Heiltsuk First
Nation
Multiple
Ford (51) 2013 Canada Iqaluit,
Nunavut
Inuit Multiple
Constantino
(52)
2012 Multiple
countries
Brazilian
Amazonia
and one in
Namibian
Caprivi
conservancie
Multiple
Indigenous
communities
Wildlife
Luzar (53) 2011 Guyana North
Repununi
and South
Rupunini
Makushi and
Wapisana
communities
Wildlife
Wesche (54) 2011 Canada Old Crow,
Yukon
Vuntut
Gwitchin First
Nation
Multiple
Marcoux (55) 2011 Canada Nunavut Inuit Wildlife
Gearheard
(56)
2011 Canada Kangiqtugaa
pik, Nunavut
Inuit Multiple
Ens (57) 2010 Australia Arnhem
Plateau,
Northern
Territory
Aboriginal
Rangers
Multiple
Mahoney (58) 2009 Multiple
countries
Barrow,
Alaska; Clyde
River,
Nunavut,
Canada; and
Qaanaaq,
Greenland
Arctic
communities
Multiple
(including
climate
change)
Setty (59) 2008 India Biligiri
Rangaswamy
Temple
Wildlife
Sanctuary,
South India
Soliga
harvesters
Forest
resources
Constantino
(60)
2008 Brazil Acre Kaxinawa
people
Wildlife
234
Tremblay (61) 2008 Canada Nunavik,
Quebec
Multiple
Indigenous
communities
(Five Inuit
communities
and one
Naskapi
community)
Climate
Ohl-
Schacherer
(62)
2007 Peru Manu
National Park
in Peruvian
Amazonia
Matsigenka c
ommunities
Wildlife
Noss (63) 2004 Bolivia Bolivian
Chaco
Isoseno-
Guarani
Wildlife
Noss (64) 2001 Bolivia Bolivian
Chaco
Isoseno-
Guarani
Wildlife
Torres (65) 2001 Mexico Michoacan in
Central
Mexico
Purepecha
community
Multiple
Brook (66) 2009 Canada Sahtu
Settlement
Area,
Northwest
Territories
Multiple
Indigenous
communities
(De´-line,
Tulita,
Norman
Wells, Fort
Good Hope,
and Colvill)
Multiple
Boissiere (67) 2014 Lao PDR Luang
Prabang
Province
Multiple
ethnic groups
(Hmong,
Khmu and
Tai-Lao)
Forest
resources
Krause (68) 2013 Ecuador Amazon
provinces of
Napo,
Orellana and
Pastaz
Kitchwa Multiple
Larter (69) 2009 Canada Dehcho
Region NWT
Dehco First
Nations
Wildlife
235
Noss (70) 2005 Bolivia Gran Chaco Isoseno-
Guarani
Wildlife
Pacheco-
Cobos (71)
2015 Mexico
La Malinche
National Park
(LMNP) in
Central
Mexico
Multiple
communities
(Nahua,
Otomi and
Mestizo
communities)
Forest
resources
Gratani (72) 2016 Australia Wet Tropics
of northern
Queensland
Malanbarra
and Dulabed
Yidinji people
Environment
Kofinas (73) 2003 MC Arctic regions Multiple
Indigenous
Wildlife
Lyver (74) 2017 New Zealand Ruatahuna,
Te Urewera
region
Tuhoe
Tuawhenua
community
Multiple
Van Vliet (75) 2017 Colombia Ticoya
Indigenous
Reserve of
Puerto
Nariño,
Amazonas,
Colombia
Multiple
Indigenous
groups
(Ticuna,
Cocama and
Yagua ethnic
groups)
Wildlife
Driscoll (76) 2013 United States Alaska Multiple
Indigenous
communities
(8
communities
across three
different
regions)
Multiple
Driscoll (77) 2016 United States Alaska Multiple
Indigenous
communities
(11
communities
across three
regions)
Multiple
Vieira (78) 2015 Brazil Piagacu-
Purus
Indigenous
communities
Wildlife
236
Reserve,
Brazilian
Amazon
(5
communities)
Obura (79) 2001 Kenya Diani-Chale
coral reef
area, South
Mombasa
Digo people Fisheries
Pollock (80) 2005 Canada Northwest
territories
Multiple
Indigenous
communities
(Dene,
Inuvialuit,
Gwich'in
people)
Multiple
Funder (81) 2013 Tanzania Iringa
Region,
southern
Tanzania
Hehe Multiple
Fidel (82) 2014 Multiple
countries
Bering Sea
(Alaska,
Russian
Federation)
Multiple
Indigenous
communities
Wildlife
Herrmann
(83)
2014 Multiple
countries
Quebec
(Canada),
northern
Sweden
Multiple
Indigenous
communities
(Sami, the
Cree, and the
Naskapi)
Wildlife
Parlee (84) 2014 Canada Northwest
Territories
Łutsël K’é
Dene First
Nation
Wildlife
Cummings
(85)
2017 Guyana Pupununi,
Southern
Guyana
Amerindian
communities
Forest
resources
Rijsoort (86) 2005 China Yunnan
province
Han, Dai,
Lisu, and yi
minorities
Multiple
Sheil (87) 2015 Indonesia Mamberamo-
Foja region,
Kay,
Metaweja,
Environmenta
l change
237
Province of
Papua
and Yoke
communities a Note that countries define their geographic boundaries differently and that articles provided
varying levels of detail on the study region. We extracted data on the study region as is
described by the articles.
238
Appendix 3: Complete list of references.
1. Bishop-Williams K, Sargeant JM, Berrang-Ford L, Edge V, IHACC Research Team, Harper
SL. A protocol for a systematic narrative synthesis review: The global impact of
environment on acute respiratory infections in Indigenous populations. BMC Syst Rev
[Internet]. 2017;In preparation. Available from: http://dx.doi.org/10.1186/s13643-016-
0399-x
2. Wong PBY, Dyck MG, Hunters A, Hunters I, Hunters M, Murphy RW. Inuit perspectives of
polar bear research: Lessons for community-based collaborations. Vol. 53, Polar Record.
2017. p. 257–70.
3. McKay AJ, Johnson CJ. Confronting barriers and recognizing opportunities: Developing
effective community-based environmental monitoring programs to meet the needs of
Aboriginal communities. Environ Impact Assess Rev. 2017;64:16–25.
4. Delgado-Aguilar MJ, Konold W, Schmitt CB. Original Articles: Community mapping of
ecosystem services in tropical rainforest of Ecuador. Ecol Indic [Internet]. 2017;73:460–71.
Available from:
http://10.0.3.248/j.ecolind.2016.10.020%0Ahttps://proxy.mendelu.cz/auth/login?url=http:/
/search.ebscohost.com/login.aspx?direct=true&db=edselp&AN=S1470160X16306069&la
ng=cs&site=eds-live
5. Dobbs RJ, Davies CL, Walker ML, Pettit NE, Pusey BJ, Close PG, et al. Collaborative
research partnerships inform monitoring and management of aquatic ecosystems by
Indigenous rangers. Rev Fish Biol Fish. 2016;26(4):711–25.
6. Wiseman ND, Bardsley DK. Monitoring to Learn, Learning to Monitor: A Critical Analysis of
Opportunities for Indigenous Community-Based Monitoring of Environmental Change in
Australian Rangelands. Geogr Res. 2016;54(1):52–71.
7. Harmsworth G, Awatere S, Robb M. Indigenous Māori values and perspectives to inform
freshwater management in aotearoa-New Zealand. Ecol Soc. 2016;21(4).
8. Cochran F V., Brunsell NA, Cabalzar A, van der Veld PJ, Azevedo E, Azevedo RA, et al.
Indigenous ecological calendars define scales for climate change and sustainability
assessments. Sustain Sci. 2016;11(1):69–89.
9. Alessa L, Kliskey A, Gamble J, Fidel M, Beaujean G, Gosz J. The role of Indigenous science
and local knowledge in integrated observing systems: moving toward adaptive capacity
indices and early warning systems. Sustain Sci. 2016;11(1):91–102.
10. Johnson N, Behe C, Danielsen F, Nickels S, Pulsifer PL. Community-Based Monitoring and
Indigenous Knowledge in a Changing Arctic : [Internet]. Available from:
239
http://www.inuitcircumpolar.com/uploads/3/0/5/4/30542564/cbm_final_report.pdf
11. Torres AB. Potential for integrating community-based monitoring into REDD+. Forests.
2014;5(8):1815–33.
12. Holmes I, Potvin C. Avoiding re-inventing the wheel in a people-centered approach to
REDD+. Conserv Biol. 2014;28(5):1380–93.
13. Torres AB, Acuña LAS, Vergara JMC. Integrating CBM into land-use based mitigation
actions implemented by local communities. Forests. 2014;5(12):3295–326.
14. Galabuzi C, Eilu G, Mulugo L, Kakudidi E, Tabuti JRS, Sibelet N. Strategies for empowering
the local people to participate in forest restoration. Agrofor Syst. 2014;88(4):719–34.
15. Gunn A, Russell D, Greig L. Insights into integrating cumulative effects and collaborative
comanagement for migratory tundra caribou herds in the Northwest Territories, Canada.
Ecol Soc. 2014;19(4).
16. Bell RK, Harwood LA. Harvest-based monitoring in the inuvialuit settlement region: Steps
for success. Arctic. 2012;65(4):421–32.
17. McClymont Peace D, Myers E. Community-based participatory process--climate change
and health adaptation program for Northern First Nations and Inuit in Canada. Int J
Circumpolar Health. 2012;71(0):1–8.
18. Laidler GJ, Hirose T, Kapfer M, Ikummaq T, Joamie E, Elee P. Evaluating the Floe Edge
Service: How well can SAR imagery address Inuit community concerns around sea ice
change and travel safety? Can Geogr. 2011;55(1):91–107.
19. Lauer M, Aswani S. Indigenous knowledge and long-term ecological change: Detection,
interpretation, and responses to changing ecological conditions in pacific island
communities. Environ Manage. 2010;45(5):985–97.
20. Valdivia C, Seth A, Gilles JL, García M, Jiménez E, Cusicanqui J, et al. Adapting to climate
change in Andean ecosystems: Landscapes, capitals, and perceptions shaping rural
livelihood strategies and linking knowledge systems. Ann Assoc Am Geogr.
2010;100(4):818–34.
21. Heaslip R. Monitoring salmon aquaculture waste: The contribution of First Nations’ rights,
knowledge, and practices in British Columbia, Canada. Mar Policy. 2008;32(6):988–96.
22. Ntk. A life vest for Hudson Bay’s drifting stewardship. In: Arctic. 2008. p. 35–47.
23. Berkes F, Berkes MK, Fast H. Collaborative integrated management in Canada’s North: The
role of local and traditional knowledge and community-based monitoring. Coast Manag.
2007;35(1):143–62.
240
24. Robinson CJ, Smyth D, Whitehead PJ. Bush tucker, bush pets, and bush threats:
Cooperative management of feral animals in Australia’s Kakadu National Park. Conserv
Biol. 2005;19(5):1385–91.
25. Wiber M, Berkes F, Charles A, Kearney J. Participatory research supporting community-
based fishery management. Mar Policy. 2004;28(6):459–68.
26. Danby RK, Hik DS, Slocombe DS, Williams A. Science and the St Elias: An evolving
framework for sustainability in North America’s highest mountains. Geogr J.
2003;169(3):191–204.
27. Berkes F, Mathias J, Kislalioglu M, Fast H. The Canadian arctic and the Oceans Act: The
development of participatory environmental research and management. Ocean Coast
Manag. 2001;44(7–8):451–69.
28. Kendrick A. Canadian Inuit sustainable use and management of Arctic species. Int J
Environ Stud. 2013;70(3):414–28.
29. Oba G. Harnessing pastoralists’ indigenous knowledge for rangeland management: three
African case studies. Pastor Res Policy Pract [Internet]. 2012;2(1):1. Available from:
http://pastoralismjournal.springeropen.com/articles/10.1186/2041-7136-2-1
30. Dabasso BH, Oba G, Roba HG. Livestock-based knowledge of rangeland quality
assessment and monitoring at landscape level among borana herders of northern Kenya.
Pastoralism. 2012;2(1).
31. Kouril D, Furgal C, Whillans T. Trends and key elements in community-based monitoring:
a systematic review of the literature with an emphasis on Arctic and Subarctic regions.
Environ Rev [Internet]. 2016;24(2):151–63. Available from:
http://www.nrcresearchpress.com/doi/10.1139/er-2015-0041
32. Ens EJ, Towler GM, Daniels C. Looking back to move forward: Collaborative ecological
monitoring in remote Arnhem Land. Ecol Manag Restor. 2012;13(1):26–35.
33. Moller H, Berkes F, Lyver POB, Kislalioglu M. Combining Science and Traditional Ecological
Knowledge : Monitoring Populations for Co-Management. Ecol Soc. 2004;9(3):2.
34. Armitage D. Adaptive capacity and community-based natural resource management. Vol.
35, Environmental Management. 2005. p. 703–15.
35. Jollands N, Harmsworth G. Participation of indigenous groups in sustainable development
monitoring: Rationale and examples from New Zealand. Ecol Econ. 2007;62(3–4):716–26.
36. Paneque-Gálvez J, Vargas-Ramírez N, Napoletano B, Cummings A. Grassroots Innovation
Using Drones for Indigenous Mapping and Monitoring. Land [Internet]. 2017;6(4):86.
241
Available from: http://www.mdpi.com/2073-445X/6/4/86
37. Macherera M, Chimbari MJ, Mukaratirwa S. Indigenous environmental indicators for
malaria: A district study in Zimbabwe. Acta Trop. 2017;175:50–9.
38. Vigilante T, Ondei S, Goonack C, Williams D, Young P, Bowman D. Collaborative Research
on the Ecology and Management of the “Wulo” Monsoon Rainforest in Wunambal
Gaambera Country, North Kimberley, Australia. Land [Internet]. 2017;6(4):68. Available
from: http://www.mdpi.com/2073-445X/6/4/68
39. Shaffer CA, Milstein MS, Yukuma C, Marawanaru E, Suse P. Sustainability and
comanagement of subsistence hunting in an indigenous reserve in Guyana. Conserv Biol.
2017;31(5):1119–31.
40. Oviedo AFP, Bursztyn M. Community-based monitoring of small-scale fisheries with
digital devices in Brazilian Amazon. Fish Manag Ecol. 2017;24(4):320–9.
41. Mateo-Vega J, Potvin C, Monteza J, Bacorizo J, Barrigón J, Barrigón R, et al. Full and
effective participation of indigenous peoples in forest monitoring for reducing emissions
from deforestation and forest degradation (REDD+): Trial in Panama’s Darién. Ecosphere.
2017;8(2).
42. de Paula M, Xerente V, Pezzuti J. Hunting and monitoring: Community-based research in
xerente indigenous land, Brazilian Cerrado. Hum Ecol Rev [Internet]. 2017;23(1):23–44.
Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-
85038815827&doi=10.22459%2FHER.23.01.2017.02&partnerID=40&md5=f35ffbb57e5cf9
6acc97f18422006b8f
43. Ferguson DB, Masayesva A, Meadow AM, Crimmins MA, Ferguson DB, Masayesva A, et al.
Rain Gauges to Range Conditions: Collaborative Development of a Drought Information
System to Support Local Decision-Making. Weather Clim Soc [Internet]. 2016;8(4):345–59.
Available from: http://journals.ametsoc.org/doi/10.1175/WCAS-D-15-0060.1
44. Fragoso JMV, Levi T, Oliveira LFB, Luzar JB, Overman H, Read JM, et al. Line transect
surveys underdetect terrestrial mammals: Implications for the sustainability of subsistence
hunting. PLoS One. 2016;11(4).
45. Iverson SA, Forbes MR, Simard M, Soos C, Gilchrist HG. Avian cholera emergence in arctic-
nesting northern Common Eiders: Using community-based, participatory surveillance to
delineate disease outbreak patterns and predict transmission risk. Ecol Soc. 2016;21(4).
46. Constantino PAL. Dynamics of hunting territories and prey distribution in Amazonian
Indigenous Lands. Appl Geogr. 2015;56:222–31.
47. Bellfield H, Sabogal D, Goodman L, Leggett M. Case study report: Community-based
242
monitoring systems for REDD+ in Guyana. Forests. 2015;6(1):133–56.
48. Gill H, Lantz T. A Community-Based Approach to Mapping Gwich’in Observations of
Environmental Changes in the Lower Peel River Watershed, NT. J Ethnobiol [Internet].
2014;34(3):294–314. Available from: http://www.bioone.org/doi/abs/10.2993/0278-0771-
34.3.294
49. Lagasse CR, Ou W, Honka LD, Atlas WI, Hutton CN, Kotaska J, et al. Design considerations
for community-based stream monitoring to detect changes in pacific salmon habitats.
Ecol Soc. 2014;19(4).
50. Housty WG, Noson A, Scoville GW, Boulanger J, Jeo RM, Darimont CT, et al. Grizzly bear
monitoring by the Heiltsuk people as a crucible for First Nation conservation practice.
Ecol Soc. 2014;19(2).
51. Ford JD, McDowell G, Shirley J, Pitre M, Siewierski R, Gough W, et al. The Dynamic
Multiscale Nature of Climate Change Vulnerability: An Inuit Harvesting Example. Ann
Assoc Am Geogr. 2013;103(5):1193–211.
52. Constantino P de AL, Carlos HSA, Ramalho EE, Rostant L, Marinelli CE, Teles D, et al.
Empowering local people through community-based resource monitoring: A comparison
of Brazil and Namibia. Vol. 17, Ecology and Society. 2012.
53. Luzar JB, Silvius KM, Overman H, Giery ST, Read JM, Fragoso JM V. Large-scale
Environmental Monitoring by Indigenous Peoples. Bioscience [Internet]. 2011;61(10):771–
81. Available from: https://academic.oup.com/bioscience/article-
lookup/doi/10.1525/bio.2011.61.10.7
54. Wesche S, Schuster RC, Tobin P, Dickson C, Matthiessen D, Graupe S, et al. Community-
based health research led by the Vuntut Gwitchin First Nation. Int J Circumpolar Health
[Internet]. 2011;70(4):396–406. Available from:
https://www.tandfonline.com/doi/full/10.3402/ijch.v70i4.17846
55. Marcoux M, Auger-Méthé M, Chmelnitsky EG, Ferguson SH, Humphries MM. Local passive
acoustic monitoring of Narwhal presence in the Canadian Arctic: A pilot project. Arctic.
2011;64(3):307–16.
56. Gearheard S, Aporta C, Aipellee G, O’Keefe K. The Igliniit project: Inuit hunters document
life on the trail to map and monitor arctic change. Can Geogr. 2011;55(1):42–55.
57. Ens EJ, Cooke P, Nadjamerrek R, Namundja S, Garlngarr V, Yibarbuk D. Combining
aboriginal and non-aboriginal knowledge to assess and manage feral water buffalo
impacts on perennial freshwater springs of the aboriginal-owned arnhem plateau,
Australia. Environ Manage. 2010;45(4):751–8.
243
58. Mahoney A, Gearheard S, Oshima T, Qillaq T. Sea ice thickness measurements from a
community-based observing network. Bull Am Meteorol Soc. 2009;90(3):370–7.
59. Setty RS, Bawa K, Ticktin T, Gowda CM. Evaluation of a participatory resource monitoring
system for nontimber forest products: The case of Amla (Phyllanthus spp.) fruit harvest by
Soligas in South India. Ecol Soc. 2008;13(2).
60. Constantino P de AL, Fortini LB, Kaxinawa FRS, Kaxinawa AM, Kaxinawa ES, Kaxinawa AP,
et al. Indigenous collaborative research for wildlife management in Amazonia: The case of
the Kaxinawá, Acre, Brazil. Biol Conserv. 2008;141(11):2718–29.
61. Tremblay M, Furgal C, Larrivee C, Annanack T, Tookalook P, Qiisik M, et al. Climate
Change in Northern Quebec: Adaptation Strategies from Community-Based Research.
Arctic. 2008;61:27–34.
62. Ohl-Schacherer J, Shepard GH, Kaplan H, Peres CA, Levi T, Yu DW. The sustainability of
subsistence hunting by Matsigenka native communities in Manu National Park, Peru.
Conserv Biol. 2007;21(5):1174–85.
63. Noss AJ, Cuéllar E, Cuéllar RL. An evaluation of hunter self-monitoring in the Bolivian
Chaco. Hum Ecol. 2004;32(6):685–702.
64. Noss AJ, Cuéllar RL. Community attitudes towards wildlife management in the Bolivian
Chaco. ORYX. 2001;35(4):292–300.
65. Torres AVG. Turning scientific approaches into practical conservation actions: The case of
Comunidad Indigena de Nuevo San Juan Parangaricutiro, Mexico. Environ Manage.
2001;27(5):655–65.
66. Brook RK, Kutz SJ, Veitch AM, Popko RA, Elkin BT, Guthrie G. Fostering community-based
wildlife health monitoring and research in the Canadian North. Ecohealth. 2009;6(2):266–
78.
67. Boissière M, Bastide F, Basuki I, Pfund JL, Boucard A. Can we make participatory NTFP
monitoring work? Lessons learnt from the development of a multi-stakeholder system in
Northern Laos. Biodivers Conserv. 2014;23(1):149–70.
68. Krause T, Zambonino H. More than just trees – animal species diversity and participatory
forest monitoring in the Ecuadorian Amazon. Int J Biodivers Sci Ecosyst Serv Manag
[Internet]. 2013;9(3):225–38. Available from:
http://www.tandfonline.com/doi/abs/10.1080/21513732.2013.822930
69. Larter NC. A PROGRAM TO MONITOR MOOSE POPULATIONS IN THE DEHCHO REGION,
NORTHWEST TERRITORIES, CANADA. ALCES. 2009;45:89–99.
244
70. Noss AJ, Oetting I, Cuéllar RL. Hunter self-monitoring by the Isoseño-Guaraní in the
Bolivian Chaco. Biodivers Conserv. 2005;14(11):2679–93.
71. Pacheco-Cobos L, Rosetti MF, Esquivel AM, Hudson R. Towards a traditional ecological
knowledge-based monitoring scheme: a proposal for the case of edible mushrooms.
Biodivers Conserv. 2015;24(5):1253–69.
72. Gratani M, Royee F, Butler JRA. A research process and criteria indicators framework for
developing indigenous freshwater ecosystem health monitoring. Cogent Environ Sci
[Internet]. 2016;2(1):1–14. Available from:
https://www.cogentoa.com/article/10.1080/23311843.2016.1214228
73. Kofinas G, Lyver POB, Russell D, White R, Nelson A, Flanders N. Towards a protocol for
community monitoring of caribou body condition. Rangifer. 2003;Special Is:29–38.
74. Lyver POB, Timoti P, Jones CJ, Richardson SJ, Tahi BL, Greenhalgh S. An indigenous
community-based monitoring system for assessing forest health in New Zealand.
Biodiversity and Conservation. 2016;1–30.
75. Van Vliet N, Sandrin F, Vanegas L, L’haridon L, Fa JE, Nasi R. High-tech participatory
monitoring in aid of adaptive hunting management in the Amazon. Unasylva.
2017;68(249):53–62.
76. Driscoll DL, Sunbury T, Johnston J, Renes S. Initial findings from the implementation of a
community-based sentinel surveillance system to assess the health effects of climate
change in Alaska. Circumpolar Heal Suppl [Internet]. 2013;72:946–54. Available from:
http://10.0.13.74/ijch.v72i0.21405%5Cnhttp://search.ebscohost.com/login.aspx?direct=tru
e&AuthType=ip,url,cookie,uid&db=aph&AN=109175649&site=ehost-live&scope=site
77. Driscoll DL, Mitchell E, Barker R, Johnston JM, Renes S. Assessing the health effects of
climate change in Alaska with community-based surveillance. Clim Change. 2016;137(3–
4):455–66.
78. de Mattos Vieira MR, von Muhlen E, Shepard G. Participatory Monitoring and
Management of Subsistence Hunting in the Piagaçu-Purus Reserve, Brazil. Conserv Soc
[Internet]. 2015;13(3):254. Available from:
http://www.conservationandsociety.org/text.asp?2015/13/3/254/170399
79. Obura DO. Participatory monitoring of shallow tropical marine fisheries by artisanal
fishers in Diani, Kenya. In: Bulletin of Marine Science. 2001. p. 777–91.
80. Pollock RM, Whitelaw GS. Community-based monitoring in support of local sustainability.
Local Environ. 2005;10(3):211–28.
81. Funder M, Ngaga Y, Nielsen M, Poulsen M, Danielsen F. Reshaping Conservation: The
245
Social Dynamics of Participatory Monitoring in Tanzania′s Community-managed Forests.
Conserv Soc [Internet]. 2013;11(3):218. Available from:
http://www.conservationandsociety.org/text.asp?2013/11/3/218/121011
82. Fidel M, Kliskey A, Alessa L, Sutton O (Olia) P. Walrus harvest locations reflect adaptation:
a contribution from a community-based observation network in the Bering Sea. Polar
Geogr. 2014;37(1):48–68.
83. Herrmann TM, Sandström P, Granqvist K, D’Astous N, Vannar J, Asselin H, et al. Effects of
mining on reindeer/caribou populations and indigenous livelihoods: Community-based
monitoring by Sami reindeer herders in Sweden and First Nations in Canada. Polar J.
2014;4(1):28–51.
84. Parlee BL, Goddard E, First Nation ŁKD, Smith M. Tracking Change: Traditional Knowledge
and Monitoring of Wildlife Health in Northern Canada. Hum Dimens Wildl. 2014;19(1):47–
61.
85. Cummings A, Cummings G, Hamer E, Moses P, Norman Z, Captain V, et al. Developing a
UAV-Based Monitoring Program with Indigenous Peoples. J Unmanned Veh Syst.
2017;5(4):115–25.
86. Van Rijsoort J, Jinfeng Z. Participatory resource monitoring as a means for promoting
social change in Yunnan, China. Biodivers Conserv. 2005;14(11):2543–73.
87. Sheil D, Boissière M, Beaudoin G. Unseen sentinels: local monitoring and control in
conservation's blind spots. Ecol Soc [Internet]. 2015;20(2). Available from:
http://www.ecologyandsociety.org/vol20/iss2/art39/
246
Appendix 4: Questions to guide deductive qualitative analysis conducted on United Nations
food security evaluation reports published from 2014 to 2019.
Category Definition
Context Describes the social, political, economic, environmental, cultural, and/or
organizational setting within which a food security program was evaluated.
Evaluation
approach
Describes the purpose of the evaluation, role of the stakeholders, stages of
evaluation, and/or method of data collection and analysis.
Challenges and
opportunities
Describes limitations, challenges, opportunities, lessons learned, and/or
recommendations of evaluating climate action.
247
Appendix 5. Codebook of climate-responsive food security evaluation challenges.
Theme / codes Description Example
1. Inadequate
resources for
evaluation
The quality of the evaluation function is dependent on the amount of
resources allocated and whether resources are sufficient to cover the
scope of the evaluation.
Limited
evaluation
resources
Describes limited
resources available for
the final evaluation
function conducted at
the end of the initiative
including human
resources, capacity, time,
funding, and logistics
support.
“Given the limited time and resources
available, it was not possible for the PPE to
visit all project locations, undertake extensive
quantitative surveys, or examine the full
spectrum of project activities, achievements
and drawbacks” (IFAD 11, p. 6).
“Limited evaluation resources”
2. Weak M&E
systems
The evaluation function depends on data, support, and experiences from
existing M&E systems; yet, M&E systems are constrained by resource
availability and M&E design issues, leading to data quality and quantity
challenges for the evaluation function.
Limited M&E
systems
resources
Describes limited
resources available for
M&E
planning/implementation
conducted throughout
the initiative including
human resources,
capacity, time, logistics,
and funding.
“Historically the CO’s M&E Unit has had
limited human resources and funding and this
remains the case. WFP has insufficient human
resources to undertake adequate monitoring
of its SF programme and relies on the MoE
CCs and DFPs” (WFP 25, p. 24).
M&E design
issues
Describes challenges in
the design of the M&E
system including
indicators, targets,
criteria, and data
collection.
“In-house M&E is also an issue – some
cooperating partners found it burdensome,
and the system does not enable the reasons
for the changes in beneficiaries’ food security
status to be attributed or improved resilience
to be measured” (WFP 1, p. 43).
M&E data quality
and/or quantity
Describes implications of
M&E systems on data
quality and quantity.
“Beneficiaries reported an increase in
agricultural production and improvements in
food security at the household level. However,
quantitative data in addition to the data from
the RIMS survey reported below are not
248
available to support the attribution of these
benefits to the project” (IFAD 11, p. 29).
“Limited M&E resources” OR “M&E design issues” OR “M&E data quality and quantity”
3. Limited
climate action
focus
Lack of climate action focus in programming leads to limited prioritization
of climate change in the evaluation design, and consequently, limited
activities, outputs, and outcomes to be assessed in the evaluation.
Climate-related
programming
Describes climate
considerations in the
program objectives,
components, or activities.
“… strengthen the resilience of rural
populations and agricultural production
systems in the central part of the cattle
corridor and build the capacity of
communities, commercial farmers and the
Government of Uganda to cope with climate
change” (FAO 4, p. 5).
Climate-related
evaluation
Describes climate
considerations in
evaluation methodology,
methods, or analysis
“Crosscutting issues such as gender and other
equity issues, human rights, climate change
and partnerships are addressed under each of
the evaluation questions” (FAO 1, p. 14).
Climate-related
outcomes
Describes climate-related
outcomes.
“Adaptation to climate change was not part of
the project design. However, this criterion is
assessed for two reasons: climate change is a
threat for a country like Guyana that is heavily
agriculture-based, and where parts of the
country are below sea level; and some of the
project's activities could have helped in
climate change adaptation” (IFAD 3, p. 24).
Implications of
climate focus
Describes implications of
lack of climate action
focus on climate action
assessment and
outcomes.
There was also no indication of contribution to
climate change adaptation over the life of the
3PAD project, even if the awareness and
reporting on severe weather events and the
negative effects on crops and livestock
increased over the 3PAD project life time”
(IFAD 2, p. 32).
“climate-related programming” OR climate-related evaluation” OR (“climate-related outcomes”
AND “implications of climate focus”)
4. Dependency
on food security
evaluations
That the final evaluation function relies on how food security evaluations
are conducted present challenges if the quality of food security
evaluations are poor.
Food security
measurement
challenges
Describes challenges of
measuring food security
including attribution,
“This issue of attribution also arises strongly in
the sustainable agriculture development
component. There are a range of other
249
data availability, and data
quality.
programmes supporting agriculture in the
CBARDP states” (IFAD 40, p. 25).
Influence of
external factors
on food security
measurement
Describes influence of
external factors on food
security measurement
including climate change,
animal/crop diseases,
and poverty.
“Performance on results measurement for the
PRRO has been very minimal, for a variety of
reasons, of which the main ones are lack of a
well-developed M&E system, attribution
problems, dependability of food security on
external factors like rainfall and access to
agricultural inputs, incomplete sets of final
reports prepared by the partners, etc.” (WFP 3,
p. 40).
“Food security measurement challenges” OR “influence of external factors on food security
measurement”
250
Appendix 6. Codebook of climate-responsive food security evaluation solutions.
Theme Description Example
5. Invest in M&E
systems
To integrate climate action considerations into food security evaluations, a
strong M&E foundation is needed.
Provide
evaluation
support
Describes the need for
evaluation support
including mentoring,
guidance documents, or
manuals.
“Support the teams of future projects to
develop simple and effective devices for
monitoring and evaluation through the
development of a manual or monitoring-
evaluation guide and set up a database of the
project to contribute to better supervision
and better capitalization of actions and
results of future projects” (UNDP 1, p. 41).
Invest resources Describes the need for
resources including
human resources,
capacity, time, and
funding, communication,
and logistics support.
“Recommendation 4. Ensure sufficient
investment in and support for M&E, analytical
studies and documentation” (IFAD 3, p. 37).
Focus on
outcomes
Describes the value of
assessing outcomes /
impacts over activities /
outputs.
“The collection of the mass of output data
leaves limited space and time for the
collection of more detailed and useful
information on impact, attribution, and the
strength and sustainability of institutions
formed– issues which will be particularly
important in the move towards resilience
programming” (WFP 10, p. 48).
Link to decision-
making
Describes the need to link
data collection to data
needs of decision-makers.
“In terms of developing methods to measure
resilience and gathering and analysing data
for decisions, data need to be better linked to
decision making and the needs of decision
makers. FAO should better coordinate
activities so that resilience measurement
occurs in the same areas where social
protection analysis and policy and capacity
development are occurring” (FAO 5, p. 6).
“Strengthen evaluation capacity” OR “provide evaluation guidance” OR “invest resources” OR
“focus on outcomes” OR “link to decision-making”
251
6. Prioritize
climate action in
programming
Climate change considerations should be integrated into program and
evaluation planning.
Include climate
action strategies
Describes the importance
for programs to include
climate action goals,
strategies, or plans.
“Climate change is expected to have various
effects, requiring different coping strategies
for affected people in Guatemala’s diverse
eco-geographical environment. To enhance
impacts, FFA programming should pay more
attention to environmental factors” (WFP 4, p.
xiii).
Integrate climate
change in
program theory
Discusses how climate
change is integrated into
the program theory, logic,
or impact pathway.
See FAO 3, FAO 4, and UNEP 1.
Measure climate
action outcomes
Describes the need to
systematically measure
climate action outcomes
alongside food security
outcomes.
“Resilience measurement should be more
effectively linked to programme and policy
work carried out by FAO and should be more
crosscutting, linking to social protection,
nutrition and Disaster Risk Reduction” (FAO 5,
p. 6).
“include climate action strategies” OR “integrate climate change in program theory” OR
“measure climate action outcomes”
7. Prioritize
climate action in
evaluation.
During the conduct of the evaluation, it is important to integrate climate
change in throughout evaluation processes and products.
Climate change
as a cross-cutting
theme in the
evaluation
Describes the need to
integrate climate change
as a cross-cutting theme
in the evaluation.
“Crosscutting issues such as gender and other
equity issues, human rights, climate change
and partnerships are addressed under each of
the evaluation questions” (FAO 1, p. 14).
Climate action
co-benefits
Describes co-benefits of
climate action.
“The ET identified a number of environmental
benefits from the CO’s use of FFA to build
productive assets… There was also
widespread support for the FFA component’s
focus on water resources management
because of its ability to address Productive
(irrigation/drought) and Risk (drainage/ flood)
vulnerabilities at the target sites” (WFP 10, p.
19).
252
Climate action
trade-offs
Describes trade-offs of
climate action.
“The strategic choice to envisage an in-depth
community consultation processes, as was
done with the Joint Resilience Programme, is
very positive and consistent with the
resilience building aim. However, priorities
and trade-offs between different components
within a programme aiming to improve
something as broad as ‘resilience’ need to be
managed…” (FAO 8, p. 76).
Assess relevance
to climate
agendas
Describes a need for
establishing program
coherence with climate-
related programs,
policies, or strategies of
the national government
or UN entity.
“Given the thematic areas covered by the
PRRO, and especially with the introduction of
SO2 in July 2013, it could have been relevant
for the PRRO to seek coherence with national
policies and strategies focusing on rural
development, food security and adaptation
and mitigation to climate changes, and as
such to develop forms of coordination with
some technical directions of the Ministries in
charge of Agriculture, Livestock and
Environment” (WFP 3, p. 12).
Develop and
apply climate-
responsive M&E
tools
Describes the need for
dedicated tools to
account for climate action
outcomes.
“There is a need to provide useful and readily
available information and analysis tools to
stakeholders in order to allow them to
identify and take advantage of climate
change mitigation opportunities linked to
food security, resilience, adaptation and rural
development goals” (FAO 6, p. 36).
“Climate change as a cross-cutting theme in evaluation” OR “climate action co-benefits” OR
“climate action trade-offs” OR “assess relevance to climate agendas” OR “develop and apply
climate-responsive M&E tools”
253
Appendix 7: Thematic map of challenges and opportunities for climate-responsive evaluation.
Challenges are presenting in the upper half and opportunities are presented in the lower half.
Dotted arrows represent relationships between themes. Positive and negative signs highlight
feedback loops; for example, increasing investment in M&E systems (+) will mitigate weaknesses
in M&E systems (-), leading to increase in climate evaluation (+).
254
Appendix 8: List of all included United Nations food security evaluation reports published from
2014 to 2019.
Author Title Year
FAO 1 Final evaluation of Sudan Food Security Policy and Strategy Capacity
Building programme
2017
FAO 2 Final Evaluation of the Project “Strengthening Climate Change Resilience
and Disaster Risk Reduction in Agriculture to Improve Food Security in
Haiti After the Earthquake”
2018
FAO 3 Final Evaluation of the Project “Integrating Climate Resilience into
Agricultural Production for Food Security in Rural Areas of Mali”
2018
FAO 4 Final Evaluation of the Global Climate Change Alliance (GCCA) – Uganda:
Agricultural Adaptation to Climate Change project
2017
FAO 5 Final evaluation of the Improved Global Governance for Hunger
Reduction Programme
2016
FAO 6 Final Evaluation of “Monitoring and Assessment of GHG Emissions and
Mitigation Potentials in Agriculture Project” - GCP /GLO/286/GER
2014
FAO 7 Final Evaluation of the Project: "Management of Chimborazo’s Natural
Resources"
2019
FAO 8 Final Evaluation of the Joint Resilience Project in Kassala 2018
FAO 9 Final evaluation of the project “Mainstreaming the use and conservation
of agrobiodiversity in public policy through integrated strategies and in
situ implementation in four Andean Highlands provinces
2018
FAO 10 Final Evaluation of the Conservation Agriculture Scaling-Up (CASU)
Project Funded by the European Union
2018
FAO 11 Evaluation of the Benefit-sharing Fund second project cycle International
Treaty on Plant Genetic Resources for Food and Agriculture
2017
FAO 12 Evaluation of the Emergency Prevention System (EMPRES) Programme in
Food Chain Crises
2018
FAO 13 Final evaluation of the project “Climate Change Adaptation to Reduce
Land Degradation in Fragile Micro-Watersheds located in the
Municipalities of Texistepeque and Candelaria de la Frontera”
2019
FAO 14 Final Evaluation of the Project for Building Resilience and Self-reliance of
Livestock Keepers by Improving Control of Foot-and-Mouth Disease and
other Transboundary Animal Diseases in Afghanistan
2019
FAO 15 Final Evaluation of the Project on Decentralized Supply and Water Use
Management in the Sana’a Basin to Sustain Water Resources and Rural
Livelihoods
2019
FAO 16 Final Evaluation of the project “Forestry and Protected Area
Management in Fiji, Samoa, Vanuatu and Niue (GEFPAS-FPAM)”
2017
255
FAO 17 Final Evaluation of the Project “Increased Household Food, Income and
Nutrition Security through Commercialization of an Integrated and
Sustainable Smallholder Livestock Sector in Zimbabwe”
2018
FAO 18 Final Evaluation of the Programme for Improvement of Irrigation
Systems in Kabul, Bamyan and Kapisa Provinces
2017
FAO 19 Evaluation of the project on Strategies for Trawl Fisheries Bycatch
Management
2017
FAO 20 Final evaluation of the project: Food Security and Nutrition Analysis Unit
for Somalia
2017
FAO 21 Final evaluation of Capacity building for South African Professionals in
the Field of Agriculture and Food Security (GCP/RAF/412/SAF)
2017
FAO 22 Final Evaluation of the Groundwater Governance: A Global Framework
for Action
2016
FAO 23 Establishing a sustainable National Information and Early Warning
System (NIEWS) on Food Security in Timor-Leste
2016
FAO 24 End of project evaluation report of the Programme for Improvement of
Irrigation Systems and Construction of Micro-hydro Power Facilities in
Kabul and Bamyan Provinces and Mid-term evaluation report of the
Programme for Improvement of Irrigation Systems in Kabul and Bamyan
Provinces
2014
FAO 25 End of project evaluation report of the Programme for Improvement of
Irrigation Systems and Construction of Micro-hydro Power Facilities in
Kabul and Bamyan Provinces and Mid-term evaluation report of the
Programme for Improvement of Irrigation Systems in Kabul and Bamyan
Provinces
2014
FAO 26 Final Evaluation of the Project for Assistance to the Recovery and
Development of the Agricultural Economy in the Federally Administered
Tribal Areas (FATA) of Pakistan
2018
FAO 27 Final Evaluation of the Project “Building a Global Agenda of Action in
Support of Sustainable Livestock Sector Development”
2019
FAO 28 Final Evaluation of the Institutional Level Component of the Project
“Support for Livestock-based Livelihoods of Vulnerable Populations in
the Occupied Palestinian Territory"
2018
FAO 29 Final Evaluation of the Project “Capacity Development of the Irrigation
Department of the Ministry of Agriculture, Irrigation and Livestock of the
Islamic Republic of Afghanistan”
2018
FAO 30 Final evaluation of the Global Programme to Support the
implementation of the Voluntary Guidelines on the Responsible
Governance of Tenure of Land, Fisheries and Forests (2012 – 2016)
2017
256
FAO 31 Final Evaluation of the Environmentally Sustainable Food Security
Programme (ESFSP)
2017
FAO 32 Final evaluation of Sustainable Management of the Bay of Bengal Large
Marine Ecosystem (BOBLME) project
2016
FAO 33 Final Evaluation of Agriculture and Food Information Systems for
Decision Support in South Sudan
2016
FAO 34 Evaluation of the project Protecting and Improving Household Food
Security and Nutrition in HIV/AIDS Affected Areas in Manica and Sofala
Provinces. GCP/MOZ/079/BEL
2015
FAO 35 Final evaluation of the project “Securing tenure rights for forest
landscape-dependent communities: linking science with policy to
advance tenure security, sustainable forest management and people’s
livelihoods”
2019
FAO 36 Evaluation of the Global Strategy to Improve Agricultural and Rural
Statistics (GSARS)
2019
FAO 37 Evaluation of the project on Mainstreaming Food and Nutrition Security
and the Right to Food into the Smallholder Commercialization
Programme of Sierra Leone
2016
FAO 38 Final Evaluation of the Project “Strengthening Agricultural Market
Information Systems Globally and in Selected Countries (Bangladesh,
India and Nigeria) Using Innovative Methods and Digital Technology”
2017
FAO 39 Final Evaluation of “Progressive Control of Peste des Petits Ruminants in
Pakistan”
2017
FAO 40 Final evaluation of the project National Food Security Information
System in Kyrgyzstan
2015
FAO 41 Final Evaluation of the Project, Development of a Framework for the
Progressive Control of Foot and Mouth Disease in Pakistan
2016
FAO 42 The Agribusiness Support for Smallholders Project in Kenya –
GCP/KEN/070/GER
2014
FAO 43 Final evaluation of the project Education for Effective Nutrition in Action
(ENACT)
2016
FAO 44 Final Evaluation of the Integrated Food Security Phase Classification
(IPC) Global Strategic Programme (GSP) 2014 -2018
2019
IFAD 1 Pastoral Community Development Project II 2016
IFAD 2 Pro-poor partnerships for agroforestry development project 2018
IFAD 3 Rural enterprise and agricultural development project 2018
IFAD 4 Post-Tsunami Agricultural and Fisheries Rehabilitation Programme 2017
IFAD 5 Root and tuber improvement and marketing programme 2018
257
IFAD 6 Rural Livelihoods Improvement Project in Kratie, Preah Vihear and
Ratanakiri
2017
IFAD 7 Post-tsunami coastal rehabilitation and resource management
programme
2017
IFAD 8 Rural Livelihoods Support Programme (2017) 2017
IFAD 9 Finance for Enterprise Development and Employment Creation Project 2016
IFAD 10 Kenya: Smallholder horticulture marketing programme 2018
IFAD 11 Northern region sustainable livelihoods through livestock development
project
2018
IFAD 12 Participatory Natural Resource Management Programme 2017
IFAD 13 Georgia: Agricultural Support Project 2017
IFAD 14 West Noubaria Rural Development Project 2017
IFAD 15 Republic of Mozambique: Sofala Bank Artisanal Fisheries Project 2016
IFAD 16 Rural Microenterprise Promotion Programme 2016
IFAD 17 Rural Livelihoods Improvement Programme in Attapeu and Sayabouri 2015
IFAD 18 Gente de Valor - Rural Communities Development Project in the Poorest
Areas of the State of Bahia
2015
IFAD 19 India: Jharkhand Chhattisgarh Tribal Development Programme 2015
IFAD 20 Community Development Programme 2015
IFAD 21 Programme for Sustainable Development in Rural Mountain Areas 2015
IFAD 22 Gash Sustainable Livelihoods Regeneration Project 2014
IFAD 23 Sustainable Agriculture and Natural Resource Management Programme 2014
IFAD 24 Rural Development Project in the Mountain Zones of Al-Haouz 2014
IFAD 25 Agriculture, Marketing and Enterprise Promotion Programme 2014
IFAD 26 Rural Development Programme for Mountainous and Highland Areas 2014
IFAD 27 Microfinance for Marginal and Small Farmers Project 2014
IFAD 28 Rural Financial Services and Agribusiness Development Project 2019
IFAD 29 Agricultural Investments and Services Project 2016
IFAD 30 Livelihoods Improvement Project in the Himalayas 2015
IFAD 31 Support Project for the Strategic Plan for the Transformation of
Agriculture
2015
IFAD 32 Western Uplands Poverty Alleviation Project 2019
IFAD 33 Kirehe Community-based Watershed Management Project 2019
IFAD 34 Rural Finance and Enterprise Development Programme 2019
IFAD 35 Smallholder Plantations Entrepreneurship Development Programme 2019
IFAD 36 Northern Region Sustainable Livelihoods through Livestock
Development Project
2018
IFAD 37 Rural Financial Intermediation Programme 2017
IFAD 38 Rural Diversification Programme 2014
258
IFAD 39 Environment Conservation and Poverty-Reduction Programme in
Ningxia and Shanxi
2016
IFAD 40 Community-Based Agricultural and Rural Development Programme 2016
IFAD 41 Sivas-Erzincan Development Project 2015
UNDP 1 Final evaluation of the project "building the capacity of the agriculture
sector in DR Congo to plan for and respond to the additional threats
posed by climate change on food production and security" or pana-asa
project DR Congo
2015
UNDP 2 TUVALU NATIONAL PROGRAMME OF ACTION: INCREASING RESILIENCE
OF COASTAL AREAS AND COMMUNITY SETTLEMENTS TO CLIMATE
CHANGE IN TUVALU
2017
UNDP 3 TERMINAL EVALUATION: ADDRESSING CLIMATE CHANGE RISKS ON
WATER AND FOOD SECURITY IN THE DRY ZONE OF MYANMAR
2019
UNDP 4 TERMINAL EVALUATION OF ENABLING ENVIRONMENT FOR SLM TO
OVERCOME LAND DEGRADATION IN THE UGANDA CATTLE CORRIDOR
DISTRICTS OF UGANDA
2016
UNDP 5 TERMINAL EVALUATION OF ENHANCING ADOPTION OF CLIMATE
SMART AGRICULTURE PRACTICES IN UGANDA'S FARMING SYSTEMS
2016
UNDP 6 EVALUATION OF FOOD & NUTRITION SECURITY PROJECT IN JORDAN
TOWARDS POVERTY ALLEVIATION
2014
UNDP 7 Evaluation of Improved Seed Production for Sustainable Agriculture and
Reduction of Post-Harvest Losses for Food Security projects
2014
UNDP 8 Evaluation of Improved Seed Production for Sustainable Agriculture and
Reduction of Post-Harvest Losses for Food Security projects
2014
UNDP 9 PROMOTION OF DEVELOPMENT AND CONFIDENCE BUILDING IN THE
CHITTAGONG HILL TRACTS FINAL EVALUATION
2015
UNEP 1 TERMINAL EVALUATION OF UNEP GEF PROJECT: A GLOBAL INITIATIVE
ON LANDSCAPES FOR PEOPLE, FOOD AND NATURE
2015
UNEP 2 Terminal Evaluation of the UNDA 7th Tranche funded UNEP project:
Capacity Building in National Planning for Food Security
2016
UNEP 3 Terminal Evaluation of the UN Environment Project: Resource Efficiency
and Eco-Innovation in Developing and Transition Economies
2017
UNEP 4 TERMINAL EVALUATION OF THE UNEP GEF PROJECT: REDUCING
DEPENDENCE ON POPS AND OTHER AGRO-CHEMICALS IN THE
SENEGAL AND NIGER RIVER BASINS THROUGH INTEGRATED
PRODUCTION, PEST AND POLLUTION MANAGEMENT
2016
UNICEF
1
2017 Uganda: End of project Evaluation Enhanced Resilience Karamoja
Program (ERKP)
2017
259
UNICEF
2
2016 Pakistan: Evaluation of Humanitarian Actions - Nutrition
Emergency Response in District Tharparkar, Sindh, Pakistan
2016
UNICEF
3
2018 Ethiopia: Summative Evaluation of the Nutrition Component of the
EU-SHARE program in Ethiopia (2015-2018)
2018
UNICEF
4
2016 Malawi: Evaluation of Community Management of Acute
Malnutrition (CMAM) in Malawi
2016
UNICEF
5
2018 Lesotho: Impact evaluation of Lesotho’s Child Grants Programme
(CGP) and Sustainable Poverty Reduction through Income, Nutrition and
access to Government Services (SPRINGS) project
2018
UNICEF
6
2018 Korea: Evaluation of Community-based Management of Acute
Malnutrition Programme supported by UNICEF in DPR Korea 2015–2017
2018
UNICEF
7
2016 ROSA/EAPRO: Maternal and Young Child Nutrition Security
Initiative in Asia [MYCNSIA]: External Evaluation of the EU-UNICEF
Partnership 2011-2015
2016
UNICEF
8
2016 ESARO/WCARO: Evaluation of the EU/UNICEF Partnership on
Nutrition Security
2016
UNICEF
9
2019 Pakistan: United Nations Maternal and Child Stunting Reduction
Programme in three target Districts in Sindh, Pakistan
2019
UNICEF
10
Zimbabwe’s Harmonised Social Cash Transfer Programme Endline
Impact Evaluation Report
2018
UNICEF
11
Impact evaluation of Lesotho’s Child Grants Programme (CGP) and
Sustainable Poverty Reduction through Income, Nutrition and access to
Government Services (SPRINGS) project
2018
WFP 1 Zimbabwe PRRO 200453 Responding to Humanitarian Needs and
Strengthening Resilience to Food Insecurity: An Operation Evaluation
2014
WFP 2 Tajikistan PRRO 200122 Restoring Sustainable Livelihoods for Food-
Insecure People: An Operation Evaluation
2014
WFP 3 Madagascar PRRO 200065 Response to Recurrent Natural Disasters and
Seasonal Food Insecurity
2014
WFP 4 Food for Assets on Livelihood Resilience in Guatemala: An Impact
Evaluation
2014
WFP 5 Food for Assets on Livelihood Resilience in Senegal: An Impact
Evaluation
2014
WFP 6 Food for Assets on Livelihood Resilience in Uganda: An Impact
Evaluation
2014
WFP 7 The Gambia PRRO 200557 Targeted Nutrition and Livelihood Support
for Vulnerable People Impacted by Floods and Drought: An Operation
Evaluation
2016
WFP 8 Bangladesh, School Feeding Programme (2015-2017): an evaluation 2018
260
WFP 9 Ethiopia, School Feeding Programme: an evaluation 2018
WFP 10 Cameroon PRRO 200552 Food and Nutrition Assistance to Nigerian and
Central African Refugees and Host Populations in Cameroon: An
Operation Evaluation
2016
WFP 11 Kenya, Food for Education and Child Nutrition: an endline evaluation 2017
WFP 12 Myanmar PRRO 200299 "Supporting Transition by Reducing Food
Insecurity and Undernutrition among the Most Vulnerable": An
Operation Evaluation
2017
WFP 13 Ethiopia PRRO 200700 (2015-2018) Food Assistance for Eritrean, South
Sudanese, Sudanese and Somali Refugees: An Operation Evaluation
2016
WFP 14 Philippines PRRO 200296 Support for Returnees and Other Conflict
Affected Households in Central Minanao, and National Capacity
Development in Disaster Preparedness and Response: An Operation
Evaluation
2014
WFP 15 Pakistan, Food Assistance to Temporarily Dislocated Persons (2015-
2017): an evaluation
2018
WFP 16 Algeria PRRO 200301: Evaluation of the Nutrition Components 2018
WFP 17 Malawi, School Meals Programme (2013-2015): an evaluation 2018
WFP 18 Sierra Leone, PRRO 200938: an evaluation 2018
WFP 19 Kenya PRRO 200174 Food Assistance to Refugees (2011-2013): An
Operation Evaluation
2014
WFP 20 Swaziland DEV 200422 Support to children and students affected by HIV
and AIDS (2013-2014): An Operation Evaluation
2014
WFP 21 Swaziland DEV 200422 Support to children and students affected by HIV
and AIDS (2013-2014): An Operation Evaluation
2014
WFP 22 Cambodia, United States Department of Agriculture (USDA) McGovern-
Dole Grant Food for Education Programme: An Endline Evaluation
2018
WFP 23 Impact Evaluation of WFP’s Fresh Food Voucher Pilot Programme in
Ethiopia
2019
WFP 24 Laos, School Feeding Programme (2014-2016): an endline evaluation 2018
WFP 25 Liberia, Food for Education and Child Nutrition: an evaluation 2017
WFP 26 Liberia PRRO 200550 Food Assistance For Refugees And Vulnerable Host
Populations: An Operation Evaluation
2016
WFP 27 India, Target Public Distribution Reforms in Bhubaneswar: Evaluation 2019
261
Appendix 9: Performance of United Nations food security evaluations published from 2014 to
2019 (n = 136) according to components of the climate assessment rubric.
Category 0 points 1 point 2 points 3 points
Evaluation scope 50 (36.8%) 65 (47.8%) 21 (15.4%) N/A
Evaluation approach 126 (92.6%) 10 (7.4%) N/A N/A
Evaluation results 38 (27.9%) 57 (41.9%) 7 (5.1%) 34 (25.0%)
262
Appendix 10: Climate change mandates of United Nations agencies.
Box 1. Climate change mandates of United Nations agencies.
UN agencies are driving many aspects of global food systems. Below, we outline the mandates of UN
agencies related to climate change action according to most-up-to-date policy or strategy document.
According to UNDP’s 2015 Social and Environmental Standards, UNDP ensures that all of its projects
are sensitive to climate change and disaster risks and do not contribute to increased vulnerability to
climate change and natural hazards.
According to WFP’s 2017 Climate Change Policy, WFP’s goal is to support food-insecure communities
in building their capacities to address the impacts of climate change on hunger. WFP will do so
through three main disaster risk reduction objectives.
According to FAO’s 2017 Strategy on Climate Change, climate action is a cross-cutting theme of FAO’s
strategic framework and is being integrated into every facet of FAO’s work.
According to IFAD’s 2019 Strategy and Action Plan on Environment and Climate Change, IFAD will
address environment and climate change across all of IFAD’s policies, strategies, and operations.
According to UNEP’s 2008 Climate Change Strategy, climate change is one of the six cross-cutting
thematic priorities in UNEP’s medium-term strategy.
According to UNICEF’s 2015 Strategic Framework on Environmental Sustainability for Children, UNICEF
is committed to increase its contribution to environmental sustainability. Five priorities were outlined
such as strengthening UNICEF policy and guidance on environmental sustainability as a cross-cutting
issue.
263
Appendix 11: Comparison between average climate action integration score of United Nations
evaluations and average program duration.
UN agency Average climate change
integration score
Median program duration
(years)
UNDP 3.3 3
WFP 2.4 3
FAO 2.3 4
IFAD 1.9 8
UNEP 1.3 3
UNICEF 0.7 2
264
Appendix 12: Comparison in climate-related reporting between high and low scoring climate
action integration of United Nations evaluations.
High climate action integration Low climate action integration
- Evaluation of a protracted relief and
recovery program focused on restoring
sustainable livelihoods in Tajikistan (WFP 2)
- Program timeline: 2010 to 2014
- Context outlined risks of Tajikistan to
disasters including floods and drought
- Assessed whether climate change
adaptation principles were integrated in the
program
- Presented climate-related outcomes and
recommended the development of a country
strategy that includes contingency plans
framed by a broader context analysis
(including climate risk and regional insecurity)
beyond food insecurity
- Evaluation of a protracted relief and
recovery program focused on food assistance
to refugees in Kenya (WFP 19)
- Program timeline: 2011 to 2014
- Context outlined the impact of more
extreme and frequent droughts in Kenya
- Logic model mentioned the risk of droughts
and floods on program impacts
- Stated the program objective of
contributing to disaster preparedness and
mitigation “cannot be adequately measured
due to lack of data” (p. 46)
- Evaluation of a food security policy in Sudan
(FAO 1)
- Program timeline: 2012 to 2015
- Context mentioned climatic variability
resulting in recurrent droughts and floods
- Stated crosscutting issues of climate change
were addressed under evaluation questions
- Presented outcomes related to drought and
flood response
- Evaluation of a food information systems
program in Sudan (FAO 33)
- Program timeline: 2013 to 2015
- No mention of climate change in the
context or methods section
- Findings included the effectiveness of the
program in contributing to the collection,
analysis, and dissemination of food security
information such as climate
- Evaluation of a climate change resilience
building program in Tuvalu (UNDP 2)
- Program timeline: 2009 to 2016
- Context mentioned the vulnerability of
Tuvalu to the impacts of climate change
- Several indicators reflected climate
considerations (e.g. % of planners with
improved understanding of climate change
risks and adaptation measures)
- Presented outcomes related to enhanced
capacity of communities in identifying
climate-related risks
- Evaluation of a program aiming to build
capacity of institutions in community
development in Bangladesh and India (UNDP
9)
Program timeline: 2003 to 2015
- No mention of climate change in the
context or methods section
- Findings included the increase use of
cooking stoves, reducing the impact of
firewood use on climate change
265
Appendix 13: Search strategy used to identify Theory of Change and Realist Evaluation studies
from AGRICOLA©, CabDirect©, Web of Science™ CORE Collection, Medline®, Scopus®, EconLit©,
and Google™.
Database/search engine/website Search string
- AGRICOLA©
- CabDirect©
- Web of Science™ CORE Collection
- Medline®
- Scopus®
- EconLit©
(food OR agriculture OR nutrition OR livestock OR
fish* OR animal OR plant OR wildlife) AND ((“theor*
of change”) OR (realist AND evaluation) OR (realism
AND evaluation))
Google™ a 1) “food security” AND “theory of change”
2) “food security” AND “realist evaluation”
Theory of Change community b food
CGSpace c “theory of change” AND “realist evaluation” a We conducted two Google™ searches using the search string (“food security” AND “theory of
change”) and (“food security” AND “realist evaluation”). The first 100 sources from each Google™
search were reviewed for inclusion in the review. As Google™ returns results based on relevance
criteria related to the search term entered, every 50 sources were reviewed. If deemed relevant,
the next source (e.g. 51st, 52nd, and so on) were reviewed until a source not relevant to the
search goal was found at which point another 50 sources were skipped to check for relevancy.
This process was repeated until the 500th source (as in Kouril et al. 201615). b We selected this resource for their repository of Theory of Change related publications. c We selected this resource because the Consultative Group on International Agricultural
Research (CGIAR) is a major feature in the agri-development space with 15 research centres
headquartered in 15 countries. Theories of Change and related impact pathways are important
evaluation tools used by CGIAR research centres.
15 Kouril D, Furgal C and Whillans T (2016) Trends and key elements in community-based monitoring: a systematic
review of the literature with an emphasis on Arctic and Subarctic regions. Environmental Reviews 24(2): 151–163.
DOI: 10.1139/er-2015-0041.
266
Appendix 14: Summary of the screening tools used in the title and abstract screening stage and
full text screening stage to identify relevant Theory of Change and realist evaluation studies.
Title and abstract screening questions and responses (level 1) a
Does this study describe the development, implementation, and/or evaluation of a food
security program?
Food security: a situation that exists when all people, at all times, have physical, social and
economic access to sufficient, safe and nutritious food that meets their dietary needs and food
preferences for a healthy life.
• Yes – include
• No – exclude (if no, questions 2 and 3 are skipped)
• Unsure - include
Does this study explicitly mention using a Theory of Change and/or Realist Evaluation
approach?
Theory of Change is an ongoing process of reflection to explore change and how it happens –
and what that means for the part we play in a particular context, sector and/or group of people.
ToC is often presented as a visual depicting the change pathway. Realist evaluation is a member
of a family of theory-based evaluation approaches which begin by clarifying the ‘program
theory’: the mechanisms that are likely to operate, the contexts in which they might operate and
the outcomes that will be observed if they operate as expected.
• Yes – include
• No – exclude (if no, question 3 is skipped)
• Unsure – include
Is this an individual research study and published in a journal or report?
Exclude conference abstracts, letters to the editor, news articles, dissertations, and reviews
• Yes – include
• No – exclude
• Unsure – include
Title and abstract screening questions and responses (level 2)
Does this study describe the development, implementation, and/or evaluation of a food
security program?
Food security: a situation that exists when all people, at all times, have physical, social and
economic access to sufficient, safe and nutritious food that meets their dietary needs and food
preferences for a healthy life.
• Yes – include
• No – exclude
Does this study explicitly mention using a Theory of Change and/or Realist Evaluation
267
approach?
Theory of Change is an ongoing process of reflection to explore change and how it happens –
and what that means for the part we play in a particular context, sector and/or group of people.
ToC is often presented as a visual depicting the change pathway. Realist evaluation is a member
of a family of theory-based evaluation approaches which begin by clarifying the ‘program
theory’: the mechanisms that are likely to operate, the contexts in which they might operate and
the outcomes that will be observed if they operate as expected.
• Yes – include
• Yes but not used correctly as per definitions of ToC and/or RE – exclude
• No – exclude
Is this an individual research study and published in a journal or report?
Exclude conference abstracts, letters to the editor, news articles, dissertations, and reviews
• Yes – include
• No – exclude a For level one screening, these criteria were stacked (i.e., if the first criterion is not met, then the
second two criteria will not be not examined). For level two screening, all of these criteria were
examined.
268
Appendix 15: The data charting form used to extract relevant information from Theory of
Change and Realist Evaluation studies.
Guiding questions Response type Response options
Characteristics of included studies
What year was the study
published online?
Radio (i.e.
select one)
2011; 2012; 2013; 2014; 2015; 2016; 2017;
2018; 2019; 2020
Is this study from the
published or grey
literature?
Radio Published literature (e.g. research article,
letter, review);
Grey literature (e.g. workshop report,
research report, book chapter)
What food security topic(s)
did the study focus on? a
Checkbox Nutrition (addresses malnutrition,
including overnutrition, undernutrition,
and associated conditions such as
stunting);
Agricultural productivity (supports the
production and productivity of crop,
livestock, and aquaculture systems);
Market-based (supports consumer
linkage to markets, value chain
development, and agri-food business
development);
Climate change (supports adaptation of
food systems to climate change);
Agricultural innovation systems
(supports collective action and
coordination, exchange of knowledge
among diverse actors, and enabling
conditions that make it possible for
actors to innovate);
Natural resource management
(supports the sustainable management,
use, and protection of forests, land, soil,
water, and biodiversity)
Food safety (addresses threats to food
security from food-borne diseases)
Community food security (address
needs and priorities within a local food
system)
Food security risk management
269
(supports management of risks including
marketing mechanisms and financial
insurance.
Food loss (supports efforts to reduce
food loss and waste including improved
harvesting techniques, on-farm storage,
and infrastructure)
What was the scale of the
study?
Checkbox Local (community level);
Provincial (regional level);
National (country level);
International (more than one country)
What region(s) was the
study conducted in?
Checkbox North America; South America; Europe;
Asia; Africa; Australia and Oceania; Not
applicable
Theory of Change studies
Did this study include a
ToC?
If yes, the following
questions below are asked:
Radio Yes; No; Not applicable
What was the purpose of
the ToC?
Checkbox Program planning; Program
implementation; Program evaluation;
Other (specify)
What evaluation approach
was used?
Checkbox Qualitative (workshops, group
discussions/meetings, interviews);
Quantitative (surveys, experiments);
Mixed qualitative-quantitative (uses
both qualitative and quantitative
methods);
Not applicable (study did not specify
characteristics of case studies);
Not specified (study did not provide
sufficient detail to categorize the
evaluation approach)
Who was involved in the
development of the ToC?
Checkbox Program staff; Program participants;
Evaluation team; Funders; Other (specify)
Did the ToC mention the
influence of contextual
factors on the program?
Radio
Yes; No; Not applicable
270
If yes, which?
Checkbox
Cultural (beliefs, attitudes, and practices
of participants);
Socio-economic (distribution of social
and economic resources);
Environment (features of the physical
environment);
Political (the political landscape);
Other (specify)
Was climate change (i.e.
climate change adaptation,
mitigation, impacts)
mentioned in the outcome
of the ToC (i.e. program
theory)?
Radio Yes; No; Not applicable
Did the study disaggregate
data by social identity?
If yes, which?
Radio
Checkbox
Yes; No; Not applicable
Age; Gender; Population (e.g. Indigenous
peoples); Other
Realist Evaluation studies
Did this study include a RE?
If yes, the following
questions below are asked:
Radio Yes; No; Not applicable
What is the purpose of the
RE?
Checkbox Program planning; Program
implementation; Program evaluation;
Other (specify)
What evaluation approach
was used?
Checkbox Qualitative (workshops, group
discussions/meetings, interviews);
Quantitative (surveys, experiments);
Mixed qualitative-quantitative (uses
both qualitative and quantitative
methods);
Not applicable (study was a
conceptual/theoretical paper);
Not specified (study did not provide
sufficient detail to categorize the
evaluation approach)
271
Who was involved in the
development of the RE?
Checkbox Program staff; Program participants;
Evaluation team; Funders; Other (specify)
Did the RE mention the
influence of contextual
factors on the program?
If yes, which?
Radio
Checkbox
Yes; No; Not applicable
Cultural (beliefs, attitudes, and practices
of participants);
Socio-economic (distribution of social
and economic resources);
Environment (features of the physical
environment);
Political (the political landscape);
Other (specify)
Was climate change (i.e.
climate change adaptation,
mitigation, impacts)
mentioned in the outcome
of the RE (i.e. program
theory)?
Radio Yes; No; Not applicable
Did the study disaggregate
data by social identity?
If yes, which?
Radio
Checkbox
Yes; No; Not applicable
Age; Gender; Population (e.g. Indigenous
peoples); Other a These food security topics were developed through an inductive thematic analysis of studies
captured in this review. Of note, these topics are not mutually exclusive and do overlap; for
example, agricultural systems that contribute to sustainable increases in food production can
relate to climate change and agricultural productivity.
272
Appendix 16: List of references of all included studies on food-related Theory of Change and
Realist Evaluation.
Álvarez-Mingote C, Moore A and McNamara P (2020) Assessing the role of stakeholder
platforms as drivers of resilient communities: the case of Malawi. Journal of Agricultural
Education and Extension. DOI: 10.1080/1389224X.2019.1674169.
Apgar JM, Allen W, Albert J, et al. (2017) Getting beneath the surface in program planning,
monitoring and evaluation: Learning from use of participatory action research and theory
of change in the CGIAR Research Program on Aquatic Agricultural Systems. Action
Research 15(1): 15–34. DOI: 10.1177/1476750316673879.
Arriola KRJ, Ellis A, Webb-Girard A, et al. (2020) Designing integrated interventions to improve
nutrition and WASH behaviors in Kenya. Pilot and Feasibility Studies. DOI:
10.1186/s40814-020-0555-x.
Atherstone C (2014) Joint CGIAR meeting on aflatoxins. Nairobi.
Baker, D., Speedy, A., Hambrey J (2014) Report of the CGIAR Research Program on Livestock
and Fish Commissioned External Evaluation of the Program’s Value Chain Approach.
Nairobi.
Baker J, Sanghvi T, Hajeebhoy N, et al. (2013) Using an evidence-based approach to design
large-scale programs to improve infant and young child feeding. Food and nutrition
bulletin. DOI: 10.1177/15648265130343s202.
Béné C, Riba A and Wilson D (2020) Impacts of resilience interventions – Evidence from a
quasi-experimental assessment in Niger. International Journal of Disaster Risk Reduction.
DOI: 10.1016/j.ijdrr.2019.101390.
Canales, C., Manson, N., Jones B (2017) Mzima Cow Strategy & Theory of Change – Translating
from Genetic Research in Africa to Adoption and Social Value: Workshop Report. Nairobi.
Catholic Relief Services (2017) Theory of Change, Metrics and Learning Agenda.
Child, K., Kidoido, M., Kanyuuru C (2015) Using the Theories of Change approach to monitor,
evaluate and learn in the CGIAR research program on Livestock and Fish. Nairobi.
Clarkson G, Garforth C, Dorward P, et al. (2018) Can the TV makeover format of edutainment
lead to widespread changes in farmer behaviour and influence innovation systems?
Shamba Shape Up in Kenya. Land Use Policy 76. Oxford: Elsevier Ltd: 338–351. DOI:
10.1016/j.landusepol.2018.05.011.
Cole DC, Levin C, Loechl C, et al. (2016) Planning an integrated agriculture and health program
and designing its evaluation: experience from Western Kenya. Evaluation and Program
Planning 56. Oxford: Elsevier Ltd: 11–22. DOI: 10.1016/j.evalprogplan.2016.03.001.
Darisi, T., Watt H (2018) Envisioning a Circular Food Economy Theory of Change for Smart
Cities. Guelph, Canada.
Douthwaite, B., Kamp, K., Longley, C., Kruijssen F., Puskur, R., Chiuta, T., Apgar, M., Dugan P
(2013) Using theory of change to achieve impact in AAS. Penang, Malaysia.
273
Douthwaite B (2019) Outcome Evaluation of the work of the CGIAR Research Program on Land,
Water and Ecosystems (WLE) on soil and water management in Ethiopia. Ethiopia.
Douthwaite B and Hoffecker E (2017) Towards a complexity-aware theory of change for
participatory research programs working within agricultural innovation systems.
Agricultural Systems 155: 88–102. DOI: 10.1016/j.agsy.2017.04.002.
Douthwaite B, Mur R, Audouin S, et al. (2017) Agricultural research for development to
intervene effectively in complex systems and the implications for research organizations.
KIT Working Paper Series - Sustainable Economic Development. Amsterdam: Koninklijk
Instituut voor de Tropen (KIT) (Royal Tropical Institute, RTI). Available at:
http://213ou636sh0ptphd141fqei1-wpengine.netdna-ssl.com/sed/wp-
content/uploads/sites/2/2017/09/KIT-working-papers-2017-12.pdf.
Douthwaite Boru, Mayne J, McDougall C, et al. (2017) Evaluating complex interventions: A
theory-driven realist-informed approach. Evaluation 23(3): 294–311. DOI:
10.1177/1356389017714382.
Farrier A, Dooris M and Morley A (2019) Catalysing change? A critical exploration of the
impacts of a community food initiative on people, place and prosperity. LANDSCAPE
AND URBAN PLANNING 192. DOI: 10.1016/j.landurbplan.2019.103663.
Floate H, Durham J and Marks GC (2019) Moving on from logical frameworks to find the
‘missing middle’ in international development programmes. Journal of Development
Effectiveness 11(1). Abingdon: Routledge: 89–103. DOI: 10.1080/19439342.2018.1551921.
Howland F (2019) Individual capacity baseline report. Hanoi.
Johnson, N., Douthwaite, B., Mayne J (2019) Using Theory of Change in Agricultural Research
for Food and Nutrition Security. Rome.
Johnson N, Mayne J, Grace D, et al. (2015) How will training traders contribute to improved
food safety in informal markets for meat and milk? A theory of change analysis. IFPRI -
Discussion Papers. Washington: International Food Policy Research Institute. Available at:
http://www.ifpri.org/cdmref/p15738coll2/id/129293/filename/129504.pdf.
Johnson N, Atherstone C and Grace D (2015) The potential of farm-level technologies and
practices to contribute to reducing consumer exposure to aflatoxins: a theory of change
analysis. IFPRI - Discussion Papers. Washington: International Food Policy Research
Institute. Available at:
http://www.ifpri.org/cdmref/p15738coll2/id/129294/filename/129505.pdf.
Johnson N, Guedenet H and Saltzman A (2015) What will it take for biofortification to have
impact on the ground? Theories of change for three crop-country combinations. IFPRI -
Discussion Papers. Washington: International Food Policy Research Institute. Available at:
http://www.ifpri.org/sites/default/files/publications/ifpridp01427.pdf.
Kidoido, M., Korir L (2013) Report of the Tanzania Dairy Value Chain Impact Pathways
Workshop, Dar-es-Salaam, Tanzania, 7-8 May 2013. Nairobi.
Leavy, J., Boydell, E., McDowell, S., Sladkova B (2018) Resilience results BRACED final
evaluation. Washington.
274
Levay A V, Chapman GE, Seed B, et al. (2019) District-level implementation of British
Columbia’s school food and beverage sales policy: a realist evaluation exploring
intervention mechanisms in urban and rural contexts. Canadian Journal of Public Health
110(1). Cham: Springer International Publishing AG: 21–30. DOI: 10.17269/s41997-018-
0159-x.
LIFT (2016) Monitoring and Evaluation for Accountability and Learning (MEAL) Framework.
Maru Y, Sparrow A, Stirzaker R, et al. (2018) Integrated agricultural research for development
(IAR4D) from a theory of change perspective. Agricultural Systems 165. Oxford: Elsevier
Ltd: 310–320. DOI: 10.1016/j.agsy.2016.09.012.
Maye D, Enticott G and Naylor R (2019) Theories of Change in Rural Policy Evaluation.
Sociologia Ruralis. DOI: 10.1111/soru.12269.
Mayer AM, Ndolo R and Keylock J (2018) Operational factors in the integration of nutrition
into agriculture and livelihoods programmes in Zimbabwe. Field Exchange - Emergency
Nutrition Network ENN (No.57). Oxford: Emergency Nutrition Network (ENN): 31–34.
Available at: https://www.ennonline.net/fex/57/operationalfactorsnutzimbabwe.
Mayne, J., Stern, E., Douthwaite B (2013) AAS Practice Brief: Evaluating natural resource
management programs. Penang, Malaysia.
Mayne J and Johnson N (2015) Using theories of change in the CGIAR Research Program on
Agriculture for Nutrition and Health. Evaluation 21(4): 407–428. DOI:
10.1177/1356389015605198.
McDonald B (2011) Managing water and land at the interface between fresh and saline
environments An impact evaluation. Colombo, Sri Lanka.
MDF (2017) Evaluation of GAIN Programme Driving Nutrition Impact in Food Security. Ede, the
Netherlands.
Merrey, D.J., Swaans, K., Borgne EL (2013) Lessons from the Nile Basin Development Challenge
Program: An Institutional History. Colombo, Sri Lanka.
Michelini L, Grieco C, Ciulli F, et al. (2020) Uncovering the impact of food sharing platform
business models: a theory of change approach. BRITISH FOOD JOURNAL 122(5). DOI:
10.1108/BFJ-06-2019-0422.
Nair S (2016) Addressing environmental degradation and rural poverty through climate
change adaptation: an evaluation of social learning in drought-affected districts of
southern India. CCAFS Working Paper. Frederiksberg: CGIAR Research Program on
Climate Change, Agriculture and Food Security. Available at:
https://cgspace.cgiar.org/rest/bitstreams/80444/retrieve.
Nimpagaritse M, Korachais C and Meessen B (2020) Effects in spite of tough constraints - A
theory of change based investigation of contextual and implementation factors affecting
the results of a performance based financing scheme extended to malnutrition in
Burundi. PLoS ONE. DOI: 10.1371/journal.pone.0226376.
Obodai J, Adjei POW, Hamenoo SVQ, et al. (2018) Towards household food security in Ghana:
assessment of Ghana’s expanded forest plantation programme in Asante Akim South
275
District. GeoJournal 83(2). Amsterdam: Springer: 365–380. DOI: 10.1007/s10708-017-
9776-9.
Ohly H, Crossland N, Dykes F, et al. (2019) A realist qualitative study to explore how low-
income pregnant women use Healthy Start food vouchers. Maternal and Child Nutrition.
DOI: 10.1111/mcn.12632.
Omore, A., Twine, E., Githinji, J., Kanyuuru, C., Kidoido M (2016) Maziwa Zaidi Theory of
Change: Context and ‘start point’. Nairobi.
Omore A, Kidoido M, Twine E, et al. (2019) Using “theory of change” to improve agricultural
research: recent experience from Tanzania. Development in Practice 29(7): 898–911. DOI:
10.1080/09614524.2019.1641182.
Pound, B., Tolera, A., Matsaert H (2015) Report of the internally-commissioned external review
of the Africa RISING project in the Ethiopian Highlands. Nairobi.
Pradel, W., Cole, D.C., Prain G (2013) Mixing methods for rich and meaningful insight:
Evaluating changes in an agricultural intervention in the Central Andes.
Roundtable on Sustainable Biomaterials (2014) RSB’s Theory of Change.
Schuetz T, Förch W, Thornton P, et al. (2017) Pathway to Impact: Supporting and Evaluating
Enabling Environments for Research for Development. In: Evaluating Climate Change
Action for Sustainable Development. DOI: 10.1007/978-3-319-43702-6_4.
Sustain Ontario (2012) Theory of Change Charting a Path to Healthy Food and Farming
Systems. Ottawa, Canada.
Thornton PK, Schuetz T, Förch W, et al. (2017) Responding to global change: A theory of
change approach to making agricultural research for development outcome-based.
Agricultural Systems 152: 145–153. DOI: 10.1016/j.agsy.2017.01.005.
Ton G, Vellema S and Ge L (2014) The Triviality of Measuring Ultimate Outcomes:
Acknowledging the Span of Direct Influence. IDS BULLETIN-INSTITUTE OF DEVELOPMENT
STUDIES 45(6, SI): 37–48. DOI: 10.1111/1759-5436.12111.
Vellema S, Maru YT, Ekong J, et al. (2017) Do theories of change enable innovation platforms
and partnerships to navigate towards impact? KIT Working Paper Series - Sustainable
Economic Development. Amsterdam: Koninklijk Instituut voor de Tropen (KIT) (Royal
Tropical Institute, RTI). Available at: https://213ou636sh0ptphd141fqei1-
wpengine.netdna-ssl.com/sed/wp-
content/uploads/sites/2/2015/06/KIT45_opmaak_WPS_13-2017_v2.pdf.
Vogel I (2012) Review of the use of ‘Theory of Change’ in international development. UK.
Wesley AS, De Plaen R, Michaux KD, et al. (2019) Integrating nutrition outcomes into
agriculture development for impact at scale: Highlights from the Canadian International
Food Security Research Fund. Maternal and Child Nutrition. DOI: 10.1111/mcn.12812.
WFP (2018) An Impact Evaluation of WFP Malnutrition Interventions in Niger. Rome, Italy.
276
Appendix 17: Coding process to analyze Theory of Change (ToC) and Realist Evaluation (RE)
studies.
277
Appendix 18. Number of publications focusing on food-related Theory of Change and Realist
Evaluation studies (note: review cut off 10 March 2020).
0
2
4
6
8
10
12
14
16
18
20
2011 2012 2013 2014 2015 2016 2017 2018 2019
Nu
mb
er o
f p
ub
lica
tio
ns
Year
Published literature Grey literature
278
Appendix 19: Codebook to identify problems as defined by Theory of Change and Realist
Evaluation studies (2nd level).
Question /
code
Description Example
Complexity The food security issue, program, and/or system (in which a program is
situated in) is complex.
Food security
issue is
complex
Describes how food security
multidimensional problem with
multiple causes.
“Malnutrition is a multidimensional
problem that requires multisectoral
interventions” (Wesley et al. 2019).
Food security
program is
complex
Describes how pathways toward
food security are complex,
involving uncertainty, non-
linearity, emergence, multiple
interacting components,
multiple actors, multiple
feedback loops, and/or lengthy
time frames.
“… agricultural development is a complex
process with a “high degree of non-
linearity”, one that is “fundamentally a
social process in which people construct
solutions to their problems” by
opportunistically selecting only those
fragments of an innovation that meet
their particular needs” (CGIAR 2014).
Complex
systems
Describes how programs
intervene in complex systems,
contexts, and/or environments.
“Aquatic agricultural systems are
complex and evolving social-ecological
systems characterized by ongoing
change and unpredictability” (Apgar et
al. 2017).
Program
evaluation
Problems center around evaluation activities including the use of ToCs,
other theory-driven evaluation approaches, and/or experimental designs.
Problems also center around difficulties in evaluating certain food security
topics.
Limitations of
ToCs
ToCs miss detail and context;
ToCs do not account for
unexpected outcomes; ToCs do
not account for complexity.
“Program theory requires analyzing a
complex chain of causality, often not
adequately spelled out much less subject
to rigorous evaluation” (Cole et al. 2016).
Theory-driven
evaluations
Logic models oversimplify the
program, ignore complexity,
miss alternative pathways,
and/or present overly linear
pathways.
“… using only a logic model to articulate
a theory of change may result in
oversimplification of an intervention”
(Levay et al. 2018).
Experimental
designs
Many activities contribute to
food security outcomes making
it difficult to attribute change to
a specific program. Alternative
“Net effects, especially those related to
business performance and income, are
influenced by a wide range of
intervening factors… This makes it
279
approaches are needed. difficult to attribute effects to the actual
interventions and provides little
information on the effectiveness of
developmental activities” (Ton 2014).
Evaluating
food security
concepts
Describes evaluation challenges
specific to the food security
topic (e.g. nutrition, resilience,
capacity-building).
“Very little has been written so far on
impact evaluation of resilience
interventions” (Bene et al. 2020).
Program
planning
Problems center around the need of describing and/or explaining how the
program works.
Descriptive Need to describe and/or guide
the process of program planning
and/or learning.
“All research for development programs
wish to achieve impact, but
understanding how to plan for and
document this has been challenging.”
(Johnson et al. 2019).
Explanatory:
process
Describes the need to
understand and/or explain
program process and/or
mechanisms.
“… more evidence is needed to illustrate
the specific pathways by which elements
of process lead to the outcomes
specified—the how of social learning”
(Epp et al. 2019).
Explanatory:
outcomes
Describes the need to
understand and/or explain
program outcomes and/or
impact.
“Healthy Start is the UK government's
food voucher programme… It was
introduced in 2006, but the impact of the
programme on nutritional outcomes
remains understudied” (Ohly 2019).
280
Appendix 20: Codebook to identify how Theory of Change and Realist Evaluation studies are
responding to problems (3rd level).
Question /
code
Description Example
Purpose Describes the aim of the response to the problem.
Guiding the
program
Describes activities that guide
and/or understand change.
“To develop a ToC for improved nutrition
to understand the complex and varied
means through which nutritional
challenges can be addressed” (Wesley et
al. 2019).
Clarifying the
program
Describes activities that clarify a
program’s logic.
“The purpose of this paper is to
contribute a more nuanced
understanding of the theory of change
underlying this increasingly popular
public health intervention” (Levay et al.
2018).
Informing
evaluation
Describes activities that will
inform monitoring and
evaluation efforts.
“The theory of change devised in this
paper was articulated to form the basis
of a realist implementation evaluation of
the BC school food and beverage sales
Guidelines” (Levay et al. 2018).
Process Describes how responses were operationalized.
Researcher-led Describes how activities were led
by researchers.
“The proposed retrospective theory of
change was presented to various
stakeholders from the health sector to
ensure plausibility and their feedback
was incorporated where appropriate”
(Levay et al. 2018).
Participatory Describes how activities were
developed in a participatory
manner.
“Can start with a blank page or with a
straw impact pathway/theory of change
In a participatory manner, one can start
with a blank page and a facilitator who
proceeds to tease out the way
participants (stakeholders) imagine that
the intervention is to work and the
implicit assumptions behind their
thinking” (Mayne and Johnson 2015).
Prospective Describes how activities were
developed before/during the
“The project's theory of change was
developed with relevant stakeholders at
281
program implementation. the same time as the scenarios process
was undertaken, to help ensure high
relevance and the inclusion of the
partners likely to be necessary for
success” (Thornton et al. 2017).
Retrospective Describes how activities were
developed after the program
ended.
“IET developed without the use of an
articulated ToC. However, it was agreed
with the research project oversight group
that it would be valuable to undertake
the participatory development of a ToC
with local community members with
experience of and interest in IET” (Farrier
et al. 2019).
Outcome Describes the outcomes of
responses.
Single ToC Describes a single ToC. “Example of the theory-of-change for an
outpatient therapeutic feeding
programme” (Floate et al 2018).
Multiple ToCs Describes multiple ToCs (e.g.
one broad ToC and several
specific ToCs).
“In developing an approach to creating a
complexity-aware ToC, we also draw on
the idea that useful ToC should be
nested (Mayne, 2015) such that a
program or research system will have an
overarching ToC, describing its high-level
causal assumptions, under which more
detailed and grounded ToC is developed
for individual projects, or elements of
them” (Douthwaite and Hoffecker 2017).
If/then logic Logic reads: if the intervention
occurs successfully, then it will
lead to the desired result.
“After considering the diverse insights
from stakeholders, in conjunction with
the depiction of the programmatic
components in the logic model, broad
logical statements were developed to
theorize how the program was expected
to work (i.e. if-then statements)” (Levay
et al. 2018).
Backwards
mapping logic
Logic reads: for the change to
occur, these preconditions need
to be met.
“Using a backwards mapping approach
[27, 56, 57], positive actions, cognitive
processes, and behaviors needed to
achieve each of the priority behaviors
282
were mapped” (Arriola 2020).
Linear Describes uni-direction between
program activities and
outcomes.
“Our programme theory using the TOC
approach differs from the logical
framework by identifying additional
short- and medium-term outcomes
needed to create positive change”
(Floate et al. 2018).
Complex Describes emergence and/or
feedback loops between
program activities and
outcomes.
“These four parts should not be seen
only as a linear sequence of events.
There are important feedback and
cyclical elements within the ToC”
(Clarkson et al. 2018).
Factors
influencing
responses
Describes factors influencing responses.
Funder Describes the influence of
donors.
“The standard donor proposal format
includes both a logical framework and
development of a TOC. We decide to
incorporate a realist approach into one
of the domains identified through our
TOC, to provide a more grounded theory
on our capacity building activities”
(Floate et al. 2018).
Resources Describes the influence of
resources.
“Development of ToC prior to launch of
an intervention does not happen in
practice due to resource constraints,
necessitating development of a
retrospective ToC” (Levay et al. 2018).
283
Appendix 21: Interview guide and Consolidated Framework for Implementation Research a
constructs used to explore the implementation and evaluation of the SafePORK program.
Interview question (Construct) Researchers Partners
Intervention characteristics
What components were you involved in and what were your
experiences? (Design quality and packaging)
x x
Did these components change over time and how have teams
stayed resilient to program changes? (Adaptability)
x
Would you consider the program to be complex or non-
complex? (Complexity)
x
What opportunities exist to scale the program after the program
ends? (Perceived scalability)
x
Process of implementation
What has been challenging and helpful in implementing each
component? (Reflecting and evaluating)
x
Did the implementation go according to plan? (Executing) x
How do you assess progress toward implementation goals?
(Planning)
x
Outer setting
What factors contributed to changes to program components?
(External policy and incentives)
x x
Have extreme weather events influenced the program? (Climate
change)
x x
Inner setting
Can you describe how teams worked together? (Compatibility) x
Individual characteristics
Do you think the intended results will be fully achieved?
(Knowledge and beliefs about the intervention)
x
Outcomes
Over the past few years, how did your
knowledge/attitudes/practice change? (Outcome)
x x
What influenced these changes? (Context) x x
How did the program contribute to this change? (Mechanism) x x a The Consolidated Framework for Implementation Research (CFIR) consists of 39 constructs divided across five
overarching domains found to influence the successful implementation of innovative programs. The domains assess
intervention characteristics, outer setting, inner setting, characteristics of individuals, and implementation process.
CFIR was selected because the multilevel factors identified by CFIR provide a promising starting point for exploring
how SafePORK is implemented and evaluated.
284
Appendix 22: Audit trail of decisions made and rationales behind them.
Decisions Rationale
We selected relevant constructs of CFIR and
adapted questions to our program evaluation
context.
Suggested by the CFIR guide (cfirguide.org).
Where appropriate, interviewees were probed
to describe factors that facilitated or impeded
the implementation of the program.
Suggested by the CFIR guide (cfirguide.org).
We added questions as appropriate for our
program. For example, under the outer
setting, we added a construct for climate
change.
Climate change increasingly impacts food
safety. For example, changes in temperature,
rainfall patterns, and increased frequency of
extreme weather events can influence food
safety by changing the population dynamics
of contaminating organisms (Mbow et al.,
2019).a
We added a sixth domain (Outcomes). to ask questions around intermediate
program outcomes, helping to validate the
program’s Theory of Change (objective three
of SafePORK). Constructs for this domain
were informed by realist evaluation, which
focuses on the relationship between program
outcomes, mechanisms, and context (Pawson,
2013).b Tracing implementation processes is
an important part of realist evaluation
because interviewees’ reasonings around
program implementation can help identify
key contextual differences in the
development of outcome patterns (Manzano,
2016).c
We added the construct “perceived
scalability”, an additional CFIR construct
proposed previously (Means et al., 2020).d
Program coordinators were also interested in
perspectives on scaling up the program.
The final interview guide for researchers was
conducted with SafePORK coordinators. After
piloting, the tool was found to be too
lengthy, and was subsequently shortened
from 18 questions to 14 questions.
Pre-test the tools but also provide rich
contextual information.
285
Midway through the interviews with program
participants, research assistants added the
question “On a scale of 1 to 10, how would
you rate the program”?
To facilitate probing on challenges. For
example, if interviewees responded 9/10, we
would ask: how could it be 10/10?
a Mbow, C., Rosenzweig, C., Barioni, L. G., Benton, T. G., Herrero, M., Krishnapillai, M., Liwenga, E., Pradhan, P., Rivera-
Ferre, M. G., Sapkota, T., Tubiello, F. N., & Xu, Y. (2019). Food Security. In P. Z. P.R. Shukla, J. Skea, E. Calvo Buendia, V.
Masson-Delmotte, H.-O. Pörtner, D.C. Roberts, J. P. P. R. Slade, S. Connors, R. van Diemen, M. Ferrat, E. Haughey, S.
Luz, S. Neogi, M. Pathak, J. Petzold, & J. M. P. Vyas, E. Huntley, K. Kissick, M. Belkacemi (Eds.), Climate Change and
Land: an IPCC special report on climate change, desertification, land degradation, sustainable land management, food
security, and greenhouse gas fluxes in terrestrial ecosystems. b Pawson, R. (2013). The Science of Evaluation: A Realist Manifesto. SAGE. c Manzano, A. (2016). The craft of interviewing in realist evaluation. Evaluation. 22(3): 342-360. d Means, A. R., Kemp, C. G., Gwayi-Chore, M. C., Gimbel, S., Soi, C., Sherr, K., Wagenaar, B. H., Wasserheit, J. N., &
Weiner, B. J. (2020). Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use
in low- And middle-income countries: A systematic review. In Implementation Science. 15:17.
CFIR: Consolidated Framework for Implementation Research.
286
Appendix 23: Impacts of COVID-19 on data collection.
Initially, community visits for the qualitative study in Chapter 5 were planned toward the end of
March 2020 to coincide with the midterm review of the SafePORK program. However, COVID-19
cases in Vietnam surged, and the country suspended the entry of all foreigners on 22 March
2020 to limit the spread of the virus. While waiting to see how the COVID-19 pandemic would
evolve, I focused on the desk-based aspects of my research (e.g. Chapter 4).
By summer 2021, travel to Vietnam was still suspended. With the rapidly approaching
dissertation timelines, I considered whether research could be done at a distance. I wondered if
asking people to participate in research at this time – a year and a half into the pandemic – will
put them under any additional unnecessary stress. I consulted with my PhD committee, research
ethics board, and local partners, who provided support to conduct this study remotely. As such,
all interviews were conducted via Microsoft Teams or phone.
One initial concern I had was the limited control over the interviews with program participants,
which were conducted by the two SafePORK team members (Sinh and Trang). Because of time
zone differences between Canada and Vietnam, the limited existing rapport between myself and
program participants, and the anticipated challenges of myself being able to capture
Vietnamese speech fully over the phone, it made more sense for the SafePORK team to conduct
these interviews. To engage in this data collection process, I held sessions before, during, and
after interviews to plan for and reflect on the interviews. I also spent time listening to the audio
recordings and double-checking the Vietnamese transcripts.
287
In the end, this concern was perhaps not significant as both Sinh and Trang have extensive
experience conducting surveys and interviews. When I asked Sinh and Trang about their
thoughts on how remote data collection went, they said “it depends”. Some participants were
open to chatting as the time was convenient for them. Others had children, customers, or
livestock in the background, which at times created interruptions in the interview process.
Importantly, Sinh and Trang said phone interviews were fine because they knew the
interviewees. If new researchers were to join in the conversation, Sinh and Trang would have had
to explain in-depth who the new researcher is to avoid skepticism.
288
Appendix 24: Presentation material for the SafePORK team.
29 November 2021
2
Retailers and slaughterhouse workers reported they adopted hygienic practices because of
SafePORK interactions, training, and equipment, as well as changing consumer perspectives (e.g. increased demand for safer food)
While retailers and slaughterhouse workers could not sell pork at a higher price, many reported improved business from higher number of customers, which is a key motivator
Retailers and slaughterhouse workers reported consumers feel more secure buying pork from them, given improved cleanliness of facilities and pork quality
Microbial sampling will be important for confirming improved pork quality
Local authorities are supportive of SafePORK interactions, trainings, and equipment, even
requesting activities to scale up ( e.g. focus on centralized slaughtering facilities)
Discussions are underway in certifying model slaughterhouses and branding Ban pork
SafePORK is well respected by food safety
actors, with slaughterhouse workers and retailers requested increased visits from the
SafePORK team to raise awareness of consumers that their facilities are clean
Local authorities also appreciated SafePORKinterventions and would like to see them
scaled up ( e.g. to centralized slaughtering facilities)
Delegating tasks for joint activities
Offering strategies for effective
multidisciplinary collaboration, especially as the program is nearing completion and
teams will be publishing findings together
Ongoing development of program theory is a promising strategy for thinking about hard
to anticipate food safety outcomes. Measuring outcomes along a program s change pathway on a real time basis provides indicator s of gradual
progress toward the long term food safety goal as well as identifies (and addresses) emerging issues as they arise.
The team noted adoption of practices was quicker in urban areas than rural areas, likely
because consumers there are more aware of food safety concerns, suggesting improving consumer awareness is important for moti vating actors
to adopt hygienic practices. Attuning to context helps identity elements to consider when interventions are scaled up or adapted to other settings.
Women and men, for example, face different challenges related
to their work, requiring tailored interventions to their circumstances. They are also not homogenous groups; for example, the approach to working
with slaughtermen may be different in privately run vs family run slaughterhouses. Paying attention to mechanisms and context behind
behavioural outcomes is important for identifying what interventions work for different food safety actors .
While not an explicit objective of SafePORK, asking questions about climate change can reveal important insights into
how food safety is impacted by climate change and identify areas where SafePORK may potentially intervene.
289
3
Because food security is a multi dimensional issue, multi sector
programs are often required to achieve food security goals. Such
programs mobilize health, agriculture, and other related sectors to
implement a package of components ( e.g. research, intervention,
policy), which targeting different levels of change.
The multiple programming components can interact with each
another along with wider food systems. While necessary, the
complexity arising from these interactions creates challenges when
evaluating, learning from, and replicating multi sector programs.
Guidelines exist to support the development of complex health
interventions, but there are comparatively few guidelines relevant
for food security. To address this gap, this study explores insights
into food security programming. We draw on the experiences of the
SafePORK food safety program in Vietnam as a case study.
Semi structured interviews with SafePORK
researchers (n=7)
Semi structured interviews with slaughterhouse
workers (n=10), retailers (n=9), local authorities (n=3), and market managers (n=1) from Hung
Yen (n=13), Nghe An (n=7), and Hoa Binhprovinces (n=3) ( ).
The following points should be kept in mind when interpreting
findings:
The primary purpose of the study
was to generate lessons for program evaluation, rather than
evaluate the program itself. While the study also generated
evaluation findings, it does not replace SafePORK research
and evaluation activities ( e.g. KAP pre and post test survey,
microbial sampling, Outcome Mapping).
Given familiarity of the interviewer with participants, it was
difficult to probe for thoughts from slaughterhouse workers
and retailers on how the program might be done better.
However, local authorities and the market manager were
very open in sharing ideas for program improvement.
Examining SafePORK from an implementation science and realist
evaluation perspective ( ), we explored questions of:
1. What outcomes have SafePORK achieved so far, and what
mechanisms triggered such outcomes?
2. What are the challenges and opportunities for implementation?
Steven conducted semi structured interviews with researchers,
while Sinh and Trang conducted interviews with program
participants. Thematic analysis was used to analyze findings.
Regular de briefing sessions were held to co develop themes.
We drew on the following data sources:
4
Understanding the implementation of food safety programs in Vietnam requires an understanding of Vietnamese food systems . Foo d safety is
one the most pressing issues among Vietnamese, often given more emphasis than education. A particular area of concern is pork the main
animal source food in Vietnamese diets which is often produced by smallholder farming systems and sold fresh in traditional we t markets. While
highly nutritious, fresh pork sold in these markets are highly susceptible to microbiological contamination, presenting healt h r isks.
Market based approaches to improving
the safety of pork in Vietnam
( SafePORK ) is a five year program
(2017 2022) aiming to reduce the burden
of foodborne disease in informal,
emerging, and niche markets through
designing, piloting, and evaluating food
safety interventions. SafePORK is
implemented in Hung Yen, Hoa Binh, and
Nghe An. In Hung Yen and Nghe An,
SafePORK is primarily focused on pork
handling practices within traditional
slaughterhouses and wet markets, two
critical points in the chain in which food
safety issues can have large
consequences. In Hoa Binh province,
SafePORK is strengthening the Ban pig
value chain by linking rural women and
men smallholder farmers to emerging
and niche markets. shows the
five core objectives of SafePORK and their
interrelations.
290
5
Below is an overview of key activities and timelines of SafePORK alongside relevant external factors ( ). The context of ASF and COVID 19
is important for understanding how and why implementation shifted over time. At the time of the study (Aug Sept 2021), Vietnam was under
lockdown due to COVID 19. Despite delays created by COVID 19 (and ASF), nearly all researchers perceive SafePORK to be implement ed as
planned. Researchers reported reaching eight slaughterhouses (out of a target of 10) and 14 retailers (out of 24) in Hung Yen and Nghe An.
Regarding Ban value chain development in Hoa Binh, the team is working with local authorities and markets to build and promote branding and
market linkages. Risk communication is well under way, with hundreds of actors trained in food safety risk management. Researche rs believe they
will achieve their original objectives by the end of 2022.
Safe
PORK inte
rventions
Exte
rnal fa
cto
rs
6
291
7
When asked about the changes resulting from SafePORK, researchers
reported:
Researchers described how multidisciplinary collaborations
led to knowledge exchange. For example, economic researchers
learned about risk assessments whereas public health teams learned
from others about how to effectively engage with communities.
Researchers described how collaborations
created outputs or led to ideas that were stronger than if team
members worked within their own disciplines. Some reported if the
other team was not involved, they would not have been able to
implement activities and/or interventions.
SafePORK is fostering the development of new
relationships and partnerships. Researchers reported engaging with
different disciplines, often for the first time. Outside of regular team
meetings, SafePORK researchers found it valuable to connect on Zalo
and Facebook to provide real time updates on SafePORK and life.
Researchers described how
SafePORK activities led to changes outside the immediate program
team. For example, senior researchers described using SafePORK as a
case study in lectures, helping to equip the future workforce with
knowledge. Others described progress made in certifying model
small scale slaughterhouses or markets, thus supporting informal pork
production. In Hoa Binh, discussions are being facilitated to promote
branding and market linkages for Ban pork, as well as support
hygienic practices at slaughterhouses and markets.
Because when we discuss on an issue with other sector, they
bring with me more chance to rethink what I thought before,
bring for me more ideas. I also have more potential partners.
Our team members most of them are lecturers in their
discipline. SafePORK give them some idea and also some
chance to upgrade to know to improve the lecture.
And for doing training for the slaughterhouse, without the
HUPH team, we couldn't do this because they have the
experience and knowledge on this.
Now I learned a lot from the VNUA team and NIAS team about
how to work with local people. And I think if we can run
SafePORK again or scale up I think we can do other things to
approach people.
8
When asked about the ways in which their food safety
thinking / practice has changed, participants reported:
Shared outcomes among women and
men included the adoption of hygienic practices, resulting
in improved cleanliness of the working area.
Women described how meat is now
cleaner, more beautiful, and has less odor than before,
enabling them to sell more pork. One retailer in Hung Yen
shared she used to sell around sold 20 kg of pork per day,
but now she can sell 25 30 kg. Men also noticed their
business was better, as one slaughterhouse worker in Nghe
An explained, because customers see [slaughtering
practices] directly, it s clean, they like it, and they introduce
us other customers who buy more pork .
Both women and men
reported that consumers trust them more because of
improved cleanliness of facilities and quality of pork.
According to them, consumers feel more secure when they
buy and eat their pork.
SafePORK interactions have led to
structural changes. For example, discussions with market
managers have led to improved access to water at markets,
supporting retailers in keeping counters clean.
Both women and men said they know more
now and recalled key messages from the training sessions.
Importantly, change in knowledge was often not enough to lead to
change in practice. For many, mechanisms were related to achieved
outcomes ( e.g. changed practice because pork looked better).
, as one
explained: if I don t, I will lose customers. And while they often
cannot sell pork at a higher price, many noticed increased number of
customers, a motivator for continuing to apply hygienic practices.
For
men, convenience of proposed recommendations was highly valued.
They appreciated how equipment ( e.g. inox grid, boots) helped keep
the floor cleaned and relieved them of not worrying about
borrowing. For women, having access to water at the market was
important, which was not available before SafePORK: now that there's
water I'll wash my hands, wash the cutting board and clean seats, then
sell [pork] without the smell anymore.
292
9
Better understanding of hygienic practices and how to communicate with food safety actors
Interactions with the team, studying project documents, and attending trainings contributed to these outcomes
Low level of knowledge of consumers about safe pork (especially in the highlands)
Low consumer willingness to pay higher prices for safer pork
Low level of knowledge of retailers/slaughterhouse workers about safe food handling practices
Limited funding for staff members to travel and carry out inspections
Climate change increases risk of pigs getting disease; farmers will use chemicals ( e.g. veterinary drugs, animal feed) to prevent pigs from getting sick,
which increases costs and affects food safety
More communication especially for consumers who have a large influence on other actors
Consumers should be encouraged to limit purchases from counters with limited hygiene
Training for producers to raise livestock that avoids risks from disease and price fluctuations
Expand the project in other communes to achieve larger impacts, as well as continuing trainings for consumers, retailers, and slaughterhouse workers
More visits from the SafePORK team to encourage maintenance of practices
Expand training to centralized slaughtering facilities to align with overall policy direction
Start training for veterinary staff around food safety management / disease detection
Traceability; cannot control quality of pork coming to the market from different sources
Hard to tell retailers what to do because it will bring him displeasure
Still notices hygienic issues ( e.g. cloths aren t washed, cutting boards aren t clean, bad smell); convenience of old habits is a barrier
Practices depend on the individual retailer (some are clean while others are messy)
Consumers don t notice change in practices because they are busy and buy meat fast
Lack of inspection being done, only he does it (not veterinary departments or authorities)
Climate change and hot weather increases appearance of flies
Would like to see more visits from the SafePORK team to inspect; visits should be sudden and unannounced
Consider adding a camera and inspecting food safety at the market from a distance
10
Finding time to get together is difficult. Often it seems the same
conversations are being held without any progress being made.
If key team members cannot make it to meetings, encourage
representative members to debrief with their team
Encourage young researchers to join these meetings as they
bring an important perspective and have more time
Share field visits in advance, while recognizing not all activities
can be prepared early
Joint activities without a clear lead/support leads to confusion on
who will do these activities. Also, after brainstorming next steps
during meetings, often next steps are not followed.
For joint activities, delegate tasks clearly, even if they are joint
Follow through on action items by tracking progress on their
achievement
Actors adopt hygienic practices when researchers come, but
quickly revert to old practices when researchers leave.
Continue regular visits to study sites as consumers think visits
from SafePORK team means the slaughterhouse/market is clean
As consumers increasingly recognize the value of safe food
handling, retailers and slaughterhouse workers have to adopt
practices
Adopting hygienic practices is challenging because they do not
yet provide incentives such as higher profit, certification, or
branding.
Explain how such practices improves the quality of pork and
facility cleanliness, which can attract customers
Explain that other participants adopting these practices have
seen increase in of customers
Participants say they have no time, but what they really mean
may be that they the need to keep pork dry (hence cannot clean
cutting boards often), there is too many customers (hence
cannot use disinfection), or need to process pigs quickly (thus
cannot clean knives).
Seek to understand what participants mean when they say they
have no time
Continue designing interventions that are convenient; continue
using nudges; continue seeking feedback regularly
Continue fostering an enabling environment (e.g. ensuring
access to clean water)
Local authorities prioritize investment projects rather than light
touch interventions.
Continue communicating to local authorities about the value of
building capacity and an enabling environment
Actors don t want to change because they don t know where
pork is coming from and where it is going. They also consider
food safety to be the responsibility of others ( e.g.
slaughterhouse workers think of retailers, and vice versa).
Select value chains with clear linkages
If not possible, continue communicating challenges with
achieving traceability and advocating for traceability
293
11
If SafePORK could be done differently:
During this activity,
key champions were identified who were supportive of SafePORK.
Risk communication can also help improve consumer awareness,
helping to improve uptake of interventions in markets and
slaughterhouses.
Preliminary findings
can be used as a risk communication tool to explain risks and
opportunities for mitigation. Findings should be presented to all
study communities. Presenting raw data is an opportunity to
collect more insights and strengthen survey findings..
Actors are likely more motivated to participate if they have a
financial stake.
Suggestions moving forward:
Consider a workshop at the beginning of collaboration with
researchers. Each researcher could share how they would address
the food safety problem using tools/approaches from their
discipline. Don t assume people have prior knowledge about
others disciplines. Provide guidance on how to work together.
Researchers suggested activities for the next phase
including adding food classes (for consumers) and developing
traceability of pork. While SafePORK pivoted activities well during
ASF and COVID 19, consider developing tools that protect food
safety and mitigate the impact of external events. For example,
researchers suggested developing an early warning system.
Engage with local authorities
earlier (e.g. local authorities, animal health Women s Union).
Budget for engagement. Continue regular visits to study sites,
which help actors maintain practice and improve consumer
awareness. Continue finding champions who will work with their
peers toward positive social change (e.g. Mr. An slaughterhouse).
12
294
13
When asked whether climate was a factor influencing
implementation, many researchers talked about the link
between heatwaves and increased microbial replication.
For slaughterhouse workers and retailers, transporting pork
can be challenging during heavy rainfalls.
Retailers talking about how heat and rain impacted pork
quality by making your meat worse, it is not fresh and
beautiful anymore
Because heat makes pork wet, retailers often use cardboard
to help absorb moisture
One local authority reported that farmers pigs would get
sick during climate variation and would often use
unnecessary substances to support pigs.
For women we interviewed, time is a scarce resource.
Retailers described cleaning only when there are no
customers: I cut it for this person first, and when no one has
come to buy, I bend down to wash quickly before someone
buys and I cut it again. Maintaining cleanliness is important
for retailers in keeping customers, not just their individual
retailer counters, but the entire market.
Women more often than men reported challenges with
climatic events ( e.g. heat reduces pork quality).
Men worried more about sourcing pigs not affected by ASF
and collecting pigs when is not too hot/cold out.
Continuously monitoring and adapting interventions in
response to implementation feedback and new emerging
conditions is critical for achieving behavioural change.
Ongoing development of program theory is a promising
strategy for thinking about potential outcomes
Measuring outcomes along a program s change pathway
provides indicators of gradual progress toward the long
term food safety goal, as well as identifies emerging issues
and opportunities for intervention adaptation
Because program components vary in complexity, different
tools are required to to assess and learn from components.
SafePORK interventions were strongly influenced by context.
The team noted adoption of practices was quicker in urban
areas than rural areas, likely because consumers there are
more aware of food safety concerns, suggesting improving
consumer awareness is important for motivating actors to
adopt hygienic practices.
Scale of pork production operations also influence
interventions; for example, more pigs are processed in Hung
Yen compared to Nghe An, often calling for slaughterhouse
interventions to be more sophisticated ( e.g. installing 2 grids
vs one grid in Hung Yen).
Attuning to context provides ideas for intervention design.
14
Were any of these findings surprising?
Do the four developed themes make sense? Should they be changed?
How can these findings be helpful for other food security researchers
and practitioners? In other words, what are the implications for food
safety program planning and evaluation?
295
15
Interview guide used to explore the implementation and evaluation of the SafePORK program.
Questions were informed by:1) , an implementation science framework used to explore factors considered important for intervention success.
2) , an approach to evaluation used to explore how, why, and under what context program outcomes were achieved.
16
What components were you involved in and what were your
experiences? (Design quality and packaging)
x x
Did these components change over time and how have teams stayed
resilient to program changes? (Adaptability)
x
Would you consider the program to be complex or non complex?
(Complexity)
x
What opportunities exist to scale the program after the program
ends? (Perceived scalability)
x
What has been challenging and helpful in implementing each
component? (Reflecting and evaluating)
x
Did the implementation go according to plan? (Executing) x
How do you assess progress toward implementation goals?
(Planning)
x
What factors contributed to changes to program components?
(External policy and incentives)
x x
Have extreme weather events influenced the program? (Climate
change)
x x
Can you describe how teams worked together? (Compatibility) x
Do you think the intended results will be fully achieved? (Knowledge
and beliefs about the intervention)
x
Over the past few years, how did your knowledge/attitudes/practice
change? (Outcome)
x x
What influenced these changes? (Context) x x
How did the program contribute to this change? (Mechanism) x x
296
17
297
Appendix 25: Characteristics of program participant interviewees.
Province Role Gender Ethnic group
Hoa Binh Local authority Man Minority
Hoa Binh Slaughterhouse worker Man Minority
Hoa Binh Slaughterhouse worker Man Majority
Hung Yen Slaughterhouse worker Man Majority
Hung Yen Slaughterhouse worker Man Majority
Hung Yen Slaughterhouse worker Man Majority
Hung Yen Slaughterhouse worker Man Majority
Hung Yen Retailer Woman Majority
Hung Yen Retailer Woman Majority
Hung Yen Retailer Woman Majority
Hung Yen Retailer Woman Majority
Hung Yen Retailer Woman Majority
Hung Yen Retailer Woman Majority
Hung Yen Market manager Man Majority
Hung Yen Retailer Man Majority
Hung Yen Local authority Man Majority
Nghe An Slaughterhouse worker Man Majority
Nghe An Slaughterhouse worker Man Majority
Nghe An Slaughterhouse worker Man Majority
Nghe An Slaughterhouse worker Man Majority
Nghe An Local authority Woman Majority
Nghe An Retailer Man Majority
Nghe An Retailer Woman Majority
298
Appendix 26: Codebook to identify overarching themes from SafePORK implementation.
Code Description Supporting quotation
Developing and using program theory: describes the use of program theory in tracking
changes to program context, strategies, and outcomes.
ASF impacts Describes impacts on ASF
on SafePORK activities
(e.g. inability to access the
field, slaughterhouse
workers more concerned
about sourcing healthy
pigs)
the district and also province did allow people
going in and out much in the slaughterhouse,
disrupting activities. There is a need to restrict
/ manage the number of people
Changing pig
production and
consumption
patterns
Describes changes to pig
deaths, consumer
demand, pork production,
and pork price
In Hung Yen during the swine fever time
because the price of pigs and price of pork
increase very significant. So, farmers try to raise
heavier pigs.
Responding to
ASF
Describes SafePORK
strategies to respond to
external factors (e.g.
practicing caution in
entering field sites,
shifting focus to other
actors along the pork
value chain)
you know, the ASF occur very strongly in
Vietnam, and it really affects our activity a lot,
because at that time we visit the SL, and from
SL to other SL, we can be the vector to bring
the virus to the pigs, so which mean we must
be very careful in our work
COVID-19
impacts
Describes changes
brought on by COVID-19,
such as inability to go to
the field, switching from
remote communications,
busyness of local
authorities in COVID-19
control
As long as COVID-19 is here, and we do not
control that, it is very difficult to reinforce what
we have done earlier in both Hung Yen and
Nghe An. Because you know the
slaughterhouse and retailer when you are in
contact with them it must be continuous
Push to
modernize
pork
production
Describes the
government’s push toward
large and centralized pork
production in order to
more carefully control
food safety issues
want to maintain and increase the influence in
centralized slaughtering facilities and raise
awareness for retailers and consumers.
Currently, interventions are only in small
slaughterhouses, while the district's and
province's orientation are toward concentrated
slaughtering points
299
Responding to
COVID-19
Describes responses to
COVID-19, particularly
leveraging existing
relationships and
maintaining contact with
food safety actors
remotely
However, because of COVID so we can
sometimes we couldn't go there, and we make
the phone call only… we would like them to
take some pics or videos of implementation
but some couldn’t because they don't have a
smartphone
Facility size Larger facilities tended to
have more difficulty with
adhering to
recommended practices
compared to smaller
facilities
you can imagine from 2 am to five am. They
have few hours to slaughter seven or eight
pigs. So they have to do it quick. Not like in
Nghe An they sometimes only have 1 pig.
Ownership
structure
Different approaches are
required to engage with
community members; for
example, for family-run
slaughterhouses, need to
work with the entire
family, whereas for owner-
run slaughterhouses,
primarily need to engage
with the owner who will
then delegate tasks to
workers
we have implemented the intervention in two
types of slaughterhouses. The first one is a very
private slaughterhouse which means that the
owner, they are not the worker, which means
that the owner, they just hire the workers. But
for other smaller scale SL, the owners are also
the workers, and they just use the labor from
their family. The way they operate is very
different. We also need to change the way we
approach them.
Government
priorities
Governments (district,
provincial, federal) now
prioritizing modern pork
production facilities
Slaughterhouses at the large scale can control
the food safety and hygiene. In smallholders
the government cannot control because they
don't have enough people to check those. So,
the trend of household-level slaughterhouse is
disappearing, gradually.
Private sector
priorities
Private sector are more
concerned about
economic profit over food
safety
the private sector they are more focused on
profit. They not always concern of improving
the facility. So that is what we see in Nghe An
and one market in Hung Yen as well
300
Location Rural areas are more
challenging to work in
compared to urban areas
due to lower consumer
awareness of food safety
if you can do that [have safe practices] in the
countryside, it's good already
Attuning to climate change: describes how climate change is impacting food safety,
how program participants are responding to climate change, and how SafePORK is
supporting climate change adaptation and environmental management.
Climate
impacts
Describes the impacts of
climate change, including
heat waves (making pigs
sick, encouraging
microbial replication in
pork products), heavy
rainfall (making
transportation of pork
challenging,
contaminating water,
delaying field visits), and
how climate change
impacts vary by region
(more-so in Hoa Binh and
Nghe An compared to
Hanoi and Hung Yen)
Climate change right now is a big problem. So
for example in Nghe An in the summer we
have to some days that are too hot. To keep
and maintain the quality of pork it is also hard
for them. Retailer do not have refrigerator to
keep the meat, meat is put on the table
Adaptation
strategies from
SafePORK
Perceptions on how
SafePORK activities and
recommendations directly
or indirectly helps
participants adapt to
climate change
So some small wet market also they operate
during the afternoon, between 4-7 pm. But
most of wet market operate early in the
morning. We encourage consumers to buy the
pork early morning and put in refrigerator. So
that in the evening they keep for cooking
Adaptation
strategies from
participants
Perceptions on how
participants are adapting
to climate change, such as
covering pork or using
water sprays to cool pigs
Sometimes if they have heavy or weather
difference in summer will have effect on pork
as well. And sellers, they have their own thing
of the cover by nylon or they put under cover
or box
Environmental
management
Describes food safety
actors’ concerns about the
environment (e.g. treating
When we do with slaughter or farm (with
PigRISK) do they have enough facility to treat
their waste to not affect sellers and community
301
waste and wastewater),
and SafePORK’s work
toward environmental
management (e.g.
discussions with market
managers and local
authorities)
around. Yeah. And recently we also talk with
local authority about how to keep the
environment of the market clean, the waste as
a market, clean every day, and how to treat
waste
Attuning to equity: Describes how SafePORK interventions are tailored to different
groups of people, what outcomes are achieved, and the context and mechanisms behind
the achievement of outcomes.
Tailoring
interventions
Describes intentional
design considerations for
different groups of people
we have some video for the retailer, but we are
thinking of making two different videos. One
by men retailer, one by women
Context Describes contextual
factors influencing the
adoption of safe practices,
including consumer
awareness and enabling
environment (e.g. water at
market)
Since the project has been discussed with the
market and commune management boards,
electricity and water have been provided.
Mechanisms Describes reasoning
behind adoption of
practices, including
changes in food safety
knowledge, concern for
food safety, consumer
perspectives, convenience
of practices, and provision
of training and equipment
I feel that not maintaining good hygiene will
affect the meat, and secondly, I’m also worried
about consumers' health when we clean and
provide better products, consumers will turn to
me more
Outcomes Described how SafePORK
research led to new
knowledge on food safety
or how outputs generated
were stronger together
compared to individually;
described changes to
enabling environment (e.g.
educating future
workforce, improvements
to slaughterhouse and
From my side, brought a lot of added value,
when working in different team or sector, I
learned to get more new ideas. Because when
we discuss on an issue with other sector, they
bring with me more chance to rethink what I
thought before, bring for me more ideas. I also
have more potential partners
302
market facilities,
promoting interest in Ban
pigs); described how
SafePORK led to changes
in food safety practice;
described how SafePORK
led to changes in
relationships
