Version 1.0, March 2015 Page 1 of 8

[7] DKH DRR Evaluation GuidelinesContentsIntroduction....................................................................................................................................................................1

Evaluation Approach Considerations...........................................................................................................................1

Different Types of Evaluation..........................................................................................................................................2

Guidance on Conducting each Type of Evaluation..........................................................................................................2

1. DRR Outputs & Outcomes Evaluation Considerations.........................................................................................2

Component 1: DAC Criteria DRR Evaluation Components.......................................................................................2

Component 2: Utilising Baselines & Project Monitoring Systems...........................................................................3

Component 3: Core Community-Based Disaster Risk Management (CBDRM) Components...................................4

Component 4: Hard Component Preparedness & Mitigation Measures.................................................................5

2. Impact (long-term)..............................................................................................................................................5

Component 5: HFA Resilient Community Methodology..........................................................................................5

Component 6: Post Hazard Event Evaluations.........................................................................................................6

3. Project Objectives / Proposed Outcomes............................................................................................................6

4. DRR Programme Management Evaluation..........................................................................................................6

Annex 1: Potential Community Resilience Outcomes.....................................................................................................6

Characteristics of a Disaster-Resilient Community......................................................................................................6

Priority Resilient Community Characteristics..............................................................................................................7

Characteristics of a Disaster-Resilient Community: slow-onset hazard, food security perspective.............................8

IntroductionThese guidelines outline types of evaluation methodologies that can be conducted to evaluate DRR projects and programmes. This guide can either be used to assist you in developing a terms of reference (TOR) for a DRR project evaluation, or to help plan your own (partner) internal project evaluation. The evaluation components described below can either be used individually or several can be combined and used within one evaluation. The type of DRR evaluation to be conducted, and components are selected according to what the evaluation needs to inform, either:

Planning of further DRR project DRR strategy development DRR communications, advocacy or publications Institutional DRR programme management knowledge, skills and procedures

Evaluation Approach ConsiderationsMany DRR project evaluations use a variety of data collection methods. A more participatory approach would include: literature review, preparatory site visit, use of semi-structured and structured questionnaires, structured and informal discussions, field observation, group discussions including workshops, informal conversations with stakeholders, and feedback sessions with local partners and project steering committees. In participatory projects it is crucial that communities are involved in evaluation and are empowered to make appropriate decisions about future activities as a result.1

1 Adapted from: Charlotte Benson and John Twigg, Measuring Mitigation. Methodologies for assessing natural hazard risks and benefits of mitigation - A scoping study (Provention Consortium, 2004).

Version 1.0, March 2015 Page 2 of 8

Different Types of EvaluationThe following table presents the four main types of evaluation that could be conducted for DRR programmes and projects. In most cases the first option will be applied.

Type of Evaluation What is Evaluated? When to Use each Type of Evaluation1. Outputs (of activities) and Outcomes (initial result level & specific objective impact)

Evaluates the achievement of

intended project activity outputs and

outcomes

Can be used for evaluations being conducted a few months / a year after project completion.

Or, conducted as a mid-term evaluation/review will enable the identification of remedial actions to be taken within the lifetime of the project.

2. Impact Evaluates the longer-term impact of the

project, i.e. achievement of the

overall project objective and other

unintended outcomes.

Can be conducted when a project has been completed for a period long enough to identify impacts could potentially be identified, i.e. a year or longer, dependent upon the interventions.

Or, can be conducted when previous DRR project target areas have been impacted by a significant hazard event in order to evaluate the effectiveness of the interventions.

3. Project Objectives / proposed outcomes

Evaluates the logic of the DRR project

design and suitability of proposed

interventions.

Conducted after a risk assessment has been conducted and project interventions have either been: completed, initiated or planned in detail.

4. Programme Management

Evaluates how the project/programme has been managed

Can be used when DKH has a large DRR program in a region or a country running for a number of years.

Provides learning on DRR programmes to inform either: Institutional DRR PCM, procedures and capacity building needs The planning and setup of future large scale DRR programmes in

other regions/countries

Guidance on Conducting each Type of Evaluation1. DRR Outputs & Outcomes Evaluation ConsiderationsComponent 1: DAC Criteria DRR Evaluation ComponentsThis table provides a set of potential DRR focused DAC criteria points. The points should be reviewed to select those which are most relevant to be investigated in the evaluation. All points will not be relevant, if for example it is know that there is no local authority DRR strategy or standardised training materials being applied within a country these points cannot be assessed.

DAC Criteria Potential Points to Analyse for each DAC CriteriaRelevance: Community DRR Action Plans assessed against Risk Assessment process, findings and analysis

o Prioritisation of hazards/hazard impacts Alignment of project with:

o Local authority DRR & CCA strategieso Standard national community/school DRR training materials

Change in DRR context during project period Proposal/donor flexibility and appropriateness of remedial actions taken based on on-going

community review of activities and context

Version 1.0, March 2015 Page 3 of 8

DAC Criteria Potential Points to Analyse for each DAC CriteriaEfficiency: Whether a partner DRR capacity building needs assessment conducted and appropriate

capacity building training and resource needs identified and implemented efficiently. This can also include institutional DRR mainstreaming

Efficiency of established community monitoring and feedback mechanisms and timeliness of responses by the project management board/implementing partner/donor

Level of community-contribution and extent of community self-action/volunteerism achieved (includes community DRR Committees)

Assessment of whether the DRR interventions implemented are in line with good practice DRR approaches (in terms of efficiency in approach to achieve intended outputs/outcomes)

Effectiveness: Assessment of whether the interventions implemented are in line with DRR good practice approaches (in terms of their effectiveness considering the project context)

Identification or development and application of appropriate risk assessment and DRR project training materials, activity standard operating procedures, specialised activity manuals

Clarity and alignment of interventions/project implementation with intended DRR outcomesImpact: Assessment of project impact against established DRR baseline/reconstructed DRR baseline

Effect of the activities on most vulnerable individuals/families within the communities Contribution of the project outcomes towards building community resilience (ranking

exercise against residence characteristics) Assessment of whether the implemented DRR interventions are in line with good practice

approaches (in terms of the outcomes achieved / level of impact) Appraisal of project interventions (e.g. post-project DRR cost-benefit analysis) of

interventions in order to compare the extent beneficiaries have benefitted from the interventions, as compared with other DRR intervention options

Extent of unintended negative or positive DRR impacts and associated gaps in planningSustainability: (includes demonstration of scale-up and replication feasibility)

Level of proactive engagement of relevant stakeholders in relevant project activities Adoption and integration of project approaches by communities and other stakeholders:

o Stakeholders’ (including communities) self-initiated replication of interventionso Ownership and integration of project components (interventions, plans, monitoring

systems) into stakeholders’ structures and mechanisms [demonstrating increased efficiency and cost-effectiveness]

o Stakeholders’ (including communities’) contribution to replication and establishment of facilities/resources and services associated with intended project outcomes

o Process applied in the development, adoption and integration of project monitoring indicators and formats into Local Authority Disaster Management system. Examples include Plans, Disaster Readiness Checklist and Needs Assessment formats

Identification of the project interventions that other communities and communes are prepared to initiate and implement with their own available resources with the potential reduction in resource inputs required from external sources

Coverage: Extent of community engagement achieved Extent of targeting and engagement of vulnerable and marginalised groups Extent of engagement of key project stakeholders (DRM Actors etc.)

Component 2: Utilising Baselines & Project Monitoring SystemsThe risk assessment and vulnerability/capacity analysis should provide good baseline data. Some of the same tools used to construct the DRR project baseline can be used again to draw meaningful conclusions about outcomes and impact.2 If an adequate project baseline was not established this will need to be ‘reconstructed’ in order to establish the extent of change that can potentially be attributed to the project. Several tools utilised in DRR project monitoring (Refer to DKH DRR PCM Tool 6: Monitoring & Baseline Guidelines) can also be used in project evaluations:

Preparedness (Readiness) Checklists Committee Development Indexes Community DRR Action Plans – measuring progress (achievement of agreed milestones) Community Resilience Ranking (See Component 5 below)

2 Charlotte Benson and John Twigg, Measuring Mitigation. Methodologies for assessing natural hazard risks and benefits of mitigation - A scoping study (Provention Consortium, 2004).

Version 1.0, March 2015 Page 4 of 8

Component 3: Core Community-Based Disaster Risk Management (CBDRM) ComponentsThis type of evaluation is useful to conduct with a group of partner organisations who are still developing their knowledge and skills in implementing CBDRM projects, understanding the process and outcomes. The approach can also be adapted to run with one partner. This methodology can be applied to community-based projects where the main project components include developing community capacity for: conducting risk assessments, developing community plans, developing committees and conducting self-implementation and monitoring of activities. The strength of this methodology is its use for guiding stakeholders at different levels; in understanding the effectiveness of their actions, to identify challenges and gaps in implementation, to enable stakeholders to identify their own recommendations, and for stakeholders to own the evaluation findings and identified action points.

To facilitate this evaluation, it is important that the project (proposal) logic has been appraised by the evaluator/facilitator and that any potential weaknesses or gaps in the proposed CBDRM process and intended intervention outputs, outcomes and short-term impacts (within the proposal) are identified in advance.

A clear ‘CBDRM model’ (see below Step 2: Common Page Workshop) needs to be developed with the project stakeholders to establish agreement on the intended project outcomes. The ‘CBDRM model’ provides the assessment criteria for the evaluation. The type of ‘CBDRM model’ developed will depend upon the type of community-based project implemented:

Shorter-term community disaster preparedness (CBDP) Medium-term community-based DRR (CBDRR) Long-term community-led/based DRM (CBDRM) Or, extended community climate change adaptation (CCA)

# Key Evaluation Methodology Steps Materials1 Activity: Development of sample community case stories.

Purpose: Provides details on the process (steps) applied by an organisation in their implementation of a CBDRM project, the support provided to enable community facilitators to do their work most effectively and the challenges, recommendations and lessons learnt at different levels in the organisation while implementing the project.

2 Activity: Project stakeholders ‘CBDRM model’ Common Page Workshop.Purpose: Provides the definition of the minimum process, outputs and outcomes that should be followed and achieved in the project, the ‘CBDRM Model’. This summary is then used to evaluate the partner’s approach and progress made by communities. Initial challenges/gaps, lessons learned and recommendations are gathered.

3 Activity: Participatory community (VDMC) focus group discussions (FGD) to assess: Community mobilisation/engagement process Effects of interventions on individuals and community Intervention challenges/gaps Intervention recommendations

Purpose: To identify with communities the effectiveness of the interventions, the level of community ownership and what they need to do themselves and what support they still require to establish an effective CBDRM system.

4 Activity: Review of the community FGD findings with the partner’s project team.Purpose: To verify the findings from the community FGD, to gather further information on the process applied and to facilitate the team’s self-reflection of their community approach.

5 Activity: Development of community critical case studies.Purpose: To develop detailed evaluation feedback on the projects achievement of ‘CBDRM model’ minimums and associated DAC criteria.

6 Activity: Review with partner(s) of the critical case studies compared with the ‘CBDRM model’ and review of partner(s) institutional arrangements and changes required to establish a more enabling institutional environment for CBDRM projects. Purpose: To develop the final evaluation findings and recommended way forward and to develop partners understanding and ownership of these findings.

Version 1.0, March 2015 Page 5 of 8

Component 4: Hard Component Preparedness & Mitigation MeasuresThis methodology could be applied to any project which incorporates hard component risk or vulnerability mitigation measures at the community level.

Key Community Level Intervention Evaluation Methodology Steps Materials Review of final project activity reports and relevant community maps DRR Committee representatives requested to lead a guided tour of mitigation measures

and requested to explain:o Positive and negative outcomes of the interventionso Recommendations for changes to the activitieso Whether activities are planned to continue in the future and if so how

Detailed physical observation and verification of mitigation measures

This methodology could be applied to any project which incorporates hard component risk or vulnerability mitigation measures at the household level. Recommended key evaluation methodology steps:

Key Household Level Intervention Evaluation Methodology Steps Materials Review of final project activity reports FGD with beneficiaries/household recipients of mitigation interventions and requested to

explain:o Positive and negative outcomes of the interventions (improvement and

deterioration in their situation)o Recommendations for changes to the activitieso Whether activities are planned to continue in the future and if so how

Visits to activities / mitigation measures for physical observation and verification Hold DRR Committee meeting to ask:

o Should this activity be offered to others in the community? o If so, are the committee supporting/promoting the replication of the intervention

by other community members?

2. Impact (long-term)Component 5: HFA Resilient Community MethodologyThis methodology could also be applied to evaluate shorter-term outcomes but is generally more applicable to longer-term projects and post-project impact evaluations. Refer to DKH DRR PCM Toolkit, Tool 6: Monitoring & Baseline Guidelines - Option B: Resilient Communities Qualitative Baseline. Even if an initial community resilience ranking baseline was not constructed during the establishment of the project this can be reconstructed from available data and by using participatory approaches. The ranking levels can be established by referring to Annex 1: Potential Community Resilience Outcomes below.

Component 6: Post Hazard Event EvaluationsIn these cases evaluations are designed in order to determine the actual contribution that interventions made in reducing hazard impacts and in improving beneficiaries’ and communities’ responses and recovery capacities for coping with the shocks they experienced.

For this type of evaluation specific questions can be developed, dependent upon the types of interventions previously implemented, in order to determine their impact or to develop a comparative scenario of what would potentially have happened without the DRR interventions. Levels of attributable cause and effect will also need to be determined dependent upon a range of contextual factors.

3. Project Objectives / Proposed OutcomesRecommended key evaluation methodology steps for assessing a DRR project’s stated objective and proposed outcomes:

Desk review of the project proposal: review of the logic within the proposal and logical framework using relevant sections of DKH DRR PCM Tool 5: Proposal Appraisal Criteria.

Field visit evaluationo Review of risk assessment methodology applied and analysis of risk assessment findings

Version 1.0, March 2015 Page 6 of 8

o Review of project logic (planned activities) against any gaps identified in the risk assessment processo Review of intended outcomes against observed outcomes

4. DRR Programme Management EvaluationKey evaluation methodology components to be considered when designing a DRR programme management evaluation include:

Initial desk review of DKH regional/country DRR strategy Participatory multi-level (involving different stakeholders) organisational DRR program review:

o Review of the DRR programming and application of DKH DRR PCM Tools Programme design/strategy Partner identification/selection Proposal Appraisal Monitoring System & Complaints Mechanisms Partner capacity building (resources and collaboration networks) Technical backstopping of partners Reporting mechanisms DRR Integration / Mainstreaming into wider programming

o Organisational DRR program structure (staffing structure)o Over-arching program management

Annex 1: Potential Community Resilience OutcomesCharacteristics of a Disaster-Resilient CommunityMany approaches have been taken by organisations for the development of characteristics of disaster-resilient communities. The following sets of characteristics have been developed based on the comprehensive list of 167 characteristics developed from the Hyogo Framework for Action (HFA). The characteristic being aimed to be established depend upon the:

Partner organisation’s project approach Focus of DRR work being conducted Scope of the project activities Regional, country and local context

The characteristics presented below provide general guidance on the types of outcomes different DRR projects might aim to achieve. Two sets of example characteristics are provided, one general set of ‘priority resilient community characteristics’ and one for ‘slow-onset disasters from a food security perspective’. Any one project may aim to work towards only a few of the characteristics from either table.

Priority Resilient Community Characteristics Thematic

AreaExample set of Priority Resilient Community Characteristics 3

1: Governance

Committed, effective and accountable community leadership of DRR planning and implementation.

Capacity to challenge and lobby external agencies on DRR plans, priorities and actions that may have an impact upon local risks.

Evidence that disaster risk is being taken into account in planning recovery and rehabilitation activities.

2: Risk Assessment

Participatory hazard/risk, vulnerability and capacity assessments carried out and updated, which provide a comprehensive picture of major hazards/risks, vulnerabilities and capacities in the community.

Community uses indigenous knowledge and local perceptions of risk, as well as other scientific assessment methods.

3 John Twigg, Characteristics of a Disaster-Resilient Community. A guidance Note, Version 2 (2009). – Characteristics adapted from those prioritised and utilised by Tearfund.

Version 1.0, March 2015 Page 7 of 8

Thematic Area

Example set of Priority Resilient Community Characteristics

3: Knowledge and Education

Possession of appropriate technical and organizational knowledge and skills for risk reduction and disaster response at local level (e.g. indigenous technical knowledge, coping mechanisms and livelihood strategies).

All sections of community know about contingency plans, facilities, services and skills available pre-, during and post- emergency, and how to access them.

DRR knowledge is being passed on formally through local schools and informally via oral tradition from one generation to the next.

4: Risk Management and Vulnerability Reduction

Food and water supply secure in times of crisis (e.g. through community managed stocks of grain and other staple foods; protected or stored water supplies).

Livelihood diversification at household and community level, including on-farm and off farm in rural areas, with few people engaged in unsafe livelihood practices or hazard vulnerable activities.

Adoption of hazard-resistant agricultural practices and sustainable environmental management (e.g. soil and water conservation, flexible cropping patterns, hazard-tolerant crops, forest management).

Existence of and access to community savings and credit schemes, and/or a community disaster fund to implement preparatory, responsive or recovery activity.

Structural mitigation measures in place (e.g. water-harvesting tanks, embankments, flood diversion channels).

Houses, workplaces and public facilities located in safe areas or hazard-resistant construction methods in use.

Measures in place to protect key assets (e.g. livestock) and items of domestic property (e.g. use of raised internal platforms or containers).

5: Disaster Preparedness and Response

Accessible emergency facilities and equipment available (for shelter, communications, rescue, etc.), owned and managed by the community.

Community-based and people-centred early warning system in place at local level, producing messages which are trusted and understood by whole community.

Community and family level contingency plans exist, developed and owned by the community, linked to higher-level plans and practised regularly.

Community has the capacity to provide effective and timely emergency response services, including training and deployment of volunteers with appropriate skills (e.g. search and rescue, first aid, managing emergency shelters, fire-fighting).

Community has appropriate plans and mutual support systems in place to take care of the most vulnerable – usually the elderly, disabled, HIV/AIDS-sufferers, mothers, young children and unaccompanied individuals.

Characteristics of a Disaster-Resilient Community: slow-onset hazard, food security perspective

Thematic Area Example set of Disaster Resilient Community Characteristics: slow-onset hazard food security perspective 4

1: Governance Committed, effective and accountable leadership of DRR planning and implementation. Community aware of its rights and the legal obligations of government and other

stakeholders to provide protection. Inclusion/representation of vulnerable groups in community decision making and

management of DRR.

4 John Twigg, Characteristics of a Disaster-Resilient Community. A guidance Note, Version 2 (2009). – Characteristics adapted from those prioritised and utilised by Tearfund.

Version 1.0, March 2015 Page 8 of 8

Thematic Area Example set of Disaster Resilient Community Characteristics: slow-onset hazard food security perspective

2: Risk Assessment

Community-level hazard, vulnerability and capacity (HVC) assessments carried out, to provide a comprehensive picture of all HVCs.

HVC assessment (above), carried out as a participatory process, involving representatives of all sectors of community, including all vulnerable groups.

Use of indigenous knowledge and local perceptions of risk as well as other scientific knowledge, data and assessment methods.

3: Knowledge and Education

Local schools provide education in DRR for children through the curriculum and where appropriate, extra-curricular activities.

Community members skilled or trained in appropriate agricultural, land-use, water management and environmental management practices.

4: Risk Management and Vulnerability Reduction

Adoption of sustainable environmental management practices that reduce hazard risk. Food supply and nutritional status secure (e.g. through reserve stocks of grain and

other staple foods managed by the community, with an equitable distribution system during food crisis).

Access to sufficient quantity and quality of water for domestic needs for 12 months of year.

Livelihood diversification (household and community level) including on-farm and off-farm activities in rural areas.

Adoption of hazard resistant agricultural practices (e.g. soil and water conservation methods, cropping patterns geared to low or variable rainfall, hazard tolerant crops) for food security.

Mutual assistance systems, social networks and support mechanisms that support risk reduction directly through targeted DRR activities, indirectly through other socio-economic development activities that reduce vulnerability, or by being capable of extending their activities to manage emergencies when these occur.

Existence of community/group savings and credit schemes, and/or access to micro-credit facilities.

Structural mitigation measures in place – e.g. for water harvesting, field bunds or irrigation dams and channels – built using local labour, skills, materials and appropriate technology as far as possible.

5: Disaster Preparedness and Response

Local organizational structures for disaster preparedness or emergency response in place (e.g. disaster preparedness committee)

Community-based and people-centred early warning system in place, which generates timely, trustworthy and understandable warnings of hazards to reach all members of community.

Community and household contingency plans in place for drought, including preservation of key assets (e.g. fodder, water and health of livestock).

Emergency supplies (buffer stocks) in place, managed by community, alone or in partnership with other local organizations (including grain/seed banks).