Evaluation Report

Evaluation of the UNICEF Sanitation Programme at Scale in Pakistan (SPSP) – Rural, Year 1 (2013-14)

Final Report: 23 October2014

List of Acronyms and Abbreviations BCC Behaviour Change Communication

BHU Basic Health Unit

BRSP Baluchistan Rural Support Programme

CAP Community Action Plan

CATS Community Approach to Total Sanitation

CCD Climate Change Division

CEDAW Convention on the Elimination of All Forms of Discrimination against

Women

CLTS Community-Led Total Sanitation

CNA Capacity Needs Assessment

CRC Convention on the Rights of the Child

CRP Community Resource Person

DAC Development Assistance Committee

DFID Department for International Development

DRR Disaster Risk Reduction

DSI Development Solutions International

FATA Federal Administered Tribal Areas

FGD Focus Group Discussion

FRDP Fast Rural Development Program

GDP Gross Domestic Product

GoP Government of Pakistan

GSF Global Sanitation Fund

HACT Harmonised Approach to Cash Transfers

HCD Human Centred Design

HH Household

HRBA Human Rights Based Approach

IEC Information Education Communication

IP Implementing Partner

IRSP Integrated Regional Support Program

JMP Joint Monitoring Programme

KAP Knowledge Attitude and Practice

KPK Khyber Pakhtunkhwa

LGRDD Local Government & Rural Development Department

LHW Lady Health Worker

MDG Millennium Development Goals

MEAL Monitoring, Evaluation, Accountability and Learning

MICS Multiple Indicator Cluster Surveys

MIRA Multi-cluster Initial Rapid Assessment

MoCC Ministry of Climate Change

MoRES Monitoring Results for Equity Systems

MOU Memorandum of Understanding

NCBP Nations Capacity Building Programme

NDMA National Disaster Management Agency

NDWP National Drinking Water Policy

NFC National Finance Commission

NOC No Objection Certificate

NRSP National Rural Support Programme

OD Open Defecation

ODF Open Defecation Free

OECD Organisation for Economic Co-operation and Development

PATS Pakistan Approach to Total Sanitation

PCA Programme Cooperation Agreements

PDHS Pakistan Demographic and Health. Survey

PHED Public Health Engineering Department

PMER Planning, Monitoring, Evaluation & Reporting

PRA Participatory Rural Appraisal

PSLM Pakistan Social and Living Standards Measurement Survey

RSPN Rural Support Programme Network

RUSFAD Rural Sanitation in Flood Affected Areas

SACOSAN South Asia Conferences on Sanitation

SAFWCO Sindh Agricultural and Forestry Workers Coordinating Organization

SLTS School-Led Total Sanitation

SMC School Management Committee

SPSP Sanitation Programme at Scale in Pakistan

SPSS Statistical Package for the Social Sciences

SRSP Sarhad Rural Support Programme

TOC Theory of Change

TOR Terms of Reference

TPM Third Party Monitors

UC Union Council

UN United Nations

UNDAF United Nations Development Action Framework

UNEG United Nations Evaluation Group

UNICEF United Nations Children's Fund

VSC Village Sanitation Committee

WASH Water Sanitation and Hygiene

WDCO Women Development & Community Organization

WSP Water and Sanitation Program

WSSCC Water Supply and Sanitation Collaborative Council

Acknowledgement The Consultants (AAN Associates) are extremely thankful to all those who contributed to the evaluation. We appreciate the support and guidance extended by UNICEFteam in Islamabad and in the provinces. In particular, Kim Duckworth, Kamran Naeem, Irfan Durrani and Mussarrat Yusuf. We acknowledge and appreciate the support and frank feedback shared by IP management and staff, communities and other stakeholders. This may not have been possible without the untiring efforts, resourcefulness and commitment of the consultants, field teams and AAN core team. The most significant contributions have come from Nadeem Haider, Wajahat Anwar, Salman Mahmood, Kashif Pervaiz, Zaki Ullah, Arbab Qadir Asmat Ali and Arsalan Kashfi. We are thankful to everyone for the hard work put into completing this evaluation. We wish UNICEF and other WASH stakeholders working in Pakistan success with their work. Nadeem Haider AAN Associates Islamabad Pakistan

Disclaimer: The views and opinion expressed in this report are of the consultants and do not necessarily reflect the views or policies of UNICEF or other organizations involved in the project design and delivery.

Table of Contents List of Tables .............................................................................................................. 7

List of Figures ............................................................................................................. 8

List of Annexes ........................................................................................................... 9

Executive Summary ................................................................................................. 10

2. OBJECT OF THE EVALUATION ...................................................................... 17

2.1. Programme Overview ..................................................................................... 17

2.2. Country Context ............................................................................................. 21

2.3. Theory of Change ........................................................................................... 24

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE .................................. 30

4. METHODOLOGY ............................................................................................. 33

4.1. Evaluation Conceptual Framework ................................................................. 33

4.2. Data Collection Methods ................................................................................ 33

4.2.1. Quantitative Evaluation Methods .......................................................... 34

4.2.2. Qualitative Evaluation Methods ............................................................ 35

4.3. Evaluation Constraints, Limitations and Mitigation ......................................... 37

4.4. Ethical Considerations .................................................................................... 39

4.5. Quality Assurance Mechanisms ..................................................................... 39

4.6. Evaluation Management and Stakeholder Participation ................................. 40

5. FINDINGS ........................................................................................................ 42

5.1. Relevance ...................................................................................................... 42

5.2. Effectiveness .................................................................................................. 49

5.2.1. Programme Design ............................................................................... 49

5.2.2. Programme Management & Coordination ............................................ 51

5.2.3. Programme Components & Results ..................................................... 52

Output 1: Demand For Sanitation...................................................................... 52

Output 2: Supply Side Interventions .................................................................. 59

Output 3: Health And Hygiene Promotion (H&HP) ............................................ 66

Output 4: Linkages With Duty Bearers .............................................................. 69

Output 5: Attaining Total Sanitation ................................................................... 70

Output 6: Knowledge Management And Accountability .................................... 72

5.2.4. Stakeholder Contribution ...................................................................... 74

5.2.5. Gender Equality, Equity & Disability ..................................................... 76

5.3. Efficiency ........................................................................................................ 78

5.4. Sustainability .................................................................................................. 82

5.5. Unforeseen Events ......................................................................................... 85

5.6. Human Rights Based Approach ..................................................................... 87

5.7. Partnership and Financial Management ......................................................... 92

5.8. Unexpected Findings ...................................................................................... 94

6. CONCLUSIONS & LESSONS LEARNED ........................................................ 96

7. RECOMMENDATIONS .................................................................................... 99

List of Tables Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province. ......... 20 Table 2: SPSP Theory of Change ............................................................................ 28

Table 3: Evaluation Criteria and Core Questions ..................................................... 32 Table 4: Sampling Calculations for Post KAP Household Survey ............................ 35 Table 5: Summary Table: Instrument, Source and Limitations ................................. 37 Table 6: Evaluation Constraints, Limitations and Mitigation ..................................... 38

Table 8: Community Mobilisation Activities & Training ............................................. 53 Table 9: SLTS Training and Events ......................................................................... 53 Table 10: BCC Campaigns Outreach (Target versus Actual Number of People) ..... 54 Table 11: Supply Side Training Events for Masons and Entrepreneurs ................... 60

Table 12: Types of Low Cost Latrines ...................................................................... 61 Table 13: Number of Latrines Constructed (Target versus Actual) .......................... 63 Table 14: Outreach of Health & Hygiene Messages through Various Mediums ....... 66

Table 15: Capacity Building & Advocacy for Government ........................................ 69

Table 16: Villages Supported in attaining Total Sanitation Status ............................ 71 Table 17: Knowledge Management and Accountability Events and Activities .......... 72 Table 17: Cost Per ODF Declared Beneficiary ......................................................... 80

Table 18: Cost per ODF Declared and Verified Village ............................................ 80 Table 21: Evaluation Recommendations ................................................................ 104

List of Figures Figure 1: SPSP Rural Districts and Agencies ........................................................... 20 Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011 ............................................................................................................. 23 Figure 3: Evaluation Data Collection Methods ......................................................... 34 Figure 4: SPSP Contribution to Sanitation Target .................................................... 42 Figure 5: Province Based Access to Sanitation Facility ........................................... 46 Figure 7: Number of Sources of Health & Hygiene Messages ................................. 54

Figure 8: Health & Hygiene Awareness Messages .................................................. 54 Figure 9: Health and Hygiene Messages that the Respondents Remembered ........ 55 Figure 10: Respondents Able to Recall Three Key Hygiene Messages ................... 55 Figure 11: Source of Health & Hygiene Messages in Schools ................................. 56

Figure 12: Topics Covered during Health & Hygiene Sessions in Schools .............. 56 Figure 13: Awareness of Hazards of Open Defecation ............................................ 57 Figure 14: Awareness of the Need for Achieving ODF Status .................................. 57 Figure 15: Proportion of Population Practicing Open Defecation ............................. 58

Figure 16: Willingness to Construct a Latrine (including methods of cost bearing) .. 58 Figure 17: WASH Facilities Constructed with External Support ............................... 63 Figure 18: Latrines Constructed By Respondent Over Last Year and Type of Support Received .................................................................................................................. 63

Figure 19: Availability of Sanitation Supplies and Trained Masons .......................... 64 Figure 20: Population Using an Improved Drinking Water Source ........................... 64

Figure 21:School WASH Facilities ........................................................................... 65

Figure 22: Availability of WASH Facilities in Boys, Girls and Co-Education Schools 66

Figure 23: Household Water Treatment and Safe Water Storage ............................ 66 Figure 24: Proportion of Respondents who Treat Water .......................................... 67 Figure 25: Hand Washing Place & Soap Availability ................................................ 67

Figure 26: Critical Moments for Hand Washing ........................................................ 68 Figure 27: Knowledge of Water Related Diseases and Incidence of Diarrhea ......... 68

Figure 30: Households Using Underground Drainage .............................................. 71 Figure 31: Gender, Disability and Equity Based Access to Latrines ......................... 77 Figure 32: School Facilities Attributes ...................................................................... 90

List of Annexes Annex 1 Terms of Reference – Evaluation of SPSP Phase 1 Annex 2 Illustration of SPSP Programme Components Annex 3 SPSP Rural Results framework Annex 4 Stakeholders & Roles Annex 5 Evaluation Matrix Annex 6 Key Indicators and Source of Information Annex 7 List of Villages for Post KAP - SPSP Final Evaluation Annex 8 List of People Met for SPSP Evaluation 2014 Annex 9 List of Participants in Provincial Bottleneck Workshops Annex 10 List of Documents Reviewed Annex 11 Evaluation Toolkit Qualitative Instruments Annex 12 Evaluation Toolkit Quantitative Instruments Annex 13 SPSP Outputs - Target versus Actual Annex 14TORs for the Evaluation Reference Group Note: All listed Annexes are consolidated in Vol II of this report.

1. Executive Summary

EvaluationObject „Sanitation Programme at Scale in Pakistan - Rural‟ (SPSP Rural)‟is a multiyear programme (2013-17), of which the first phase was launched by UNICEF in 2013. The programme targets rural parts and particularly the flood and insecurity affected areas. It builds on the experiences and lessons drawn fromUNICEF‟s Post Floods Recovery (2010) Programme named „Rural Sanitation in Flood Affected Districts‟ (RuSFAD) implemented through 2010-12. The programme purpose is; „By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices‟. The SPSP Phase I (essentially Year I) intended to reach out to 1.3 million people by implementing the Pakistan Approach to Total Sanitation (PATS)1. The approach combines the School and Community Led Total Sanitation (SLTS & CLTS) approaches. For UNICEF, the concept of Community Approaches to Total Sanitation (CATS) embodies it all. The programme consists of six interlinked components namely:i) Demand for sanitation;ii) Supply facilitation;iii) Health and hygiene promotion;iv) Linkages with duty bearers; v)Attaining total sanitation and; vi) Knowledge management. The programme design and delivery involved multiple stakeholders with UNICEF being the key agency. The most significant contributions and tasks performed by the UNICEF include programme design, resources mobilization, contracting implementation (national and international non-profit organizations) and technical oversight. The actual implementation was carried out by non-profit partner agencies referred to as „Implementing Partners (IPs)‟. Some project activities have been performed by the relevant government counterparts and field monitoring functions were performed by „Third Party Monitoring Agent (TPM)‟. Research and capacity building functions have been undertaken by Development Solutions International(DSI) (a consulting company) and Rural Support Programme Network. The implementation of SPSP Phase I started in February 2013 and activities were completed by 31 March 2014. The planned budget for Phase I implementation was US $ 13.69 million. Funding for this phase was provided by DFID. This phase was implemented in thirteen (13) districts and agencies i.e. 10 districts in Punjab, Sindh, Khyber Pakhtunkhwa, Balochistan and 3 Federally Administered Tribal Area Agencies (FATA).

Evaluation Purpose, Objectives and Scope The evaluation was commissioned by UNICEF Pakistan with the purpose of assessing the programme achievements and to feed into the design of the next phase. Objectives as stated in the TORs included, determining the relevance, effectiveness, efficiency and sustainability (as per UNEG standards), whilst also

1Pakistan Approach to Total Sanitation (PATS) is a country specific model that evolved from Community Approaches to Total Sanitation (CATS), through a consultative process during 2008-09. PATS includes additional features as compared to CATS, such as sanitation marketing (also referred to as SanMark) and component sharing,

assessing the extent to which UNICEF core programme strategies and normative principles (Human Rights Based Approach, Impact of Unforeseen Events and Financial Risk Management) were met. Scope included the target population of the 13 districts and agencies (including men, women, children, elderly and the disabled)covered during phase 1 of SPSP. The primary audience of the evaluation is the DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. The secondary audiences include other WASH sector partners and communities, who can benefit from programme learning to replicate and scale-up elsewhere.

Evaluation Methodology The evaluation methodology was evolved with a considered view of this being a formative evaluation with a clear focus on documenting key learning and identifying areas for improvement. The intent was to have the next phase informed of the opportunities for improvement. To map the extent of change for key quantitative output indicators, quasi-experimental research design was used to compare the baseline results with end line results (gathered through pre and post KAP surveys). The design however excludes the creation of counterfactual, primarily for being a formative evaluation.The other methods used include secondary sources review, in-depth interviews, focus group discussions, bottleneck analysis (workshops) and observation checklists. For the evaluation design and implementation, the consultants followed UNICEF adapted United Nations Evaluation Guidelines (UNEG), accredited researching norms and ethics An Evaluation Reference Group (ERG) with representatives from UNICEF, DFID, and government departments extended guidance and oversight.

Key Findings and Conclusions Relevance: The consultants have given their opinion on the extent of programme objectives and approaches to national and provincial policies and priorities, needs of men and women beneficiaries (including vulnerable), total sanitation approaches. Moreover, the evaluation offers assessment around level of integration for cross-cutting priorities i.e. gender equality, equity, human rights and disability. By and large, the SPSP Phase 1 programme design found consistent to the national policies and development commitments.The geographic prioritization found appropriate for having lower sanitation coverage than the national average. There are evident departures from the conventional approaches, such as provision of subsidies (for latrine construction) and use of contractors instead of resources (mason) trained in theprogramme. The design and implementation did have relevance with UNICEF policies and had expressed focus on equity (inclusion of subsidy to reach the poor), and gender (having a staff mix (of men and women) in the field, creation of women VSCs and CRPs, focus on girls schools etc.) though the integration of disability remained weak as evidenced in the construction and access to WASH facilities.

Effectiveness: The evaluator assessed the effectiveness with respect to achieving outputs/results, theory of change vis a vis approaches and interventions, across varied programme components. Moreover, the section looks at how effectively programme achieved integration of cross cutting priorities i.e. gender equality, equity, human rights& disability. The programme appears effective for exceeding most of the output targets. The indirect implementation approach where IPs were contracted did help achieving targets in shorter time span. The limited engagement of public sector in programme delivery however apparently has not contributed much with preparing public agencies for PATS replication and scale-up. Both partners and partnership for action research, third party monitoring and centralized training components, largely remained effective in helping IPs achieve targets. In terms of policy influencing UNICEF has been a major advocate and has provided leadership for adopting of PATS for sanitation scale-up. However due to other significant stakeholders, it‟s difficult to attribute improvements and positive policy measures taken at the federal and provincial level such as increases in FATA sanitation budget and PATS roll-out (selectively) across Punjab solely to SPSP. There is however evidence in the form of Provincial Government‟s Master Plans that incorporate PATS model, as well as in some cases where government allocations to WASH sector have increased. Community mobilisation followed by BCC campaigns were in particular effective in changing health and hygiene behaviour as well as covering the target population with multiple mediums, people who claimed to have received messages increased by 41% at the time of the Post-KAP survey and those claiming to adoption of practices increased by 39%. Demand side activities generally remained more effective than supply side interventions. Household subsidy needs to be revised and other modalities identified to address equity and vulnerability. SLTS and CLTS, helped in leveraging the complementarities and demonstrated effective use of PATS model. Receptiveness of messages as well as ability to recall and the element of practicing has been effectively increased in the target population. Within the programme context 65% population now has access to sanitation facility which is significantly higher than the national average and two percent points short of the MDG target. Efficiency: For this, the evaluators have looked into how well the various activities transformed the available resources into the intended results or outputs, in terms of quantity, quality and timeliness. Efficiency in terms of performance against target indicators has been achieved, Cost per beneficiary and villages achieving ODF declaration stage demonstrates efficiencies against the planned costs. The results exceeded by a number of times

pointing to miscalculation as to planned targets and resources required and raising concern around planning, target setting and best use of resources. Instances such as contractors charging the same amount of funds for building new as well as rehabilitation of old school wash facilities suggests areas of financial inefficiencies. It should be noted that efforts to produce detailed analysis were limited due to absence of expenditure data that was not made available to the evaluators. Sustainability: The section looks into the extent that various stakeholders including the public sector and civil society, are likely to sustain the positive outcomes of the project after the end of phase 1. The interaction with UNICEF suggests it envisaged SLTS and CRPs as potential sustainability agents, however across IPs it was not found to be translated into a workable action plan.IPs found no/little time to prevent the process of slippage following ODF declaration and certification.The linkages with duty bearers component could achieve little in terms of fostering ownership amongst public agencies to either takeover responsibility and/or scale-up the efforts. The community structures and mechanisms, such as CRPs and VSCs are likely to sustain but with limited success. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as demand issues. The SLTS component without continued support and monitoring is less likely to sustain either. Knowledge, attitudinal and behavioural changes that this programme has been able to bring amongst men, women and children, are the most significant and likely results that may sustain. The rehabilitated WASH facilities at schools and health outlets may not sustain without proper O&M mechanisms. Gender Equality, Equity & Disability: The design and PCA refer to gender considerations in terms of boys/girls schools, men and women CRPs and VSCs, as well as IP field staff. Moreover responsive BCC campaigns ensured participation of men and women as well as having a specific women focus. The programme also considered women and girls privacy and safety aspects (though the elevated WASH facility design has drawn criticism in regards to privacy). However there were gaps for instance formal gender analysis and study for the programme for instance examining the burden on women and children due to increased access to WASH facilities (increased water fetching, cleaning of sanitation facilities etc). The programme demonstrated the equity focus by using PRA Techniques to identify the vulnerable and extremely vulnerable and then extending household subsidy for 10% beneficiaries. However household subsidy approach has certain challenges (creation of dependence, community friction at identification stage etc) which require further consideration on modalities of subsidy for future programmes.Addressing of disability however remained a weak aspect of the programme at both community as well as school level in terms of sensitisation as well as actual infrastructure designs and sanitation supplies. Overall in terms of access to sanitation facilities, results of the post-KAP survey confirm an increase for women, as well as people in the lowest

income quintile, however future programming should further strengthen the modalities of addressing gender, equity and disability. Unforeseen Events: PCAs in general seemed to lack any planning or comprehensive risk assessment of the programme areas and overall security environment. Fortunately nothing significant happened during the course of implementation that would have derailed programme interventions. IPs knowledge of local communities and in most cases having local staff helped in ensuring smooth running of activities. However given the environment within which the programme is implemented, there is a need to have a Risk/Disaster Strategy in place in the programme design/PCA. Partnership & Financial Risk: To a large extent appropriate partners were selected (with some weakness elaborated in detail in main report), However Third Party Monitor reporting was able to identify operational and programme delivery challenges with the umbrella arrangement in Sindh as well as in the partnership in Baluchistan which could have been avoided by building capacities. Financial risks were mitigated through some basic level capacity building. UNICEF did well to have conducted a partner capacity baseline which can now be used to compare the implementation weaknesses to help identify capacity needs of these partners, were they to be part of next phase. The programme however did very little to build capacity of government departments. At provincial level, there was disconnect with duty bearers in terms of taking them on board, engaging them in planning to execution level activities, introducing rewards or incentives through the departments. Overall the experience of SPSP confirms that PATS can be implemented on a large scale with key requisites such as increased capacity of and advocacy with government, sector stakeholders and communities to apply PATS approach effectively.

Lessons The key lessons learnt are outlined below; a) Access To Sanitation Facilities and Reduction in Violence Against Women. The evaluation findings have contributed to unravelling a critical link between open defecation and extreme cases of gender violence. A significant number of respondents shared that the construction of latrines have reduced the incidences violence against women such as karo kari (honour killing). To them, the construction of latrines have reduced women need to step out, hence the probability of being accused of illicit relations and subjected to violence. Results of the KAP survey suggest that 83% feel it is unsafe for women to go out to defecate in the open. The lesson that is derived from this instance is that though an indirect result of WASH intervention, the benefit of contributing to a reduction in violence is a substantially important aspect and should therefore be integrated in future programmes in a cognizant way. b) Shame, Shock and Disgust versus Pride, Self Respect and Dignity There is an ongoing debate on which approach to use, it must be noted that the purpose of both the approaches is to trigger the CLTS process, with a larger

objective of saving children‟s lives and prevention from diseases. Experience of SPSP suggests that to bring about a change in people‟s behaviour resorting to the use of both approaches within the same programme can ensure more effective and wider reaching triggering, than to fixate on the use of one or the other. c) Business Incentive and Opportunity Is a Larger Motive for Response. DSI‟s action research suggests that manufacturers, entrepreneurs and mart owners who have a business risk bearing attribute and see supply to the underserved areas as an opportunity to expand their business are more likely to sustain sanitation supply in order to meet the demand created through CLTS and SLTS. Therefore instead of focusing on extending incentives to creating and supporting entrepreneurs/sanitation mart owners from within the community, the programme should identify and work with manufactures/entrepreneurs to help realise the market potential and opportunity offered by these underserved areas for business expansion, this will ultimately ensure continued and sustained supply facilitation. d) Using Subsidy to Address Equity Can Lead to Dependency The purpose of the subsidy was to ensure equity aspects were addressed. However this approach has some inherent problems as evidenced in SPSP Phase 1.

a) In extremely underdeveloped and impoverished areas, concluding through PRA that on comparative basis one is worse than the other does not address the fact that many families in absolute terms are extremely vulnerable and are likely to be excluded from benefits of the subsidy.

b) Subsidies also reduce the willingness of families to put their own very limited resources into latrine facility construction, given a free handout in the form of subsidy may become available to them, thus acting as a detrimental incentive.

c) Lastly by the end of the project those who have not received this subsidy, remain dependent and wishful for similar other interventions.

Future programming therefore needs to re-examine the modality for subsidies.

Main Recommendations Following are key recommendations across design, operation and policy areas, 1) Future programming should provide adequate duration for all stages of the

programme i.e.inception phase, PATS implementation phase and exit phase. This should include defining the duration it takes for a standard PATS implementation cycle.

2) The programme Theory of Change needs to be articulated with clear linkages, pre-requisites, risks and assumptions. Similarly a sustainability strategy, focusing on sustaining ODF status should be developed. PCAs should include specific IP exit plans, and stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to delivery it (Community as well as role expected from Government Departments).

3) Following aspects are recommended for the Supply Facilitation component, a) Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. Innovative results emanating from Action Research carried out by DSI need to be examined for instance letting the market

forces define supply side, with the programme highlighting the demand created through CLTS and SLTS as business opportunity. b) Community Resource Persons (CRPs) Concept of service fee for Community Resource Persons (CRPs) worked and should continue for the period of mobilisation. Options should be considered to utilise these resources for sustained focus on sanitation by transforming them into small entrepreneurs and help expand the capacity of the market to supply services and products. c) Household Subsidy Modality and intended benefits of providing a household level subsidy should be reassessed, including possible detrimental effects on the community. Other mechanism should be tested (such as providing collective incentives at the community level). For equity purposes and the needs of the vulnerable revolving credit/contributory fund can be considered as an option to be managed by the VSC.

4) Gender equality, equity and vulnerability along with special needs of the disabled should be addressed through clear processes which need to be articulated in the programme design, the TOC (explaining the change process and results), while PCAs should include sections detailing specific interventions, criteria and processes that IPs have to implement to ensure inclusiveness (such as during PRA, School WASH Facilities, targeted BCC, disability integration in Latrines).

5) UNICEF should ensure that there is clarity and agreement on how awards, incentives and rewards to departments at the district level will work and whether there are budgetary allocation implications that need to be agreed addressed at the provincial level.

6) In SPSP Phase II, UNICEF should pilot PATS implementation with Public Stakeholders in at least one Province (which is receptive and committed to addressing sanitation issues).

2. OBJECT OF THE EVALUATION

2.1. Programme Overview „Sanitation Programme at Scale in Pakistan - Rural‟ (SPSP)‟ was launched by UNICEF in 2012. It is a multi-year national sanitation programme focusing on rural parts of flood and insecurity affected areas (2013-17). It builds on lessons and experiences of UNICEF‟s „Rural Sanitation in Flood Affected Districts‟ (RuSFAD) programme (2010-12). Purpose The purpose of the programme as stated in the project documents is; „By 2017, most vulnerable and marginalized children and women will be using safe drinking water, improved sanitation and hygiene practices‟. The aim of the programme is to reach 1.3 million people by the end of year one or first phase2. The programme designed is built on the principles of Community Approaches to Total Sanitation (CATS), which with slight adaptations for Pakistan is referred to as Pakistan Approach to Total Sanitation (PATS). The PATS model advocates use of both School Led and Community Led Total Sanitation (SLTS & CLTS) approaches, and has been applied in parallel, for the programme under evaluation. Components and Indicators There are six components or pillars in the programme design. In the results matrix, these have been equated with outputs (with 18 output indicators) contributing to the programme result or outcomestatement. The outcome has 3 indicators to be realized by 2017. The six components/outputs are as follows: 1. Demand for sanitation to achieve open defecation free status created in targeted

communities 2. Sanitation demand sustained with supply side interventions 3. Improved Health and Hygiene behaviours promoted with the target community 4. Linkages developed with duty bearers/public institutions to support PATS 5. Villages supported in attaining Total Sanitation status 6. PATS programme knowledge management and accountability Framework

implemented The outcome indicators as appear in the results framework are as below (result framework attached as Annex 3): , 1. Proportion of the population using an improved drinking water source increases

to 94% in rural areas. 2. Proportion of population practicing open defecation reduced to 35% in rural

areas. 3. Government expenditure on WASH as a proportion of total government

expenditure increases by 1%.

2. The terms SPSP Year 1 and SPSP Phase 1 have been used interchangeably, however both imply the implementation period February 2013 to March 2014. The evaluation covers the programme activities implemented through this period.

18

The overall programme outcome is to be achieved through phased implementation of year or plus implementation cycles. Hence, this being the year 1 ex-post evaluation (for project implemented between 2013-14) excludes progress assessment of outcome indicators. This remains a formative evaluation for UNICEF with the clear focus on learning documentation to inform the programme design and implementation for follow-upphases i.e. between 2014-17. Stakeholders The programme design and delivery involved multiple stakeholders ranging from relevant public agenciestocivil society organizations, research & monitoring organizations to leading capacity development networks (refer to Annex 4 for detailed breakdown of stakeholders and respective roles). Significant public agencies involved in the programme (first phase) include:Climate Change Division (CCD, formerly the Ministry of Climate Change) at the federal level, and public health engineering (PHED), health, education, local government (LG) departments at provincial and district levels. The programme design offers a clear delineation of roles between stakeholders including coordination arrangements. For instance, theprogramme envisaged CCD to facilitate public sectorcoordination, oversight and provide policy guidance with respect to programme coherence with PATS principles and guidelines. The public stakeholders at provincial and district levels such as health, PHED, LG, education were expected to provide field implementation oversight including implementation of selective hardware/engineering components (identification of schools for schools sanitation interventions, planning and execution of water schemes rehabilitation),distribute and contribute to rewards and provide ODF certification. UNICEF contracted Development Solutions International (DSI) Pakistan, a marketing research and training enterprise to develop and implement(on pilot scale) a Sector Mechanism for Sanitation Marketing (SMSM). Moreover, DSI provided technical support for latrine designs. The Rural Support Programme Network (RSPN), a leading national network for technical advice and capacity development took lead in training and mentoring related capacity development of the Implementing Partners (IPs), Community Resource Persons (CRPs), government departments and officials on PRA, community mobilization methods, PATS/CATS approaches including IEC material. APEX, a consulting company was contracted for„Third Party Monitoring (also referred to as TPM) services. The role included developing a complete process and output driven M&E system, undertaking field monitoring at all project locations and providing performance assessment reports for management decision making. Knowledge management was one of the key functions to be performed by TPM.

19

The non-profit organizations, contracted as implementing partners (IPs) took lead in field implementation3.These included national and international NGOs, with portfolio of WASH projects in designated geographic regions. The programme envisaged multiple roles for UNICEF mainly technical & operational oversight and backstopping, resource mobilization, stakeholders‟ coordination, sub-contracting and contract management, advocacy and knowledge management. The following descriptionelaborates on specific UNICEF contributions.. Programmatic Redesign & Management of TPM:This included redesigning of the integrated sanitation model and programme implementation framework incorporating learning and recommendations from the RusFAD programme. In addition to this UNICEF also contracted and managed an Independent third party monitoring system within the MoRES framework. Capacity Building: This involved building a cadre of PATS master trainers in all the provinces, while also liaising with and building capacity of government departments to implement and prioritize PATS. Policy Advocacy & Integration into Provincial WASH Master Plans: UNICEF provided policy oversight and contributed to decisions ensuring the Provincial Government‟s Master plans incorporated UNICEF‟s model of PATS programme implementation. Knowledge Management: Activities included preparing PATS training documentary, commissioning thematic issue papers, action research by DSI, Menstrual Hygiene Matters (MHM) research and WASH Equity study, Development of PATS training manual for master trainersand guidance booklets for CRPs, SOs, School teachers, WASH clubs, Masons, entrepreneurs and Village Sanitation Committees (VSCs) Financial: This included capacity building of IPs and Government counterparts, as well as focusing on lowering third party independent monitoring costs to 3% of overall programme cost. Duration & Resources The evaluated programme is part of the UNICEF‟s medium term rural sanitation programme planned for 2013-17. The phase evaluated commencedin February 2013 and activities were completed by 31 March 2014. The allocated budget for the first phase (2013-14) was US $ 13.69 million.Funds were provided by Department for International Development (DFID – UK)from two humanitarian funding streams; one for assisting flood affected areas while other targeted people insecurity(militancy) affected areas.

3 Muslim Aid, Qatar Charity, TVO (Trust for Voluntary Organization), BRSP (Baluchistan Rural Support Programme), WDCO (Women Development & Community Organization), NCBP (Nations Capacity Building Programme), NRSP (National Rural Support Programme), SAFWCO (Sindh Agricultural and Forestry Workers Coordinating Organization), Sabawoon (Sindh Agricultural and Forestry Workers Coordinating Organization), SEED (Social Efforts for Education & Development), IRSP (Integrated Regional Support Program), SRSP (Sarhad Rural Support Programme)

20

Implementation Status The activities for the first phase (2013-14) were completed on 31 March 2014. However, the government partners in KPK province and FATA regions mandated to restore/rehabilitate water schemes were given a no-cost extension until end of July 2014 to facilitate complete utilization of funds. Coverage During Phase 1 of the programme, SPSP was implemented in 13 districts and agencies (10 districts in Punjab, Sindh, Khyber Pakhtunkhwa and 3 Federally Administered Tribal Area Agencies - FATA). Find below the details of programme coverage in terms of geographic spread with corresponding beneficiaries (intended) in different regions. Province/Area District/Agency Number of Beneficiaries

Flood Affected

Punjab Sindh Baluchistan

Dera Ghazi Khan, Rajanpur Jacobabad, Kashmore, Shikarpur Jaffarabad, Naseerabad

680,000

SecurityAffected

Khyber Pakhtunkhwa FATA

Kohat, Tank, DI Khan Kurram , Mohmand and Bajur

620,000

Table 1: Programme Coverage in Terms of Beneficiaries per Area/Province.

The following map (Figure 01) illustrates the geographic coverage of the programme.

Figure 1: SPSP Rural Districts and Agencies

21

2.2. Country Context Pakistan has an estimated population of close to 180 million,of which almost 1/3rd population lives below the poverty line. With a population growth rate of more than 2.9 percent a year,every year an additional 4.0 million people4are added needing additional clean water and sanitation facilities. This has long been established that lack of access to clean water and sanitation facilities carries wide variety of impacts. The data and evidence to establish the extent of the burden imposed on the people of Pakistan has been limited, which to a degree contributed tolower prioritization and consequently resource allocations to water and sanitation in public services delivery. Moreover factors such as insecurity leading to internal displacement, recurrent/annual floods and cyclic pattern of disasters has further aggravated both the access and quality issues. It is therefore not surprising that Pakistan lags behind its committed MDG (relating to sustainable access to safe drinking water and basic sanitation, MDG 7-C) target forsanitation outreach and access to 67% of population by 2015. The Joint Monitoring Programme (JMP) 2012 report indicates that only 48% people are using improved sanitation (72% urban and 34% rural), while 23% are still practicing open defecation. Moreover, poor sanitary conditions remain one of the major contributing factors to modest progress on key health indicators. Diarrhoea caused by these conditions is one of the leading causes of mortality for children under 5. Children also suffer disproportionately from sickness due to diarrheal diseases with almost 25 million cases (PSLM 2006-7) reported annually. UNICEF‟s „Investing Wisely, Sanitation & Water, Saving Lives‟ report5 indicates that between 60 to 75 million people are affected by diarrheal diseases annually, which costs the country 3.94% of gross domestic product (GDP) in terms of health expenditure. Another report (DFID 2013) states that, it is diarrhoea that is causing severe malnutrition (amongst 5% at least) and moderate malnutrition (amongst 20%) of the children under five, which places Pakistan behind Sub Saharan countries e.g. Ghana (DFID 2013). In addition to health concerns, lack of water and sanitation services are also associated with socio-economic disadvantages. A recent DFID study6 points out that there is evidence to suggest that lack of availability of water, sanitation and latrine facilities can create gender gap at homes as well as significantly decrease girls‟ attendance at schools. The social analysis of WASH illuminates gender and age disparities, limited community engagement in services delivery. For instance, within gendered domains, mostly women and sometimes children are tasked to fetch water from nearest water point. The RuSFAD programme evaluation raised concern around added workload for women for houses that constructed pour-flush latrines. Similarly, where open defecation is a norm, the standards for observing privacy are relatively strict for women. Hence, left with the option to go out in the dark, which again poses both physical and protection risks. Children and elderly people are often excused for age and mobility considerations.During field evaluation, some communities even referred

4WSP: Economics Impacts of Inadequate Sanitation in Pakistan 5Investing Wisely Sanitation and Water Saving Lives, Pakistan Sector Status Report 2012, UNICEF

6Water, Sanitation and Hygiene, Evidence Paper, DFID 2013

22

Price for Poor Sanitation: 116,013 children (under five) die due to diarrhoea each year in Pakistan – approximately 13 deaths per hour - many of these deaths can be prevented by adequate sanitation, safe drinking water and improved hygiene (Pakistan Demographic Health Survey (PDHS) 2007-8). In Pakistan poor sanitation and hygiene accounts for an annual loss of PKR 343.7 billion. (Pakistan Sector Status Report 2012). “Pakistan is one of the few fateful countries in the world that have witnessed large-scale outbreak of cholera and typhoid fever”. (Mukhtar et al as quoted in DFID 2013).

to men preferring open defecation despite having latrines at home, primarily due to the embarrassment they felt of sharing the same space with family women. In terms of involving users or communities in services delivery design and management, very few examples are available where public agencies may have successfully engaged communities for design and management of water points/services. The policy initiative largely failed to translate into practical examples for limited technical capacities within public agencies for community mobilization, sustained engagement and users unwilling to contribute (financially) to operations and maintenance. Through the last decade, the prioritization of water and sanitation sector has led to steady increases in the budgetary allocations (refer Figure 2), a trend noted across all provinces. However unfortunately increased financinghas not translated into improved service delivery for multiple reasons. Some of those include under-utilization for delayed releases, budgetary cuts and diversions. The joint budgeting/allocation, as is practiced in most provinces isskewed in favour of water in relation to sanitation. Moreover, the anecdotal accounts from the field indicate steady increase in latrine construction costs (on average between US$ 50-100), which in a way discourages people, more so the poor, to invest in constructing latrines. Those at risk to economic shocks (for natural disasters and insecurity) are also reluctant to invest in latrines. As for the groups who could afford, latrine construction rests at a much lower priority compared to food, clothing or entertainment. On a positive note, the last decade has seen some encouraging public policy shifts. For instance, formulation and approval of the National Sanitation Policy (2006),which set the goal to reduce the number of people without sustainable access to improved sanitation by 2015, and provide improved sanitation to 100% population by 2025. The policy, while referring to strategic areas of intervention, mentions application of Community Based/Led Total Sanitation (CLTS) approaches to achieve desired results. In order to contextualize communities' involvement in sanitation, a Pakistan-specific model of Community Approaches to Total Sanitation (CATS) was evolved, i.e., „Pakistan Approach to Total Sanitation (PATS), through a consultative process during 2008-09. The PATS, though an evolution of CLTS entails additional features than traditional CLTS approaches. For instance, introduction of sanitation marketing, component sharing (public-private partnership) and others. PATS has a geographic focus on peri-urban and rural areas. It aims to end the practice of open defecation while transforming hygiene behaviour.

23

Another policy milestone achieved was the National Drinking Water Policy (NDWP 2009), which acknowledges water as a fundamental right of every citizen and aims „to improve the quality of peoples‟ lives by reducing water-borne diseases with the provision of safe and sustainable drinking water by 2025 at affordable cost‟. In last few years, the public sector spending on water and sanitation has seen a significant increase, which also reflects prioritization of water and sanitation on the public policy agenda (Pakistan Sector Status Report, 2012).

Figure 2: Budgetary Trends for Water & Sanitation in Pakistan Source: Water Sector Report 2011

The 18thConstitutional Amendment & 7th National Finance Commission Award(NFC) stand out as major developments, offering huge potential for devolved planning and management of civic services including water and sanitation. This shall give provinces greater access to resources and mandate to make their own specific policies and actions plans. It offers the opportunity in terms of greater ownership at local level and drawing need based and context specific plans and interventions. However,the dividends of positive policy shifts are yet not visible on the ground in terms of improved service delivery. The dismal situation of sanitation and clean access to water (as highlighted above) was further exacerbated in Pakistan, due to the repeated floods during 2010, 2011 and 2012, which affected at least 20 million people in the country. Some have been affected repeatedly According to the National Disaster Management Authority (NDMA) Pakistan, the heavy monsoon flooding in 2012, affected an estimated 4.8 million people in the provinces of Punjab, Baluchistan and Sindh. According to a MultiCluster Rapid Needs Assessment (MIRA) an estimated 1.4 million children (of which 392,000 under five) were directly affected by the floods. These floods have resulted in contamination of wells and other sources of drinking water, further compromising the sanitation and hygiene environment. This has contributed to visible surge in water, sanitation and hygiene related diseases such as diarrhoea. Moreover, the insecure environment along the western borders and security operations in the Federally Administered Tribal Areas (FATA) have resulted in mass displacement across the region. Millions have been displaced since 2008. The authorities struggled to tend to the humanitarian needs of the displaced. The displacement further strained the natural resource base and services delivery in host areas and communities. UNICEF conceptualised and rolled out SPSP to address the unmet sanitation needs in the rural parts of the insecurity and flood-affected areas.

24

2.3. Theory of Change The project documents shared for review did not include an explicitly defined„Theory of Change (TOC)‟forthe programme7. However,as required the consultants in consultation with UNICEF and IPs have developed the following as TOC for the programme evaluated. The SPSP design is drawn on PATS framework with some adaptations. The SPSP design is drawn on a set of core assumptions that explain what changes should happen and why in order to lead to an ODF environment with improved health and hygiene practices and access to water. Moreover, there are assumptions, which explain how the envisaged changes happen and how they can be catalysed at each level (community, at local, provincial and national level). These constitute the building blocks of the overall envisaged/assumed change process. Based on the understanding of why, what and how changes may happen with respect to sanitation behaviours culminating in creation ofOpen Defecation Free (ODF) environment, the consultants have developed an intervention logic. The logic includes strategies and activities, thattrigger the change process. The description below entails core assumptions and the two partscomprising a matrix and an illustration, which cumulatively explain the TOC. Core Assumptions The programme is based on the following core assumptions, with a logical flow between them from 1 to 5 (that is 2 logically follows 1, it does not represent hierarchy i.e. 1 is not more important than 2) describing the process of change: 1. Open defecation causes serious health and environmental concerns. It is an

individual and a collective community issue. Its practice should be abolished and

a new social norm of ODF environment should be instituted.

2. The decision to end Open Defecation (OD)has to be collective, explicit and

widespread, thus giving individual families the confidence that others are also

abandoning the practice. When social pressure is transformed into collective

commitment by the community to end the practice of open defecation, it can

become self-sustaining.Once it reaches the tipping point the change is more

rapid and far-reaching.

3. Open defecation happens because of lack of awareness, resources and socio-

cultural reasons. Even with awareness and resources, it may continue because

of the socio-cultural acceptance within the community. This needs to be replaced

and addressed through recognition of new behavioural expectations.

4. Due to socio-cultural and collective dimensions, the decision to end open

defecation lies with the community. Ending open defecation is therefore a

collective social change process led by informed and empowered communities

that challenge and change pre-existing normative beliefs and attitudes.

5. A population wide shift to a new social norm of not continuing with open

defecation requires an enabling environment.Firstly,a clear recognition of the

7 UNICEF PAKISTAN WASH Security 2012 PROPOSAL

UNICEF PAKISTAN WASH FLOOD 2012 PROPOSAL

25

hazards and consequences of open defecation to communities, who then

consider it a practice not to be engaged in. Secondly, the enabling environment

needs to provide for sustainability in terms of options (low cost options, drainage,

supplies accessibility etc) as well as a connect with duty-bearers so that

community is protected from future hazards of open defecation.

Activties

Outputs

Outcomes Result (Impact)

Short Medium Long

SPSP Rural – Theory of Change – Part 1

27

- Formative research feeds into BCC

- BCC campaigns launched - SOs facilitate PATS in

communities and schools. - Wash Clubs formed and

strengthened - Participatory Health and Hygiene

Promotion activities conducted - School WASH Facilities provided - CRPs trained and triggering CLTS. - VSCs formed and their capacity

built. - Community Action Plans

facilitated. - School teachers and education

officials trained on facilitating SLTS

- Masons, Entrepreneurs, Sanitation Mart Owners Trained

- Community informed on technical options

- Demo Latrines installed - Water facilities rehabilitated and

built. - Linkages for accessing loans to

create household sanitation facilities.

- Capacity development and advocacy events with Duty Bearers.

- Advocacy for rewards by government for ODF communities.

- Advocacy with duty bearers to support Total Sanitation programmes

- Mass Media BCC campaigns launched.

Demand for sanitation to achieve open defecation free status created in targetedcommunities Sanitation demand sustained with supply side interventions Improved Health and Hygiene behaviors promoted with the target community Linkages developed with duty bearers/public institutions to support PATS Villages supported in attaining Total Sanitation status PATS programme knowledge management and accountability Framework implemented

At & Across Community Level

Community aware, mobilized and empowered to take action (through VSC), CRP & VSC formed, CAPs facilitated.

Community and School and engaged in BCC campaigns &attaining ODF status.

Low cost options, supplies and labour introduced.

ODF Certifications and Incentives awarded

Demonstration of model ‘Total Sanitation’ villages.

Linkages with Duty Bearers

Improvement in health condition and reduction in diarrheal diseases

Safe water usage given households are treating water and storing safely.

At School Level

Health and Hygiene Promotion, formation of Wash Club, school taking responsibility to ensure ODF status in community.

Functional WASH facilities help maintain ODF status in school.

Schools Duty Bearers trained to take action

Children, especially girls regularly attending school

At Provincial/District Level

Improved Provincial/District budget allocation to WASH specific basic social services

PATS supported/implemented by the government.

Change in social norms. The decision to abandon open defecation has to be collective, explicit and widespread thus giving individual families the confidence that others are also abandoning the practice. When social pressure is transformed into collective commitment by the community to end the practice of open defecation it can become self sustaining, once it reaches the tipping point the change is more rapid and far-reaching and those who chose to practice OD are denounced.

Social norm related to OD is addressed through recognition of new behavioural expectations as OD happens because of lack of awareness, resources and socio-cultural reasons. Even with awareness and resources it may continue because of the socio-cultural acceptance within the community

Due to socio-cultural and collective dimensions, the decision to end open defecation lies with the community. Ending open defecation is therefore a society level change in social norm led by informed and empowered communities.

2.4. A population wide shift to a new social norm of not practicing open defecation requires an enabling environment, firstly a clear recognition of the hazards and consequences of open defecation to communities, who then consider it a practice not to be engaged in. Secondly the enabling environment needs to be provided for sustainability in terms of options (low cost options, drainage, supplies accessibility etc) as well as a connection with duty-bearers so that community is protected from future hazards of open defecation.

By 2017, most vulnerable and marginalized children and women are using safe drinking water, improved sanitation and hygiene practices Open defecation causes serious health and environmental concerns. It is an individual as well as a collective community issue. Its practice is abolished and a new social norm of Open Defecation Free Environment is instituted.

Access of targeted and neighbouring communities to safe drinking water, improved sanitation and hygienic practices, leading to conditions that do not cause diarrhoea and other water borne diseases

Sustainable ODF communities climbing up the sanitation ladder with duty bearers responding to infrastructure development needs through allocation of resources in an equitable manner.

28

Table 2: SPSP Theory of Change

SPSP Rural – Theory of Change – Part 2

Enabling Environment Assumptions and Risks

Changing Policy and Institutional Environment (18th Amendment, Local Government Elections, Provincial Policies and Programmes)

Proper Identification of Partner with capacity, outreach and ability to deliver with accountability.

Support at implementation level from the various departments of District and Sub-District Government.

Quality advice and facilitation, by UNICEF, IPs and involved experts.

Identification and effective use of natural leaders (whether in Government, Community (CRPs, VSC Member, Religious Clergy or Teachers etc.)

Proper sanctions (authority to work, NOCs, registration of organizations (IPs) in place).

Key local community persons engaged through the process.

Availability of sound technical advice (UNICEF Senior Management, Subject Experts, 3rd Party Entities, Field level advice from IPs and input from Government (at provincial, district and sub-district level)

Programme implementers with appropriate capacities to engage people in programme activities i.e. CRPs, Mason, Mart Owner, VSC Members, and Teachers etc.

Programmatically sound and locally viable interventions.

Effective financial controls and M&E arrangements in place (Financial management and reporting capacities of IPs, M&E arrangements for soft activities as well as technical activities.

Qualified staff and implementing partner’s identified/recruited/trained for the programme, having the requisite skills and knowledge.

Adequate availability and utilization of resources (both in terms of human resources as well as financial resources).

The programme implementation (its various phases community mobilization, triggering, BCC etc.) is not impeded by influential or other locals.

Community engages fully with the various programme interventions and stages.

Programme interventions are supported at collective as well as individual level, to ensure desired change occurs and necessary actions follow.

Programme based participatory techniques (whether through community, CRPs, VSC, government etc) encourage and ensure individual and collective action.

Programme activities are not hindered by environmental and ecological constraints or issues, in which case alternate approach would have to be identified.

Resources for the programme remain accessible through the life time of the first phase, including financial as well as human resources.

Technical options and interventions are appropriate for the use and local context.

Security situation in few selected areas may delay achievement of results.

29

3. EVALUATION PURPOSE, OBJECTIVES AND SCOPE

Purpose This is formative evaluation with the purpose to assess programme design and delivery to inform the follow-up phases of SPSP. Theevaluation is commissioned with the primary intent to accumulate learning, whereas accountability appears to be the secondary purpose. The evaluation TORs underline assessment of programme achievements especially in terms of reaching out to the most vulnerable children and women. The assessment around choice to implement PATS with government and non-profit partners and integration of UNICEF programming priorities (normative principles) such as human rights and gender. The primary users or audiences of the evaluation include DFID, UNICEF, Implementing Partners (IPs) and government stakeholders. Each of these stakeholders have something to gain from the evaluation. For instance, the findings may facilitate UNICEF readjust the design and guide IPs with effective and efficient programme delivery. The government stakeholders may pick ideas of how PATS implementation could be adopted and up-scaled within public sector. WASH sector partners as secondary users could benefit in terms of picking up best practices for replication elsewhere. Objectives The evaluation TORs outline the specific objectives(refer to Annex 1 for complete TORs), which are reproduced below. These are:

1. Determining the relevance, effectiveness, efficiencyand sustainability of UNICEF SPSP-Rural Phase I programme (as per UNEG standards);

2. Assessing the extent to which UNICEF core programme strategies were met. 3. Assessing how UNICEFs normative principles were reflected in the approach. 4. Identifying bottlenecks and success factors for programme implementation. 5. Commenting on the constraints and opportunities of programme

implementation. 6. Assessing effectiveness of the strategic partnership for process monitoring

and of the collaborative relationship for programme implementation (RSPN for Training);

7. Providing recommendations to strengthen PATS programme; 8. Assess the integration of UNICEF normative principles i.e. human rights

based approach, gender equity and environmental sustainability. Scope The evaluation was commissioned by UNICEF Pakistan. The evaluation covered the SPSP implementation (activities undertaken) between February 2013 – March 2014, hence could be referred to as SPSP Phase or Year 1 evaluation. As desired, the evaluation focused on determining the relevance, effectiveness, efficiency and sustainability of UNICEF SPSP-Rural Phase/Year I. The evaluation

31

scope included assessment of all six key programme areas/outputs(explained in the introduction section) with more focus on areas where design had changed based on learning from RUSFAD III evaluation (RuSFAD III preceded SPSP I roll-out and implemented in flood and polio affected regions). The evaluation scope included assessment of UNICEF‟s choice for strategic partnership for process monitoring (TPM)including the collaborative relationship for programme implementation. Moreover, SPSP-Rural‟ coherence (in design and implementation) to UNICEF core programme strategies i.e.

i) Capacity assessment and capacity development ii) Service delivery iii) Advocacy iv) Strategic partnership v) Knowledge management vi) Communication for development

The evaluation TORs prescribed the evaluation scope in terms of methods, which included:,

i) representative assessment of post KAPs (post KAP for communities and non-representative post KAP for schools) that helped assess the progress or extent of change on quantifiable indicators (thirteen outcome and output level indicators),

ii) qualitative assessment of programme achievements and learning as per selected OECD-DAC criteria,

iii) evaluation of capacity development of SPSP-rural non-governmental and governmental stakeholders,

iv) PATS related bottleneck analysis for each province. The geographic scope of the evaluation includes thirteen districts and agencies (where SPSP-Rural Phase I implemented) of Baluchistan, Punjab, Sindh, KP and FATA. The evaluators interacted with/consulted stakeholders at national, provincial district/Sub-district and community levels. Evaluation Framework The evaluation was guided by and followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG) norms, standards and ethical guidelines. The evaluation TORs outlined evaluation purpose, objectives, scope and criteria, broadly found consistent with OECD-DAC evaluation framework i.e. relevance, effectiveness, efficiency and sustainability. The evaluation included assessment of financial risks, integration of human rights and the impact of unforeseen events on programme delivery.Furthermore, evaluators assessed the gender equality and equity aspects,as crosscutting theme for the programme design and delivery. The element of financial risk assessment was added to evaluate UNICEF‟s ability to enable IPs to demonstrate transparent and accountable management of programme funding. The aspect of human rights was assessed to give a measure of howe design and delivery incorporated WASH as rights issue, given that Pakistan has ratified several international treaties, under which the provision of/access to safe

32

drinking water and improved sanitation is guaranteed.It was included to help assess the translation and application of HRBA approach in WASH services both by UNICEF and implementing partners8. Lastly, the impact of unforeseen events was assessed in view of repeated floods (in some areas) as well as the insecure environment. The evaluators are of the view that TORs could have been made more explicit in terms of defining evaluation questions and sub-questions. During inception phase of the evaluation, the consultants together with relevant UNICEF team members reviewed and revised the evaluation questions to enable more precision in terms of information needs for the evaluation. The evaluation matrix drawn describes the information needs, sources and methods for data collection (refer to Annex 5 – Evaluation Matrix for detail breakdown). The Annex 6 provides a complete list of key result indicators used for assessment of programme achievements (refer to Annex 6 – Key Indicators and Sources of Information).The following table outlines the core evaluation questions. Evaluation Criteria Core Questions

Relevance To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities?

Effectiveness To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities?

Efficiency To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)?

Sustainability To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended?

Impact of Unforeseen Events on Project

To what extent was the project implementation affected/influenced by unstable, insecurity or emergency situation?

Equity and Gender Equality

Though listed as separate criteria in the original TORs, these were included as cross cutting elements across the other evaluation criteria following UNICEF review and advice.

Human Rights Based Approach

To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project?

Financial Risk Management

To what extent the project has observed and applied UNICEF‟s standard financial management practices with Implementing Partners as well as with Government?

Table 3: Evaluation Criteria and Core Questions

The evaluation design took note of established research and evaluation principles and standards such as UNEG standards, norms and ethical guidelines. These

8 These include the International Covenant on Economic, Social and Cultural Right (ICESCR), the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW), and the Convention on the Rights of the Child (CRC).

33

standards have been complied with at varied levels such as design of evaluation framework, tools, training of staff, field data collection. For instance, requesting prior consent from respondents for research tools application, assuring data anonymity and others (more details in the methodology section).

4. METHODOLOGY

4.1. Evaluation Conceptual Framework The evaluation conceptual framework has been developed while keeping in view the formative nature of the evaluation, as outlined in the TORs. The evaluatorsused the quasi-experimental researchdesign, where Pre and Post-KAP results have been compared to assess programme/approach effectiveness for quantitative result indicators. This was done with the view to assess effectiveness of PATS approach in terms of improved health and hygiene awareness, access to water and sanitation and use of improved/unimproved sanitation. However, keeping in view the formative nature of the evaluation, the quasi-experimental design developedfor the evaluation comparative analysis between control and experiment groups. Hence, the readers may not find any reference to creation of counterfactual for quantitative data analysis. The evaluation design is drawn on mixed method approach to leverage the complementarities of secondary and primary data. The primary information collection entaileduse of both quantitative and qualitative methods. The quantitative information is gathered through representative householdsurvey i.e. Post-KAP surveys and school sanitation surveys. The qualitative methods include in-depth interviews, focus group discussions, field observations and consultative workshops with key stakeholders. For data analysis, the evaluators used data triangulation techniquesto further substantiate commentary and analysis. This facilitated in measured assessment of the extent and mapping the reasons for change. Tounderstand the programme effectiveness in terms of inclusion/equity (as per UNICEF‟s MoRES framework) the quantitative analysis offers useful segregation at income levels, gender and disability. In addition to gender and equity it also examines vulnerability (including for the women, children, disabled and the elderly) and focuses on elements such as provision of subsidy and construction of demo latrines for the vulnerable,as well as on household and school wash facilities (taking into account the needs of the disabled). To map programmes contributions to larger context related changes (in comparison to the baseline), the evaluators have drawn province based comparative bottleneck analyses. For this consultative workshops have been organized at provincial level.

4.2. Data Collection Methods The data or information gathered through both secondary and primary information sources. The secondary sources included project documents, national and regional research reports, published and unpublished data, web search and review of data-bases. For primary data collection, both quantitative and qualitative methods were used. The quantitative data gathered through a representative household level Post KAP survey. For assessment of school WASH facilities, a non-representative school

34

sanitation survey was undertaken. The quantitative tools used for the SPSP Baseline were used for the evaluation, with minimal changes. The data entry was done on Census and Survey Processing System (CSPro), a specialized software for data entry. The data was then exported to Statistical Package for Social Sciences (SPSS) and MS Excel, the software used for the data analysis. The qualitative tools included IDIs, FGDs, interactive workshops, field observations checklists, and photographs. The qualitative tools were applied to a wide variety of stakeholders (directly and indirectly involved in the programme) to get their reflections around programme design and delivery. The qualitative tools were used across all regions/provinces, where at least one site/community was picked up from each province/region. Although, there are no prescribed standards for sampling for application of qualitative tools, however, the evaluators are confident that the scale and breadth,at which quality toolshave been applied, enabled to capture the depth and diversity of views in varied contexts. The evaluators have reached out to all key stakeholders (some even multiple times), which is quite reassuring. The figure below illustrates the range of research methods used for the evaluation.

Figure 3: Evaluation Data Collection Methods

4.2.1. Quantitative Evaluation Methods Post-KAP Household Survey Post-KAP survey was administered at the household level to collect representative quantitative data, which enabled to draw trends and patterns analysis with respect to changes in knowledge, attitudes and practices around water, sanitation, health and hygiene aspects. The analysis has been undertaken to capture trends for equity and gender aspects. The Post-KAP administered on 5185 households, where respondents were selected through multi-stage random sampling (refer Annex 9 for more details on sampling approach). The respondents included both adult men and women. Equal number of men and women respondents were interviewed in all districts, however in FATA only men were interviewed.

Evaluation

(UNEG, OEAD/DAC Criteria, UNICEF Core Strategies, Normative Principles)

(disaggregated by gender, income quintile and vulnerability status)

Quantitative Data

Post KAP Household

Survey

Post KAP School Survey

Qualitative Data

Secondary Data

Review

In-Depth Interviews

Focus Group

Discussions

Bottleneck Analysis

Workshops

School WASH FAcility

Observation

Checklists

35

Find below the sampling calculations, where detailed list of villages is attached as Annex 9.

Categories of

Respondents

No. of districts/

locations

Total Remarks

Male

13 + 3 (additional

FATA regions)

3189 In 3 FATA regions data

was collected only from

male respondents that‟s

why male respondents

are calculated against

16 districts/locations and

female respondents

against 10 districts and

locations.

Female

13 – 3* = 10 (*FATA

regions)

1996

Total Sample 13 5,185

Table 4: Sampling Calculations for Post KAP Household Survey

Note: The above calculations were based on 13 districts/agencies considering, the overall margin of error for target sample as 1.35% at 95% confidence level.

Post-KAP School Survey Non-representative School Sanitation Survey was also administered in 78 schools, where at least 6 schools were selected (randomly) in each district/agency. Only those schools were selected, which had received assistance to rehabilitate the WASH services. The respondents comprised school head teachers or senior teachers. The inclusion criteria for schools included equal number of boys and girls (except FATA, where only boys schools selected) schools, primary and secondary level.

4.2.2. Qualitative Evaluation Methods Secondary Data Review The evaluation involved extensive desk review of relevant documentation comprising Government Policies (PATS, NSP), UNICEF policies and programme documents (such as gender equality, HRBA, CATs, Child Friendly Schools etc), sector reports, databases, TPM reports, national and regional survey reports etc. The extensive review helped understand the programme, larger context and challenges. Also, it fed into the evaluation design especially tools development. The numeric data (secondary) was used to draw comparison with field survey findings and referenced as such (refer Annex 10 carrying details of the documents reviewed). In-Depth Interviews In-depth interviews (IDIs) were carried out with range of stakeholders primarily at national, provincial and district level public officials (MoCC, PHED, Health, Local Government, Education and others), managers of project partners (including UNICEF), non-profit organizations active in sanitation, political and community leaders, masons, third party monitors, mart owners and others.In total 36 IDIs conducted where provincial and districts level public officials were covered multiple times (refer Annex 9 – List of People Met for the SPSP Evaluation). The senior

36

members of the evaluation team administered interviews to ensure consistency and quality. The instrument design took note of generating complementary information to help validate Post KAP results and triangulate information to deepen the analysis. The focus was to establish how far and in which ways the programme interventions brought changes in engaging public stakeholders and delivery of WASH services. The sample was evenly spread across the regions (across districts/agencies in each province/area) to capture regional disparities and reflect as such in the analysis and recommendations. Focus Group Discussions Like IDIs, the evaluators administered FGDs to draw complementary primary information to deepen commentary and analysis. Varied community or beneficiary groups comprising members of village sanitation committees (VSCs), members school health club (children), community resource persons (CRPs), and School Management Committees (SMCs) participated in the FGDs. The discussions were carried out to understand the involvement, contributions and results of these efforts in changing knowledge, attitudes and practices around WASH. In total 40 FGDs were undertaken in five selected communities in as many districts/agencies i.e. one each in the provinces and an additional in the FATA agency. Please refer Annex 9, which carries the list of people who participated in the FGDs. Bottleneck Analysis Workshops The evaluators facilitated day-long bottleneck workshops, one each at provincial headquarters (four in total), in line with UNICEF MoRES framework to assess the extent change that SPSP I has been able to bring to the larger sanitation environment at provincial level. The participants included both public officials from relevant depts. and IPs from respective provinces. Find attached the reports that offer analysis as to the extent of change and causes in the larger sanitation sector (in relation to what transpired during baseline stage) including list of participants i.e. Annex 8 School WASH Facility Observation Checklists To assess the quality and consistency with child friendly WASH facility standards, evaluators developed and administered WASH facility observation based checklists. The checklists were used to assess whether the facilities met the design specifications as provided in the manuals, and establish facility usability, standard of maintenance and likelihood to sustain/remain functional. The checklists were administered in selected schools.

Find below the table that offers summary of the methods/tools used, sources/respondents and limitations.

Instrument Source Limitation

Post-KAP Household Survey Men and Women at household level from target geographical locations

Absence of Control Group Some instances of Security Incidents and inaccessibility to areas, requiring a change in sample villages.

Post-KAP School Survey Teachers and School Children No Specific Limitation

Secondary Data Review Various Reports, Assessments and Policy Documents

Dependency on the existing body of literature, data and statistics. In some cases data desegregation is not available for comparison (geographically in the context of districts or in terms of gender).

In-Depth Interviews Government Departments (Health, Education, PHED, LGRDD at Provincial and District Level),

Availability of Government Officials in all project locations for interviews during the evaluation period. Frequent change of officials during the project life.

Focus Group Discussions Children (boys and girls), VSCs (Men and Women), Entrepreneurs and Mart Owners, CRPs (Men and Women)

Availability and sufficiency of numbers to allow for FGDs. In some instances due to limited attendance FGDs had to be managed as Interviews.

Bottleneck Analysis Workshops Public Sector Partners (Government Officials) drawn together at the provincial level.

Official deputed from relevant department may not have been involved during the project cycle.

School WASH Facility Observation Checklist

Actual visits to School WASH Facilities (Evaluation Team Observations)

Checklist is focused on assessing whether provided criteria and design specifications were observed or not. In some cases requires qualitative comments.

Table 5: Summary Table: Instrument, Source and Limitations

4.3. Evaluation Constraints, Limitations and Mitigation The evaluation went largely as per plan.As happens with most of the evaluations, the evaluators faced some constraints and challenges, for which appropriate remedial actions were adopted to not compromise quality. The key constraints and challenges are listed below together with mitigation measures undertaken by the evaluators.

Constraints/Limitations Mitigation Measures

Terms of Reference: Ambiguities around evaluation key questions and sub-questions.

Consultants sought clarificationsand convened meetings with UNICEF team (including PMER section) to get clarity on key evaluation questions and sub-questions, which later were made part of the inception report.

Theory of Change: Shared documentation and programme proposals lackedan articulated Theory of Change

The consultants in consultation with relevant UNICEF staff developed Theory of Change, and used as such for evaluation.

RBM Logic: The results framework found Inconsistent with RBM principles e.g. missing outcome statement.

This could partly be attributed to gaps in the donors‟ results framework template, however evaluators designed comprehensive tools (post-KAP, IDIs and FGDs) to capture all results indicators desegregated across income quintiles, gender, disability

Creation of Counterfactual: The evaluation design excluded establishing counterfactual to assess the element of attribution.

The evaluation being formative in nature did not require establishing counterfactuals (as it mostly required for impact evaluations), hence excluded from the evaluation design. Moreover, it might have a challenge in itself to find a reliable control group, given the fact that communities (in project areas) are receiving humanitarian assistance from multiple stakeholders. .

Security Situation Unstable security environmentconstrained accessing some project sites and caused delays.

A robust security management system established with senior resources on the ground taking lead in security assessment, liaison with local authorities and taking on the spot decision around withdrawal, rescheduling and changing field sites/villages.

Gender Context: Beliefs and cultural sensitivities especially in parts of FATA constrained access to women beneficiaries.

To overcome local sensitivities, locally recruited female enumerators and FGD facilitators were involved to administer household interviews and FGDs from female beneficiaries.. However, in view of the particular context in FATA agencies, the consultants in consultation with UNICEF agreed to interview male respondents only.

Methodological & Individual Biases:

Methodological and individual (core team members and field staff) biases potentially affecting evaluation findings and analysis.

The evaluation took note to address this at all stages such as using both quantitative and qualitative methods, use of data triangulation techniques for analysis, instruments design with the intent to limit human biases, deploying experienced and trained resources for qualitative data collection. Moreover, extensive consultations within the core evaluation team and finally discussions with the Evaluation Reference Group to keep check on methodological and personal biases.

Non-availability of Some Key Project Documents: Consultants did not receive the key project documents such as expenditure statements constraining desegregated cost benefit analysis.

The consolidated expenditures statements were not available hence evaluators used the budget figures gathered from PCAs.

Table 6: Evaluation Constraints, Limitations and Mitigation

4.4. Ethical Considerations The evaluators take note of and followed the UNICEF adapted United Nations Evaluation Guidelines (UNEG), standard researchnorms andethics, for evaluation design, field data collection and reporting. Some of these include demonstration of impartiality, seeking prior consent from the respondents, ensuring data and informant anonymity. The evaluators sought prior permission to record conversations (digital recording) and for field photography. UNICEF focal points were kept informed of the progress and where required consulted during field data collection, analysis and reporting. The members of reference group were also kept informed mostly through UNICEF focal points. s.

4.5. Quality Assurance Mechanisms The ensure consistency to the accredited and agreed research and evaluation standards, the evaluators developed a set of quality assurance mechanisms for data collection, field work, data analysis and reporting. Some of those mechanisms are listed below; a. Where respondents agreed and allowed, the IDIs and FGDs were digitally

(audio) recorded and transcribed to avoid loss of field information;

b. To the extent possible, the evaluators hired the local youth/resources and

rigorously trained them before having them involved in field work. Female

local resources hired to overcome cultural sensitivities around interacting with

outsiders;

c. The field training of enumerators was conducted by senior evaluator, involved

in evaluation and instrument design. The training and contents and delivery

approach was standardized. Moreover, to ease work of field teams, field

guidelines were developed for field staff comprising District Coordinators,

Supervisors, Enumerators and Data Entry Operators;

d. To evaluators used the frequently used and reliable industry

software/packages for data entry (CSPro) and data analysis such as SPSS

and Excel;

e. 5% of the Post KAP forms were validated through field quality assurance

staff;

f. Distinguished sector professionals with complementary training and field

experience were engaged in the evaluation design, field data collection,

analysis and reporting;

g. Field application of IDIs and FGDs (the key quality tools) by senior evaluators

(some involved in design also) to ensure consistency and quality

administration;

h. Security protocols were developed and implemented throughout field data

collection, regular contact maintained within team members, field teams.

Local authoritiesthrough respective IPs and UNICEF staff.

40

4.6. Evaluation Management and Stakeholder Participation The evaluation was commissioned and managed by UNICEF Islamabad Office. The WASH Team was in the lead with technical oversight from the Evaluation Unit. The UNICEF Provincial Offices extended field level coordination and facilitated in organizing province based bottleneck analysis workshops. UNICEF teams at varied levels assisted in linking up the evaluation team with respective stakeholders (Government officials, IPs and community/beneficiary groups). A regular contact was maintained between UNICEF and evaluation team focal points. Significant time was taken during inception phase to clarify expectation, responsibilities, methodology, process and the output. The field activities were primarily facilitated by the IPs. This included scheduling interviews with local officials, FGDs with school children, VSCs, CRPs, SOs and School Teachers and application of school based observation checklist. By and large, the national and district level government officials made themselves available to discuss in detail and respond to evaluation questions comprehensively and candidly to help better inform the evaluators on their perspective of what worked well and what could be improved in the programme. At national level, UNICEF facilitated initial contacts whereas in the field IPs helped connecting to the local officials. Community was involved in various forms in the overall evaluation process, including making themselves available and giving time to the various evaluation processes and use of instruments (such as household members giving time and sharing their views and feedback in the household survey and similarly for children and school teachers during the School Survey). FGDs were organized with groups formed or supported during the project such as CRPS, VSCs, sanitation mart owners,entrepreneurs and masons.. The IPs facilitated gathering community members at common spot (mostly guest room of local influential) to help evaluators interact with respondents. The female FGDs were largely performed inside the houses. It was encouraging to note that women and girls shared their views openly. An Evaluation Reference Group was formed (refer to Annex 14 for Evaluation Reference Group TORs) comprising representatives from UNICEF, DFID, CCD and relevant provincial government departments. The evaluation team maintained regular contact and sought inputs both from the Reference Group and UNICEF‟s Evaluation Management Committee (mandated for quality assurance) through PMER Section. A two days Reference Group consultation session was organized towards the end, where the evaluation team presented the key findings, analysis, conclusion and recommendations. Over the course of the consultation session, the recommendations were refined and translated into action plans for UNICEF to incorporate into the next cycle of SPSP programme. Third Party Monitors, Apex Consultants,on the advice of UNICEF monitored and reported on compliance to agreed standards. For instance the TPM monitors participated in the enumerators trainings, observed the implementation of the Post-

41

KAP household and school surveys (by observing the conduct of field enumerators) and other field activities. The AAN consultants were responsible for evaluation design, field planning, organising field resources, training, logistics and security management. The firm was also responsible for conductinga professional evaluation, and producing high quality evaluation report.

Constitution of Pakistan does not recognise sanitation as a right yet, however the national sanitation policy recognises that right to health cannot be achieved without sanitation. (http://www.washwatch.org/southern-asia/pakistan) Following SACOSAN 4, Pakistan (including other SAARC Countries) adopted Colombo Declaration which commits governments to legislate and make the right to sanitation a part of the constitution.

5. FINDINGS This section of the report offers key findings and analysis. The description is structured around key evaluation criteria and questions.

5.1. Relevance Evaluation Q1: To what extent the programme design and implementation was relevant to national/provincial priorities and needs of men and women beneficiaries in terms of achieving expected results across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have evaluated the relevance of the programme against a range of aspects including, relevance to international commitments, national and provincial policies and priorities, to the needs of men and women beneficiaries including the vulnerable, to total sanitation approaches and also to the integration of gender equality, equity and disability. Government’s International Commitments The programme objectives, selection of approach and geographic prioritization all point to relevance to Pakistan’s international commitments such as MDGs and SACOSAN.The relevant MDG sets the target to achieve 67% sanitation coverage by 2015, which at present stands at 50%. The programme was planned to increase sanitation coverage in the target villages and communities and successfully brought to 65% (SPSP Evaluation Post KAP results - refer to Figure 4).

The programme also conforms to the Kathmandu Declaration issued at the end of SACOSAN V, where Pakistan pledged to contribute to the ODF status for South Asia by 2023. The SPSP made a noticeable contribution by having successfully reached to 1.2 million people (men, women, boysand girls) with demand creation for sanitation messages and increasing

67%

50%44%

65%

0%10%20%30%40%50%60%70%80%

MDG Target Current National Coverage

SPSP Baseline SPSP Evaluation

Proportion of Population having Access to Sanitation Facility

SPSP Contribution to Sanitation Target

Figure 4: SPSP Contribution to Sanitation Target

43

SPSP contributes to the following WASH related CCCs: Commitment 3:Children and women access toilets and washing facilities that are culturally appropriate, secure, sanitary, user-friendly and gender-appropriate; Commitment 4:Children and women receive critical WASH-related information to prevent child illness, especially diarrhoea; and Commitment 5: Children access safe water, sanitation and hygiene facilities in their learning environment and in child-friendly spaces.

the number of people living in ODF villages by 711,467. UNICEF Global and National Commitments Programme objectives also have close relevance to UNICEF’s Pakistan country programme commitments and to its global Core Commitments to Children (CCCs). The objective isto ensure children and women access to safe, sufficient and reliable WASH facilities. UNICEF‟s work in Pakistan is guided by cooperation programme for 2013-2017, aimed at improving the lives of children and women in Pakistan. WASH is reflected as one of the priority areas for UNICEF and interventions are guided by UNICEF CCC and United Nation Development Assistance Framework i.e. UNDAF. Hence, it could be argued that SPSP I demonstrates coherence to organizational commitments and standards and larger UN mandate and priorities. National and Provincial Policies The programme has relevance to the National Sanitation Policy 2006 and draws on the Pakistan Approach to Total Sanitation (PATS) 2009. The programme design is found consistent with PATS approach, which is integral part of both national and provincial sanitation policies, even if available in draft form. The PATS approach places communities (beneficiaries) at the centre of driving and sustaining changed sanitation behaviours. The programme is strategic in terms of rolling out PATS and engaging the relevant public sector stakeholders selectively to work together with civil society to better understand PATs principles and components, and in the due course get prepared forcomplete ownership, replication and scale-up. Particular components and principles were found aligned to other relevant national level polices e.g. Environment Policy 2005, Drinking Water Policy 2009 and Climate Change Policy 2012. In the wake of devolution of public functions and resources to provinces after 18th Constitutional Amendment and7th National Finance Commission (NFC) respectively, provincial WASH policies and plans have either been approved or being finalized. Across all provinces, PATS remains they key policy priority for sanitation services, which again points to consistency with the evolving policies within provinces. Sector Coordination and Harmonization Activities The programme contributes/informs the UNICEF lead and supported sector coordination and harmonization efforts with government in the lead. The focus is relatively more at national level coordination compared to provincial and district levels. however is more national with limited.For instance,UNICEF organizedSector

44

Partners Workshop in June 2013, which brought the stakeholders together and event was used to showcase UNICEF work and findings of the RuSFAD III Evaluation and SPSP I Baseline. There are multiple thematic coordination forums, mostly at the national level where UNICEF is represented and participates regularly. UNICEF leverages its presence to share learning from SPSP and other programmes to inform sector discussions and policy decisions. The forums available are as under (some have become dormant also);

National Drinking Water & Sanitation Committee;

National Sanitation Policy Implementation Committee; and

PACOSAN

CLTS Core Group under PATS (UNICEF is one of the members) As far as harmonization of efforts is concerned, neither federal government nor UNICEF is actively involved to facilitate harmonization ofWASH stakeholders and activities. The situation is more or less the same in province with exception of Punjab, where Planning &Development department has recently formed a Provincial Sanitation Committee to guide and coordinate the sanitation sector interventions. For programme interventions, the evaluators did not find evidence as to allocation of resources and active contributions by relevant district authorities such as contributions to rewards, though envisaged in the programme design. This may be the result of limited guidance by UNICEF provincial offices and lack of focus by IPs on mobilizing resources from concerned local officials. This may need a different delivery approach by IPs and considerate guidance and support from UNICEF provincial offices. Theevaluators suggest that the programme roll-out in following years must prioritise and set guidancefor formal coordination at national, and more importantly provincial and district levels. The relevant departments such as PHED or LGRDD (as varies in different provinces) must be given due support to take initiative for harmonization of efforts and resources of WASH stakeholders. The provincial level coordination and harmonization by lead public agencies may create enabling environment for IPs to work with district officials and cultivating greater sense of ownership amongst public agencies on ground. Total Sanitation Approaches The integration of SLTS and CLTS approacheswhere SLTS is seen to contribute to the continuity of WAH work in communities, found consistent with both PATS and CATS. The simultaneous application of SLTS and CLTS was found complementary and contributed to reinforcing the message.The provision of subsidy to poorest, though appear relevant on equity grounds, together with construction of demo-latrines, remain the major departures from the CATS/PATS essential elements. Moreover, the use of pride, respect and dignity as integral to triggering appear another departure from the conventional approaches. Overall, the use of SLTS and CLTS contributed to complementing project interventions and also reaching out to all family members, hence resulted in reinforcing the message at household level.

45

CATS Essential Element 4: Subsidies – whether funds, hardware or other forms – should not be given directly to households. Community rewards, subsides and incentives are acceptable only where they encourage collective action in support of total sanitation and where they facilitate the sustainable use of sanitation facilities. CATS Essential Element 5: CATS support communities to determine for themselves what design and materials work best for sanitation infrastructure rather than imposing standards. External agencies provide guidance rather than regulation. Thus households build toilets based on locally available materials using the skills of local technicians and artisans. UNICEF Field Notes – CATS

FGDs and feedback from IPs highlights several challenges around provision of subsidies 3% and 7% subsidies to the vulnerable and extremely vulnerable respectively. Some relate to distinguishing between vulnerable and extremely vulnerable, which at places caused frictions in the community. Some IPs referred to vulnerable being so poor (or in some case lack of interest or commitment) left the latrines incomplete (as IPs supported latrine construction except super-structure). Some even not used at all and most being unusable anymore for lack of cleaning.The future programme design may need to take more considered approach with this. Use of„Pride, Self-Respect and Dignity‟instead of conventional „Shame, Shock and Disgust‟ approach remains another key departure. The evaluator found it relatively more consistent with rights based programming, which remains cardinal to UNICEF programme design. The post-KAP results point to greater dependency amongst respondents, no matter what economic strata they belong. This is evident from survey results where overall numbers for people wiling to construct latrines have increased with a corresponding increase in people asking for external assistance/cost sharing. It is difficult to establish in concrete terms what may have contributed to this, however apparently it may have been for project subsidy/demo latrines and larger environment where non-profit organizations have been distributing free of cost latrines. The consultants propose review of subsidy delivery mechanics with greater involvement of VSCs and communities to decide about subsidies and rewards. One option could be to transfer funds to these community forums to prioritize localized interventions, which may very well include latrine construction for poor and extremely poor households. Masons were trained, as part of supplies facilitation to help improve supply of trained and locally available masons. The evaluators noted departure from CAT application which emphasizes engagement of local skills (to support local livelihoods) whereas the programme required IPs to engage contractors (not necessarily local) for hardware components such as rehabilitation and construction of school and health facility latrines. The future programme design may need to take note of it in future Community Priorities and Programme Area Selection The programme areas demonstrated greater deprivation in terms of poverty, insecurity and gaps with respect to sanitation coverage in comparison to urban areas. The programme areas found falling far below the national targets set for MDG

46

Goal 7. Moreover, districts illustrated low scores on UNICEF’s Child- Wellbeing Index, which as per UNICEF contributed to their prioritization. In essence, programme coverage demonstrates coherence to larger programme goal/aim. The multiple information sources such as Joint Monitoring Programme (JMP 2012), Pakistan Social and Living Standards Measurement Survey (PSLM) and UNICEF Multiple Indicator Cluster Survey (MICS), all show the selected districts being low on WASH indicators. The Fig # 5, offers comparative analysis for MICS and SPSP pre-KAP (baseline) results, and illustrate the significant gaps that exist in terms of access to sanitation in urban and rural areas. The baseline results indicate these all largely being the poorest districts across Pakistan. The numbers indicate that by and large the programme prioritised districts, which had low sanitation figures including poor and security affected. Moreover, the interaction with both public officials and communities (varied groups) in programme districts point to sanitation being a priority need, particularly for women, children and poor.

The UNICEF team shared that as children focused organization, they selected districts based on child wellbeing index, which points to the larger deprivation of these districts. Gender Equality, Equity & Disability Integration The programme design and implementation demonstrate a clear focus on gender integration and targeting the most vulnerable i.e. equity. In comparison, the focus and the demonstrated integration for disability have been far less e.g. in BCC, latrine designs and others. The programme did demonstrate incorporation of gender equality at varied levels. These include gender being a key element in defining vulnerability, thus ensuring access to subsidized sanitation inputs. The women and girl centred BCC components supported by women staff and volunteers. Moreover, creation of men and women community groups, which demonstrates recognition that women are as important to take this forward as men. The collection of gender desegregated data during baseline,evaluation and during monitoring to track progress on gender equality. Women were part of the delivery apparatus, where field staff had healthy women representation.

96%

72% 77%

91% 93%

38%

64%

26% 29%

68%

33%22%

50%45%

61%

0%

20%

40%

60%

80%

100%

120%

Sindh Punjab FATA Balochistan KPK

MICS Urban MICS Rural SPSP Baseline

Figure 5: Province Based Access to Sanitation Facility

47

As described above, UNICEF justifies inclusion of subsidy (a departure from PATS and CATS) for equity consideration, to support and enable poorer segments to have access to sustainable sanitation. The poverty context in rural Pakistan and programme focus on rural poor, to a degree justifies the provision of subsidy, despite the fact that this at times creates frictions (in community) and element of dependency amongst communities (as results of post KAP illustrate) to seek outside assistance for construction of latrines. The programme design refers to activities being designed and implemented with due regard to addressing the needs of the disabled. The elements assessed such as BCC, triggering and education and awareness were all found less disability informed. Similarly, neither prescribed latrine designs and products available, appeared disability friendly. The design and delivery of future programme may need to demonstrate greater focus on disability integration particularly with respect to BCC campaign, latrine designs and products. Theory of Change The programme documents did not have an articulated ‘Theory of Change (TOC)’. However, as part of the evaluation, a TOC was documented by holding repeated consultations with relevant UNICEF staff. The TOC found largely consistent with PATS and SPSP results framework.

The IPs and Third Party Monitors (TPM) shared that the programme documents may not have an expressed TOC, however it was verbalized during the Inception Workshops while explaining the programme implementation plan. This helped to create a common understanding between UNICEF and the IPs on inputs, outputs, outcomes and possible challenges and risks. This to a degree helped delivery of programme in line with verbalized TOC, where some incidences of variations noted amongst IPs. Key variations noted around departure from the set sequence to roll-out BCC. UNICEF is advised to either incorporate the TOC in the programme concept note (even if it does not mean submission of revised concept notes to DFID and other (existing and potential) donors) and ensure incorporation of TOC in IPs PCAs. This shall facilitate common understanding of results, approaches, interventions, inputs and risks and minimize implementation level variations, unless essentially required for context reasons..

Overall, the programme design found largely relevant and consistent with national

and international commitments and policy frameworks including those of UNICEF.

The geographic prioritization found extremely relevant, as evident from the low

progress vis a vis MDG 7 targets. By and large the design found consistent with

PATS and CATS approaches, with evident departures from key conventions such as

provision of subsidies, non-engagement of trained masons/skilled workers and

others. Both the design and delivery merit re-consideration to achieve consistency

with principles of CATS and PATS. The evaluators found an expressed focus on

equity and gender equality, however with relatively less focus on disability

48

integration. The programme documents lackan articulated TOC, which to a degree

compensated by its verbalization during inception phase.

5.2. Effectiveness Evaluation Q2: To what extent were project implementation strategies effective and successful in achieving the planned outcomes/results for rich and poor men and women across all socio-cultural groups including the vulnerable in the targeted communities? The evaluators have assessed effectiveness of the programme across varied programme aspects, i.e. design, programme pillars/interventions and strategies (demand creation, supply facilitation, health and hygiene promotion, linkages with government, attainment of total sanitation, knowledge management and accountability) as well as effectiveness of integration of gender equality, equity &disability.

5.2.1. Programme Design The programme achievements as evident from comparative analysis of pre &post KAP results, point to effectiveness of PATS/CATS especially the parallel implementation of SLTS and CLTS. This helped leveraging complementarities and to a degree has planned to set foundations for continued community mobilization through schools. UNICEF used indirect implementation approach, whereby it contracted International and national non-profit organizations to lead implementation. Similarly, arrangements were made with relevant public agencies to work closely with these IPs. The implementation modalities did help with efficient delivery, however limited engagement of public agencies remains a concern (at both provincial and district levels). This may affect having willing and capacitated public sector for PATS replication and scale-up. UNICEF and other development partners such as WSP, WSSCC, WaterAid etc, are advocating PATS adoption at varied levels across the country. There are developments that point to the success of advocacy such as increased budgetary allocations for FATA, selective rollout of PATS in Punjab and others. This, however cannot be attributed as the result of UNICEF efforts, as there are other stakeholders involved in advocacy. The evaluators did not find reference to stakeholders leveraging there engagement and resources to piece together a collective advocacy plan to maximize impact. UNICEF may need to prioritize this for subsequent cycles of SPSP rural. UNICEF contracted specialized partners for action research, third party monitoring and centralized training, which largely proved effective.DSI representative, the partner contracted for action research shared that there are several key points that UNICEF can integrate in the next phase. For instance,broadening the scope of BCC materials by adding multi-dimensional focuse.g. emphasizing sanitation vis a vis health, privacy, religion and others. Moreover, providing entrepreneurs with business and motivational training. APEX Consultants, the TPM worked as per the framework available, provided real-time monitoring information to help UNICEF keep track of progress and make informed decisions. Centralised training by RSPN was effective in terms of standardization, deployment of dedicated teams for capacity building across different functions and layers. The RSPN staff and IPs both pointed to gaps in implementation mainly for overstretched RSPN staff. This may have been because

50

of under-estimation of human resources required on the part of RSPN. The training materials/manual produced shall be available for future SPSP cycles. The programme design offers expressed focus on integration as to gender, equity and disability. For some elements the inadequacies as to operational guidelines and standards (PCAs offer limited information) resulted in in-consistent implementation across sites and partners. For instance, a girl school visited did have elevated latrine including water points, where the students and teachers complained that as the boundary wall is low, they feel more exposed to people outside whenever they move up either to collect water or use latrines. This may have helped securing the WASH facilities from floods, however in absence of elevated walls, they feel that their privacy has been compromised. The future programme design and delivery may need to consider this and couple elevated WASH structures with raising boundary walls, particularly at girl schools. Similarly, for equity integration household subsidies were provided, however identification of vulnerable and extremely vulnerable individuals was done during the PRA exercisescarried early on, which caused friction at later stages. There are indications that the subsidy provisions are working to discourage communities (even the well-off) to construct latrines on their own, as evident from the survey results (shared above). For disability integration, the WASH manual (latrine designs) provides guidelines, however evaluators did not find much evidence suggesting the disability integration for facilities restored at schools, communities and health centres. The evaluators feel that ambitious targets were set especially with respect to time allocated. The partners were stretched further for delayed start, owing to PCA approvals and seeking field level authorization from district authorities. This left not more than 9 months (on average) for actual field implementation, making it even more ambitious. The field interaction with communities points to design and planning inadequacies around taking note ofand adjusting activities to the community work cycle (e.g. harvesting activities, closure of schools during summer holidays), despite the fact that seasonal calendars were prepared through PRA. Thiscontributed to IPs circumventing (to the extent of compromise) the laid processes to save time, especially capacity development, post ODF monitoring and others. Thisneeds serious consideration from UNICEF to ensure that each IP gets at least 12 months of implementation time. For lack of clarity as to what it entails (from UNICEF) forthe first programme component or pillar i.e. coordination with public stakeholders, the IPs used varied interpretations and implemented whole range of different activities. The evaluators did not get more clarity from UNICEF either as to what it entailed. If it meant more structured and systematic engagement (with respect to contributions to rewards, increased budgetary allocations etc) with local and provincial public agencies, the results fall short of expectations. The evaluators noted gaps in communication chain (both at horizontal and vertical levels), again probably result of un-explained pillar. The provincial PHED which supposedly is the lead agency in Punjab communicated being out of reporting loop, hence being unaware of the field progress.

51

UNICEF appear to have consulted stakeholders particularly public agencies, however the extent to which these consultations contributed or informed programme design was difficult to ascertain. The other stakeholders like WSP and WaterAid did refer to being aware of the programme, however none referred to having been consulted for the design. UNICEF apparently guided and supported IPs for the design of PCAs, however how much have IPs contributed to the design is unknown. The evaluators view is that a programme like SPSP with a range of partners and of significant magnitude, may need to evolve mechanisms to foster consultation and wider ownership by the stakeholders. This would surely have bearing on both the design and implementation.

5.2.2. Programme Management & Coordination Overall arrangements for programme management and coordination were found appropriate and relevant to the natureand scale the programme. The timely completion and achievement of results point to these being effective also. UNICEF WASH Unit based at Islamabad worked as the lead management and coordination point. Support was provided by the UNICEF Provincial Offices for coordination with public stakeholders, oversight of province level IPs, field monitoring and troubleshooting. Overall the decisions were drawn in time to facilitate timely completion, however with few exceptions such as withdrawal/disengagement from loans and grants to entrepreneurs. The IPs shared that they were given sufficient flexibility, which they appreciated given the diversity of contexts. IPs referred to UNICEF being supportive through the implementation. Also, that they were able to communicate with confidence and freely, and received timely guidance and support with any issues or bottlenecks highlighted. For instance, an IP in Sindh shared that demo latrine cost revised upwards as were found to be under-budgeted in the original budget. The umbrella arrangements as used in Sindh added an extra layer with raised operating costs, however evaluators were unable to find a clear reference or evidence to the value addition of the umbrella partner. In absence of any significant value addition, UNICEF is advised to re-consider umbrella arrangements. More analysis around this is available in the partnerships section. The practice of monthly substantive monitoring reports helped keep track of progress, take note of deviations and constraints and take necessary management decisions for course correction. , The evaluators were told of some of key challenges that programme faced, which are as below;

i) The TPM were contracted centrally by the Islamabad Office and relationship featuring direct reporting to WASH Unit Islamabad completely excluding the provincial UNICEF offices. The review of the monitoring framework (templates and tools available to TPM) indicates lesser flexibility and scope to highlight and incorporate the operational constraints. This IPs and UNICEF provincial offices referred to the process being rigid and less sensitive to the field context they operated. The

52

UNICEF provincial offices raised concerns around being out of reporting loop for TPM, which UNICEF Islamabad office may need to look into. The consolidated versions could be shared with provincial offices for information purposes.

ii) UNICEF (both at Islamabad and provincial offices) could have played more effective role with public sector advocacy had more structured process followed for pubic sector coordination at varied levels. The limited success with budgetary (provincial) increases, contributions by public sector into rewards and others, demonstrate UNICEF and in particular provincial offices may need to take more hands on approach for public sector coordination. This impacted IPs engagement with district level public stakeholders and level of ownership of the programme by PHED and LGRDD in districts.

iii) For lack of systematic and structured coordination, the approvals for implementation by the district authorities got delayed invariably for all partners. This delayed the project kick-off thus stretched the IPs resources further to deliver on committed results. Once such example is the NOC for Muslim Aid in DG Khan, which never got resolved. Moreover, the relationship and terms of engagement (including the focal agencies) between IPs and district governments kept on evolving. Some IPs were lucky to find receptive focal points, thus were able to mobilize public sector support, whereas for others it was struggle all along.

5.2.3. Programme Components&Results Note:The results framework lacked outcomes statements and indicators, hence evaluators used outputs indicators (which in strict results term could be equated to outcomes) to map the extent of change the programme implementation brought (by comparing pre and post KAP results).

Output 1: Demand For Sanitation SPSP used both CLTS and SLTS approaches, which helped to sensitize and educate communities and thus resulted in increased demand. The results are indicative of the approaches complementing each other which was supported further by the cyclic BCC campaigns. The objective to reduce practice of open defecation, raise community’s awareness of the need to achieve ODF status in their villages, and to create demand for sanitation facilities by the target population, all showed encouraging upward change towards the end of the year 1. Community Mobilisation The decision to engage RSPN (to help train SOs, CRPs etc) and draw on RSPs experiences and approaches for community mobilization (in use for over three decades) was well-considered and proved effective. The results in Table 5 points to IPs exceeding targets for most of mobilization and sensitization indicators.

Community Mobilization Activities & Training Target Actual

SOs Training for PATs (participatory tools, IEC material) 8 85

53

CRP Training on Conducting CLTS 46 67

PATS Facilitation Session by SOs in Villages 230 911

CRPs facilitation session on CLTS 1,385 3,471

VSCs Formed and Trained 1,385 2,994

Community Action Plan (CAP) Developed 1,385 1,878

Table 7:Community Mobilisation Activities & Training

Generally the feedback has been positive on the quality, duration and content of the RSPN Social Organisers (SOs) training. The SOs shared that the training helped them get prepared for the expected work, however highlighted that one extra day for field work (practical demonstration) may have added more value. Moreover, they shared that the scale and frequency (considered excessive) of mentoring activities and refreshers (to be undertaken by RSPN) affected SOs mobilising schedule. Most of the SOs were found satisfied with the workload, however for being of time for delayed start, they were left with no option but to execute their field plan, even if at times it clashed with community activity cycle (rural).The availability of both men and women SOs were appreciated to help overcome the gender divide. Again, the programme performance found encouraging where the IPs exceeded almost on all targets (refer Table 9).

School Led Total Sanitation Training and Events Target Actual School Teacher Training 23 84

WASH Clubs Formed in Schools 692 881

Events Conducted by School Wash Clubs 886 1,433

Boys and Girls reached with sanitation messages in Schools. 103,000 176,022

Table 8: SLTS Training and Events

Behaviour Change Communication (BCC) The post KAP results and interaction with stakeholders involved including communities point to the effectiveness of BCC strategies and interventions. The key factors that appear contributing to the success are i) use of a variety of mediums targeting varied groups (IEC material, theatres &mass media) and; ii) repetitive rounds of multiple activities. The helped targeting different intended population groups with varied messages and mediums (matching the information preferences and interests) and in a way re-assured that message gets across to everyone at least once. The involvement of opinion leaders such as Masjid Imam (prayer leaders through Friday sermons), health workers, teachers further strengthened the communication. Refer table 7 for details. The reports indicate IPs being able to meet the targets for most of the activities. BCC Campaigns Outreach Target Actual

Children (boys and girls) reached with demand creation for sanitation messages in Schools. 103,000 176,022

People (men, women, boys, girls) reached with demand creation for sanitation messages 776,000 1,200,529

People reached with demand creation for sanitation messages through electronic/print media. 970,000 540,786

54

People reached with demand creation for sanitation messages through community events (theatres, puppet shows) 291,000 195,450

Table 9: BCC Campaigns Outreach (Target versus Actual Number of People)

The effectiveness of BCC strategy is evident from the number of people claiming to have received the health and hygiene messages in last one year. The KAP results indicate that this has increased from 18% (during baseline) to 59% (pre-post KAP results). This shows that within the span of 8-10 months, the number have risen from 1 to 3 (for every five) people having received at least one health and hygiene message. The effectiveness of the mixed medium and repetitive BCC campaigns contributed to a significant proportion of respondents claiming to have received information from one or more sources (36%, 37% and 27% claimed to have received information from one, two and three sources respectively).It means that those who received messages, 2 out of three had received from two or more sources (refer Figure 7).

The awareness and knowledge factor (in terms of being able to remember and recall messages) has also been effectively improved from 71% to 98%. The most significant is the change in behaviours, where 92% respondents claimed that they have adopted the practice (following receipt of message) compared to 53%, at baseline (refer to Figure 8).

One Source, 36 Two Sources, 37

Three Sources, 27

Number of Sources of Health & Hygiene Messages

One Source

Two Sources

Three Sources

Figure 6: Number of Sources of Health & Hygiene Messages

18%

71%53%59%

98% 92%

0%20%40%60%80%

100%

Health & Hygiene Message Received

Remember H&H Messages Practice H&H Messages

Health & Hygiene Awareness Messages

Figure 7: Health & Hygiene Awareness Messages

55

Some elements of the BCC campaigns have been appreciated and being practiced more than others. Once such example is the 10 Steps of Hand Washing. The Figure 9 illustrates that 93% respondents were able to remember/recall to health and hygiene specificmessages compared to 67%. Similarly 66% and 40% (from 56% and 24% respectively) of the respondents were able to remember/recallmessages about significance of clean and safe water and water borne diseases. .

The number of people who could recall three key hygiene messages doubled over the course of year (refer Figure 10).The Punjab province is in the lead followed by Sindh and Baluchistan.

The BCC strategy has also been effective in delivering health and hygiene messages at schools, as noted in Figure 11. 77% schools confirmed having received health and hygiene sessions over the last six months, whereas 41% schools reported to have had sessions conducted in-house by the school teachers. Gender focus in terms of coverage of boys and girls schools also comes across as sufficiently balanced.

77%

41% 47%27% 27%

59%

9%31%

0%20%40%60%80%

100%

Imp

lem

enti

ng

Par

tner

Sch

oo

ls

Bo

ys

Co

Ed

uca

tio

n

Gir

ls

Bo

ys

Co

Ed

uca

tio

n

Gir

ls

Implementing Partner School

Source of Health and Hygiene Messages in Schools

67%56%

24%

93%

66%

40%

0%20%40%60%80%

100%

Personal Hygiene Clean And Safe Water Water Born Diseases

Health & Hygiene Messages that the Respondents Remembered

Health and Hygiene Messages that the Respondents Remembered

Baseline Evaluation

Figure 8: Health and Hygiene Messages that the Respondents Remembered

24%43%

23% 26% 23% 27%

80%64%

48% 46%33%

51%

0%

50%

100%

Punjab Balochistan Sindh KPK FATA Total

Proportion of Population Recalling Three Key Hygiene Messages

Baseline Evaluation

Figure 9: Respondents Able to Recall Three Key Hygiene Messages

56

Health and hygiene topics covered through school sessions, wheremessage around personal hygiene were appealed the most. This is evident fromthe fact that 73% school teacherswere able to recalling delivery of this message, followed by environmental and domestic hygiene i.e. 69%.

In terms of effectiveness of BCC mediums, face to face community sessions bythe SOs and the CRPs were referred to as being most effective in terms transfer of knowledge. The respondents shared that these were more useful as it provided an opportunity for them to interact with the facilitators, ask questions and get response for their queries. It was also noted by a range of stakeholders (SOs, children, VSC members and CRPs) that theatres, cricket matches, key events celebrations (including world water day, environmental day and hand washing day) and use of mass media, was also effective though with varying degree of results and responsiveness. These elements need to be kept and further strengthen in the subsequent years. The BCC campaigns proved effective in terms of making people understand and recall hazards of open defecation.Similarly, a significantincrease is noted in terms of better awareness as to the types of hazards associated with open defecation, Figure 13 highlights the proportion of post-KAP respondents, who remembered and were able to recall various hazards related to open defecation.

14%34%

54%

24%10%

54% 62%77%

47%

18%

0%20%40%60%80%

100%

Awareness Message about

Hazards of Open Defecation

Dirty Surroundings Diseases Environmental Problems

Health Issues Related To Women

Hazards of Open Defecation

Awareness of Hazards of Open Defecation

Baseline Evaluation

Figure 10: Source of Health & Hygiene Messages in Schools

73%

58%

36%

69%

42%

42%

0% 10% 20% 30% 40% 50% 60% 70% 80%

Personal Hygiene

Clean & Safe Water

Water Born Diseases

Environmental & Domestic Hygiene

Safe Disposal of Human Excreta

Solid Waste Disposal

Figure 11: Topics Covered during Health & Hygiene Sessions in Schools

57

The extent of change in terms of respondents being able to realize and communicate desire to make their settlement/community ODF, points the success of BCC. Results indicate significant increase across all provinces (refer Figure 14).

Another measure of success is the overall decrease in number of people defecating in open, refer Figure 15.The numbers have come down from 68% to 47% overall, with Punjab and Sindh showing the most significant changes. The results are negative for FATA, which need to be probed further.

Figure 12: Awareness of Hazards of Open Defecation

14% 14% 24% 25%1% 10%

61% 73% 93% 84%32% 39%

0%50%

100%

Total Punjab Balochistan Sindh FATA KPK

% of targeted population aware of the need for achieving ODF status in their settlement

Proportion of population aware of the need for achieving ODF status in their settlement

Baseline Evaluation

Figure 13: Awareness of the Need for Achieving ODF Status

83%

56%

85%

47%

70% 68%

44% 39% 35% 34%

76%

47%

0%

20%

40%

60%

80%

100%

Punjab Balochistan Sindh KPK FATA Total

Proportion of Population that Practices Open Defecation

Baseline Evaluation

58

Another measure of success comes in the form of increasein numbers for respondents willing to construct a latrine due to the hazards associated with open defecation (refer Figure 16). The respondents willing to construct a latrine has increased from 61% (baseline) to 86%. Majority of those referred to cost-sharing arrangements as preferred option to construct latrines, whereas the numbers for self-construction have reduced from 18% to 10%.

As highlighted above, this trend could be for subsidy factor or for the provision of free latrines by other stakeholders (particularly after floods). However this merits further exploration as to whether household subsidy is playing a positive role in increasing the sanitation coverage or creating a dependencyfactor.UNICEF is advised to explore alternative methodsfor incorporation of equity principles especially in view of the excessive poverty in rural Pakistan. The BCC campaign faced some constraints particularly around short time-frame (for implementation), timely production and delivery of IEC materials and sequenced implementation of BCC campaigns. The IPs underlined that delayed delivery of IEC materials shortened the time for complete BCC campaigning. This pushed to have additional per week time engagement with the communities (also affected by community‟s unavailability during the harvesting season), and accelerated delivery to a degree lead to information overload and reduced absorption time for beneficiaries. Within the mediums used, theatres performances though liked by children and adult men, excluded women as in most cases these were organized at cluster (village) level, where participation required considerable travel – a constraining factor for women for limited mobility. The gender exclusion also constrained mixed gathering for theatres. The next phase may need to consider easier access and if possiblemore privacy for women (may also consider women exclusive theatre performances).

Figure 14: Proportion of Population Practicing Open Defecation

61%28% 25% 23%

86%

10% 20%62%

0%20%40%60%80%100%

Yes Self-finance Loan Cost-Share Basis

Willing to Construct a Latrine Due to Hazards

of Open Defecation

Method of Bearing the Cost of Construction

Willingness to Construct a Latrine (Including Methods of Cost Bearing)

Baseline Evaluation

Figure 15: Willingness to Construct a Latrine (including methods of cost bearing)

59

The engagement of religious leaders and institutions proved useful in adding the religious element that appealed to the religious groups,a significant majority in the rural Pakistan. School health and hygiene sessions (plays, puppet shows and others) were found effective in terms of packaging messages in child friendly manner. The use of mass media was relatively less, which may need to be looked into for future.

Output 2: Supply Side Interventions Overall supply side interventions remained effective in terms of meeting the increased demand for supplies. Some of the interventions could have proven more effective given greater clarity and guidance such as involvement of trained masons for hardware (project supported) components, loans and grants for entrepreneurs and marts. The range of supply facilitation interventions included training of masons (for better design and low-cost sanitation solutions) training of entrepreneurs and sanitation mart owners (to encourage business activity and ensure sustained and subsidized supply of materials near community) , and construction ofdemonstration latrines for both demonstrative and equity purposes (byproviding subsidy or full costs for vulnerable and extremely vulnerable families). Moreover, it included rehabilitation of child friendly WASH facilities at selected schools and at health service centres.

60

In Kohat a local mart owner said that he was dealing in cement, sand and bricks. After the entrepreneurs training, he stocked supplies and was able to sell items for approximately 150 latrines through the voucher system. From this he was able to purchase more latrine supplies and sustain the sanitation business. However, he was of the view that availability of loans/grants would have been helpful, given village level vendors are generally cash-strapped and capital to invest in new products is not available. He also felt that IPs should have focused more on creating demand and publicising their sanitation business to the communities.

Training of Entrepreneurs. Sanitation Mart Owners and Masons, The RSPN developed training contents and trained entrepreneurs and sanitation mart owners, through Master Trainers. The IPs and trainees both point to ineffectiveness of the training, mainly for weak training contents and limited facilitation capacities of Master Trainers - lack of understanding of marketing and supply chain issues. Realizing the gap, UNICEF contracted DSI to develop contents and train the trainees through refreshers(refer results in Table 11). These were apparently more effectivein articulating expectations. The key learning from the experience include a full day training and training/sensitization of relevant IP staff to enable productive engagement (post training) with the entrepreneurs and marts.

Training Events for Mason and Entrepreneurs Actual Target No. of masons trained on sanitation construction and using their learnt skills

692 1,250

No. of entrepreneurs/manufacturer trained and producing low-cost sanitation solutions 692 469

No. of facilitated sanitation marts doing profitable business 346 82

Table 10: Supply Side Training Events for Masons and Entrepreneurs

DSI also established a business helpline for entrepreneurs and sanitation mart owners for all 13 SPSP programme districts as part of active research and support. The helpline was launched to provide the entrepreneurs with the opportunity to seek business related advice from DSI SanMark expert and engineers. The records indicate that maximum number of calls were received in October and November 2013,mostly regarding the withdrawal of loan / grant by UNICEF and IPs, financing options, andsupply and pricing issues.This service was considered by entrepreneurs and sanitation mart owners to have been effective in helping them with teething problems of setting up the marts. Masons trainings proved relatively less effective for largely being informed of the training contents, before taking up the sessions (as DSI research). The programme may have benefitted more in case it had employed the trained masons to rehabilitate the school and health WASH infrastructure. This omission runs against the CATS principles. Financial Support for Entrepreneurs and Sanitation Marts The programme envisaged supporting entrepreneurs and marts with loans, which was not appreciated much. More interest was shown in grants, which ran against UNICEF rules prohibiting use offunds for setting up businesses. In view of the resistance, the loan element was withdrawn, which impacted negatively. As the component was withdrawn during implementation, where some

61

provinces were found have had advanced into implementation. For those, UNICEF created modality of vouchers and distributed to the most vulnerable/vulnerable groups to get supplies from the supported marts and entrepreneurs. This to a degree proved useful in terms of localized availability of supplies and at subsidized rates. With programme closure the voucher system has stopped, and this shall have affect on their businesses. The component needs deeper analysis and more innovative approach to facilitate both the users and the business involved. UNICEF could leverage the findings of DSI action research. Design Options and Manuals for Low-Cost Sanitation Facilities UNICEF engaged DSI to help develop low cost latrine designs. DSI a consulting company (for profit) and usedHuman Centred Design (HCD) process to developed latrine design, which are as below (refer toTable 12) Type Description Cost Factor (maximum)

Latrine Type A Ventilated Brick-Lined Direct Pit Latrine

PKR 4,000

Latrine Type B Pour Flush Off-Set Lined Pit Latrine

PKR 7,000

Latrine Type C Pour Flush Latrine With Septic Tank

PKR 10,000

Table 11: Types of Low Cost Latrines

For field interventions, another manual i.e. Latrine Construction Manual, was developedfor sub-engineers, masons and local facilitators in local (Urdu) language. This manual was used for field training and for project supported hardware interventions. The document was found comprehensive and incorporated disability and age considerations. The evaluators did technical assessment of the manual and deeply looked at constructed latrines, which have outlined below as key observations/comments.

The latrine construction manual is comprehensive,however it be more useful if additional information regarding latrine super-structure options as well as low cost options for pit lining (using local materials) could be added.Translation in Urdu contributed to improved delivery and understanding of trainees.

DSI‟s report lacks design recommendations related to disabled/elderly persons needs. The manual however covers these elements comprehensively. It appears that the rehabilitation work overlooked the manual prescriptions around designs.

Communities (in hot zones) referred to latrines being small in size, low roofand low walls, hence lead to discouraging some for fear or perception of lack of privacy. Although designs prescribed local adaptation (which probably not followed in some cases) and hence UNICEF and IPs are advised to adapt design with respect to roof height, larger ventilation points and higher side walls.

The prescribed pit size for the demo latrineshighlighted being insufficient by the communities. There are reported cases where community tried to dug bigger pits,however the cover provided (by the project) found to be of smaller.

62

This alludes to the fact that in some cases the process of agreeing a participatory design with community was not followed.

Life cycle cost analysis of the latrine options also seems to not have been done. Depending on the material used, design life of the latrine superstructure is expected to vary considerably.

SPSP aims to improve peoples‟ access to safe drinking water. It merits further researching as to what impact increased sanitation coverage have had on water quality and accessibility in target.

Design of hand wash facility needs further clarity and needs to be communicated to the IPs particularly in areas where water is being accessed through hand pumps. It merits consideration whether to have separate hand wash facility or not, and if is not required, then suggest what actions to take.

School WASH facilities were found not being disability friendly both due to stairs leading up to the hand wash facility, as well as the design of the hand wash facility itself, which in most cases requires children to squat and reach out for the tap.

The evaluators did use an observation checklist (inline with standard design guidelines and standards followed worldwide). Significant deviations were noted and in some cases communities and teachers shared that for limited school space, the compliance to the set standards (for instance the minimum or safer distance prescriptions) was not possible anyways. Local Context and Issues The evaluators found quite a few non-functional demo-latrines, which amounts to lack of investment (cost sharing required) from communities and in some case discouraging attitude from the landlords (in Sindh particular). Some of the community feedback articulating the reasons for non-functioning of demo-latrines is outlined below (all by and large point to non-compliance to prescribed participatory design process):

Some shared that even if the sides of the latrines are covered with cloth/provide some form of superstructure, the females were reluctant to use for privacy reasons for low height of boundary walls;.

Some referred to land ownership issues where landlords (particularly in Sindh) did not allow the tenants to construct latrines on their lands by tenants/ workers. There was visible reluctance from the landlords to allow seasonal tenants to construct permanent latrines on their land.

The implication of the points highlighted above include women being dissuaded to use the latrine due to lack of privacy and a possibility ofconflict with landlords. The larger implication are that, a facility that has been built is not used, but still counts to be contributing in efforts to achieve ODF targets. The IPs may need to be advised to integrate local context and follow the prescribed process to have people‟s views before such interventions. This is likely to contribute to continued and effective use of the facilities or investments made. For latrine construction, the targets were met by an large (see table 13)

Latrines Constructed under SPSP Target Actual

63

No. of latrines (other than demo latrines) constructed by masons trained under PATS programme

13,857 20,040

No. of complete demonstration latrines constructed for extremely vulnerable families as per agreed criteria with community (maximum 3%)

4,157 5,212

No. of demonstration latrines constructed through materials provision/vouchers for vulnerable families as per agreed criteria with community (max 7%)

9,700 8,581

Table 12: Number of Latrines Constructed (Target versus Actual)

For supply facilitation (as noted in Figure 17), the results indicatean overall 39% of the respondents found using latrine facilities constructed with external support in the form of subsidy, loan, materials or trained skilled labour (please note that for this indicator the baseline value was assumed to be 0%).

The results indicate increase in people claiming to have constructed latrine in last year. The numbers have gone up from 26% to 47%, which is significant. The respondents shared that supplies facilitation as major support available for latrine construction (only for those who had constructed latrines last year). .

The key challenges include withdrawal ofgrant/loan for sanitation mart and entrepreneurs. This affected the supplies facilitation element negatively, however the positive results (around latrine construction and availability of materials) indicate that existing mart owners were able to meet up the increased demand. The increase for availability of materials and skilled labour both point to larger effectiveness of the interventions (see Figure 19).

39% 45%17% 18%

53% 55%

0%

50%

100%

Total Balochistan FATA KPK Punjab Sindh

Proportion of Population Using WASH Facilities Constructed With External Support

Total Balochistan FATA KPK Punjab Sindh

Figure 16: WASH Facilities Constructed with External Support

26%

4% 2%8%

47%

12%6%

31%

0%

10%

20%

30%

40%

50%

Latrine Constructed in the Last One Year in Respondents House

Demo Latrine Cost Share Supply Facilitation

Type of Support Received by the Respondent for latrine construction

Figure 17: Latrines Constructed By Respondent Over Last Year and Type of Support Received

64

The programme may need to evolve more effective and both user and enterprenuere friendly solution to this, one being available from DSI action research.

Overall the proportion of people having access to and using improved drinking water

source has improved from 76% to 82% (refer to Figure 20). This points to the

success of programme with respect to increasing access to drinking water (excludes

water quality).

School WASH Facilities Post-KAP results show a positive trend in all key aspects, including 97% access to latrine facilities and 79% access to water facilities (in some cases new water schemes were not installed for poor water). The results indicate a significant increase in availability of soap close to larine, which increased from 11% to 30%. This points to huge space/need and potential for improvement.

16%

53%

27%

60%

0%

10%

20%

30%

40%

50%

60%

70%

Availability of Sanitation Supplies in the Village Availability of Trained Skilled Mason/Labour in Village to Construct the Latrine

Figure 18: Availability of Sanitation Supplies and Trained Masons

74%53%

31%11%

97%79% 77%

30%

0%20%40%60%80%

100%

Schools with Latrine Facility

Schools with Drinking Water Points

Schools with Hand Washing Facilities

Schools with Soap Close to the Latrines

School WASH Facilities

Baseline Evaluation

76%60%

82% 82%

53%

96%82%

68%86% 85%

66%

99%

0%20%40%60%80%

100%120%

Total FATA KPK Punjab Balochistan Sindh

Proportion of the population using an improved drinking water source

Population Using an Improved Drinking Water Source

Baseline Evaluation

Figure 19: Population Using an Improved Drinking Water Source

65

The delivery mechanisms must have given consideration to the customary summer holidays, which apparently was not given much thought into while laying the plans. In line with the „Islamabad Commitment (2010)‟, signed by all provinces for Child Friendly Inclusive Schools, the programme did effort and largely remained successful to improve school WASH facilities to make them child friendly9. The programme prioritised girls‟ school for WASH facilities rehabilitation. It was found more relevant for the fact that culturally availability of safe and secure facilities at schools is relatively more significant for girls. The results indicate better performance in girls schools compared to boys such as availability of soap, though still insufficient. This merits greater attention from teachers and availability/use of SMC funds to ensure continued availability of soap, as well as proper functioning and cleaning of the facilities. Positive developments in this regards include Punjab government‟s approval to use SMC funds for repair and maintenance, and the change in resourcing criteria for SMCs in Sindh from a fixed yearly amount to a per child allocation basis. Despite policy changes, the actual impact in terms of funds getting spent is quite limited. Often these remain unused or underspent for procedural reasons (possible audit objections, low engagement and capacities of SMCs, limited interest of school head teacher and others), hence the future programme design may need to prioritize addressing those procedural bottlenecks

9According to „Development of a framework and an action plan for school sanitation and hygiene education‟ (SSHE) in Punjab, Analytical Report On Current

Status of School WES in Punjab School; Final Report December 2007. UNICEF Punjab, child friendly facilities have the following attributes,

easy and comfortable seating

proper ventilation and light arrangement

appropriate space inside the toilet

special non-slip ceramic tiles in pour flush latrines

provision for looking glass (mirror)

suitably designed hand washing points

appropriate height of door knobs/locks

appropriate height of hand washing basin

provision of shelf and hook to safely keep belongings while using toilet

provision for children with special needs

Figure 20:School WASH Facilities

85%97%

77%

23%

74%

95%

74%

26%

65%

100%

80%

45%

0%

20%

40%

60%

80%

100%

Availability of Drinking Water

Availability of Latrine Facility

Handwashing Facilities Availability of Soap

Boys Co-Education Girls

66

Output 3: Health And Hygiene Promotion (H&HP) The results indicate largely positive performance for health and hygiene promotion component. However, for some indicators the results are not very encouraging. The following table illustrates the mixed performance for some of the health and hygiene activities.

Description Target Actual

No. of people reached with WASH BCC messages through electronic/print media. 970,000 609,530

No. of people reached with WASH BCC messages through community events (theatres, puppet shows) 183,300 165,092

No. of school children (boys and girls) reached with messages through SLTS activities. 63,941 154,526

No. of people (men, women, boys, girls) reached with WASH BCC messages through CRPs 291,000 846,444

Table 13: Outreach of Health & Hygiene Messages through Various Mediums

Overall the results are positive with respect to increasing knowledge and bringing positive attitudinal and behavioural changes. For instance, water treatment and storage practices show improved results where those treating water (through multiple tools) increased from 4% to 13% (a three-fold increase).Similarly, those storing water has increased from74% to 91%. Balochistan province leads in terms of extent of change followed by Punjab (see Figure 24).

4%13%

2%

41%

18%0%

26%

74%

13%23%

10%

48%

1% 5%13%

91%

0%

20%

40%

60%

80%

100%

Water Treatment

Boil Add Bleach & Chlorine

Strain it through a

Cloth

Use a Water Filter

Solar Disinfection

Let it Stand & Settle

Water Storage Practice

Water Treatment Methods '

Figure 22: Household Water Treatment and Safe Water Storage

Figure 21: Availability of WASH Facilities in Boys, Girls and Co-Education Schools

67

Another measure of success and effectiveness of H&HP is increase in the number of households reporting availability of soap in the house. Similarly, there is an increase in designated hand washing places (available). The increasesindirectly points to heightened awareness around hand washing and linkages to diseases (refer Figure 25 for details). The availability inside or near latrines have not increased much though. This can be attributed to the fact that people in rural areas tend to not leave the soap near or inside the latrine for risk of dissolution (waste).

The encouraging increase in handwashing practice at critical moments, points to effectiveness of H&HP. The results are particularly more encouraging for post defecation (refer to Figure 26) compared to other elements such after urination, cleaning child etc.

19%

47%27%32%

60%

25%

0%20%40%60%80%

Separate Handwashing Place Anywhere in the dwelling Near/Inside Latrine

Soap

Hand Washing Place & Soap Availability

Baseline Evaluation

Figure 24: Hand Washing Place & Soap Availability

50%

29%20%

15% 15%10%

62%68%

22%14% 12%

7%

0%10%20%30%40%50%60%70%80%

Before Eating After Defecation

After Urination Before Cooking After Cleaning a Child that has

Defecated

Before Breastfeeding a

Child

Respondents Practice of Hand Washing at Critical Moments

Hand Washing at Critical Moments

Baseline Evaluation

5% 5% 9%3% 0%

8%13% 18%

34%

10%2%

11%

0%10%20%30%40%

Total Punjab Balochistan Sindh FATA KPK

% of respondents who treat drinking water

Baseline Evaluation

Figure 23: Proportion of Respondents who Treat Water

68

The effectiveness is evident from encouraging increase on awarenessaround water related diseases. The heightened awareness however has not contributed to any significant reduction of diarrhoea incidence though. This could have the seasonal element also, as numbers increase in summers anyways (refer Figure 27 for details).

Figure 25: Critical Moments for Hand Washing

52%28% 27% 22% 27%

79%44% 44% 33% 24%

0%20%40%60%80%

100%

Diarrhea Cholera Hepatitis Skin Disease Incidence of Diarrhoea in Last

Two Weeks

Knowledge of Diseases Caused by Using Unsafe Water '

Knowledge of Water Related Disease andIncidence of Diarrhoea

Figure 26: Knowledge of Water Related Diseases and Incidence of Diarrhea

69

Output 4: Linkages With Duty Bearers The level of interest and engagement of public sector entities varied significantly across provinces as well as IPs. However generally there was not much success with respect to fostering ownership of PATS and mobilizing resources for planned rewards/incentives. The district governments however helped with selection of villages, identification of schools and health facilities which needed rehabilitation. Overall the interventions still contributed to raising knowledge and awareness of public officials both at the provincial and district levels though the extent of it merits further investigation. Bottlenecks identified through the provincial workshops as part of the baseline did not translate into activities to address them, this may need to be prioritized in future programming. This component aimed at developing linkages with duty bearers and public institutions to support PATS to positivelyinfluence budgetary increases (at provincial level)for WASH services, encourage PATS adoption in the public sector and finally leverage public funds as rewards for villages turned ODF. Multiple activities carried out for this and largely the results are encouraging (refer Table 15). Description Target Actual Percentage

Capacity Needs Assessments (CNA) 14 6 43%

Capacity Development Plans 14 6 43%

Capacity Development Events 14 15 107%

Number of Government officials and political leaders that participated in capacity development and key events of the programme.

185 230 124%

Advocacy initiatives undertaken with duty bearers/political leadership to support PATS

14 17 121%

Villages jointly selected for PATS programme by relevant Government duty bearers with IP

1,385 1,791 129%

ODF communities received rewards/incentives by government 39 62 159% Table 14: Capacity Building &Advocacy for Government

The level of interest and engagement of public sector entities i.e. Public Health Engineering Department (PHED), Local Government (LG), Health and Education departments,varied significantly across provinces. For limited guidance and standardization, the focal points in the public sector were found to be different for different IPs (for some it was DC office, others worked with either PHED or LGRRD), however generally it did not achieve much success with respect to fostering ownership of PATS and mobilizing resources for planned rewards. This could be attributed to lack of policy guidance from provinces resulting in limited interest from senior local officials. The IPs were found to be connectedto bottom tiers of government officials. In some cases where the focal points were more sensitive and aware of potential NGO contributions, showed greater interest, ownership and leveraged the partnership for common objectives. The district governments helped with selection of villages and education and health authorities facilitated identification of schools and health facilities needing rehabilitation. The interventions did contribute to raising knowledge and awareness of public officials (particularly around PATS), both at the provincial and district levels. To what

70

extent to which capacities been raised and contributing to improved public services is unknown and merits investigation. The achievement of leveraging public resources both in terms of increased budgetary allocations (at provincial level) and mobilizing funds for rewards at districts are insignificant. This again reflects of lack of planning and guidance for the intervention area (more details in Figure 29).

The frequent transfers of the public officials also did affect the efforts, as those who were sensitized and trained got posted elsewhere, leaving IPs to work with new officials with no/limited exposure to programme and areas where it needed their support. This element together with short time frame (requiring focus on delivering outputs) apparently work to discourage IPs to work more closely with local officials. Bottleneck analysis workshops conducted at provinces as part of the evaluation, which were attended by LGRDD, PHED andEducation Departments. It was noted that not much progress been made with respect to actions identified in the bottlenecks carried out during baseline. This may need to prioritized for the coming years.

Output 5: Attaining Total Sanitation The programme appear effective with respect achieving the targets for transforming the targeted communities or villages into ODF. This marks progression on the sanitation ladder. There are however concerns as to communities being able to sustain the ODF status. The table below presents the reported results with respect to the total sanitation component. The programme has achieved (overachieved) on all sub-indicators. Villages supported in attaining Total Sanitation status Target Actual

No. of households with new latrines constructed after start of the PATS programme 17,182 33,673

No. villages declared as ODF by community 831 1,436

No. of villages Verified as ODF by Implementing partner and independent monitors 831 1,250

No. of people (men, women, boys, girls) in communities declared as ODF 581,700 711,467

No. of people (men, women, boys, girls) in communities verified as ODF 581,700 679,967

No. of model villages with laid sewers, WW collection and treatment system through PATS programme 4 18

0.28%0.39% 0.37%

0.12%

0.00%

0.20%

0.40%

0.60%

KPK Punjab Balochistan Sindh

Government expenditure on WASH as a proportion of total government expenditure

Figure 28: Government Expenditure on WASH

71

A CRP told the evaluation team that in Uthman Khel village a household latrine was being drained directly on to the street. Due to this the area was very dirty and no one was able to walk past that part of the street. After the PRA exercise the people contributed through VSC to construct drainage / sewer along the streets. Therefore, the streets are now clean.

No. of advocacy initiatives undertaken to promote communal underground drainage system linked WW treatment system and communal SWM system 2 22

No. of ODF villages provided with underground drainage and WW treatment facility by relevant public sector stake holder/duty bearer 2 0

Table 15: Villages Supported in attaining Total Sanitation Status

This performance however comes with caveats around sustainability. In general there is an understanding the focus was on delivery rather sustainability and for shortage of time the IPs were left with no time to monitor and to sustain ODF status in the certified villages. The communities have started showing signs of slippage. The programme has also been able to demonstrate healthy increase overall in communities/households connected to under-ground drainage (details in Figure 30). The evaluators did however notice significant variations in terms of what constitutes drainage system.

The evaluators noted heightened awareness evident from the identification from the communities that there latrines and the single pits designed, do not seem to have proper and adequate drainage. Also, that in short time they would be the cause of pollution and possible spread of diseases.

27%

52%

0%

15%

35% 31%44%

65%

33%43%

58%

24%

0%10%20%30%40%50%60%70%

Total FATA KPK Punjab Balochistan Sindh

Households Using Underground Drainage

Households Using Underground Drainage

Figure 27: Households Using Underground Drainage

72

Output 6: Knowledge Management And Accountability The knowledge management component being an integral part to the programme design and delivery has contributed to critical learning around use of subsidy, sanitation marketing and low-cost sanitation options. The third party monitoring mechanism instituted for the programme proved to be particularly useful in real time monitoring and complementeddecision making. The new learning resulting from this phase of the programme have already been fed intothe design and modalities of the next phase of SPSP, and offers the opportunity and potential of informing a PATS review. Overall, the component contributed to improving the programme implementation and interventions by informing management and programmatic decisions through real time monitoring of programme processes, challenges and issues whilst helping in course correction. The component also focused on programme learning and documentation for future design, possible replication and scaling up through several lesson learning workshops, baseline and evaluation studies and action research.Theachievements for the components are presented in the following table. Knowledge Management and Accountability Actual Target

No of Inception workshops conducted in the beginning of programme 12 12

No. of villages with baseline established in the beginning of programme by IP 1,385 7,448

No. of joint progress monitoring reports (IP and public institution) 80 39

No. of monthly substantive monitoring reviews‟ visits conducted for confirming/measuring performance indicators in results framework 80 216

No. „monthly substantive monitoring review‟ reports shared among partners 63 55

% of responsive progress reports submitted as per MEAL framework by IP - 33

No. of quarterly progress reviews conducted with IP and minutes documented 2 15

No. of mid-term review workshop report endorsed by UNICEF 1 4

No. of Human interest stories documented by IP and accepted as per UNICEF guidelines 80 78

No. of end line survey reports accepted by UNICEF 11 2

No. of Lessons sharing workshop conducted and feedback documented 1 1

Table 16: Knowledge Management and Accountability Events and Activities

Third Party Monitors (contracted for process improvement, verification and feeding into management decision making) work was a major factor in redesigning programme activities based on ground realities that emerged over the course of programme implementation. This monitoring process allowed UNICEF to monitor and oversee ground implementation in real time and push for more effective and efficient implementation through IPs (particularly those that were lagging). It also helped inform the management of the need for corrective measures where weak performance of IPs was spotted. Third Party Provincial Monitoring meetings (with IPs and UNICEF)also allowed for active and effective debate that helped keep the project on track.Monitoring was of four types, planned, verification purposes, spot checking and follow-up. Regular monthly reviews helped in making the programme more effective whilst ensuring efficiencies were achieved, instead of monitoring activities periodically (3-4 months of work) and not allowing sufficient time for course correction given the first phase was only for a duration of one year.

73

Whilst most of the Third Party Monitors work proved effective, there was some feedback that at times monitoring took the form of policing, moreover there was confusion in respective roles of several firms contracted for the range of monitoring. These aspects need to be clarified and addressed in the next phase and in future programmes. Given the delayed start of the project, tight timeline for activities and an ambitious monitoring schedule, more focus and emphasis remained on monitoring and less on knowledge management and documenting of key learning during the programme implementation period. However the component did contributeto knowledge management by commissioning useful baseline and end of phase evaluation. The baseline findings and learning have been fed into the implementation of Phase 1 of SPSP, whilst the evaluation findings and lessons learnt have been used to inform the design and interventions of Phase 2 (during the presentation and workshop with the Evaluation Resource Group, UNICEF Staff and other Stakeholders). The component also made available data and evidence on the use of PATS in Pakistan, with a potential of using this information in a structured way to inform PATS and sector stakeholders of the experiences, challenges, successes of SPSP implementation. Information is now available however it needs to be consolidated, disseminated as well as used for a possible review of PATS based on the experiences, and lessons learnt through the implementation of SPSP Phase 1, particularly around low-cost sanitation options, use of subsidy for household latrines, and sanitation marketing.Moreover the action research commissioned to DSI should also be proactively shared with stakeholders to help inform their interventions.

74

5.2.4. Stakeholder Contribution10 As highlighted in the section above related to programme components and results, the inputs of the range of stakeholders was responsible for delivering the outputs with varying degrees of success and effectiveness. UNICEF (Islamabad and Provincial Offices) provided effective programmatic and management oversight to ensure that the programme was delivered according to the specific PCAs with IPs within the available resources. The number of actual beneficiaries as well as villages exceed the planned targets. UNICEF also put in place a robust monitoring function for the overall programme which helped inform better decisions and course correction during implementation. Throughout the programme UNICEF offered a range of capacity building interventions which helped develop the community‟s capacity, build a cadre of PATS master trainers, increase capacity of non-profit entities as well as government departments (though the focus on government capacity building could have been greater) to implement sanitation responses. Though it‟s difficult to attribute improvements and positive policy measures taken at the federal and provincial level to one stakeholder i.e. UNICEF, however there is evidence in the form of Provincial Government‟s Master Plans that incorporate PATS model, as well as in some cases where government allocations to WASH sector have increased. However engagement with government needs more concerted effort from UNICEF Islamabad as well as Provincial Offices particularly in resource mobilisation for PATS.Other UNICEF contributions included a focused emphasis on Knowledge Creation including project and sector specific training manuals, IEC material, training documentary, producing thematic issue papers, focusing on issues such as Menstrual Hygiene Matters (MHM) research and WASH Equity study, Lastly, UNICEF‟s contribution has been of raising sufficient funds for the two programme contexts i.e. follow up in flood affected areas as well as funding for WASH sector in the insecurity affected areas. UNICEF‟s contribution had an impact across all the outputs of the programme. DSI‟s specific contributions relate to sanitation marketing, design manual for latrine facilities and focusing on action research in SPSP pilot area. Sanitation marketing is an evolving process in WASH sector in Pakistan, and DSI demonstrated their flexibility in capacity building interventions during SPSP where circumstances necessitated that they provide recurring capacity building support. Moreover offering a business advice phone service to the Mart Owners and Entrepreneurs specifically contributed to building the confidence as well as helping through the initial teething issues. DSI‟s contribution resulted in building and sustaining demand through supply side interventions. RSPN‟s capacity building activities (in particular social mobilisation training, use of PRA and mentoring activities) played a critical part in the delivery of results as it helped a range of non-profit organisations to deliver the SPSP programme across geographic regions using a uniform implementation methodology to carry out SLTS and CLTS. Some points have been highlighted above regarding changes that can be

10TORs required the evaluation to focus on a sample of partners, there was no requirement to conduct a comparative analysis of IPs.

75

incorporated in the training, mentoring approach as well as social mobilisation to ensure it is more effective for future phases, however in the context of Phase 1, the activities were carried out effectively both in terms of number of activities and beneficiaries as well as in terms of delivery of results. RSPN‟s contribution resulted in generating demand for sanitation. APEX contribution to the programme remained its regular monitoring reports and feedback which informed real time decision making to keep the programme on course both in terms of its delivery of outputs and achievement of results. Recommendations have been made to address issues that were highlighted in reference to monitoring i.e. rigidity of approach, which if addressed will further improve the role of third party monitoring in the next phases of the programme. APEX‟s direct contribution is to the knowledge and accountability output however due to their regular monitoring the results of their contribution have an across the programme impact for all output areas. Government Departments also played a contributory part although with substantial variation across geographic regions, i.e. in some cases close coordination and engagement with the project, but through lower tiers of the departments (e.g. Dera Khan District), whilst in other areas limiting departmental engagement to only certification of ODF villages through senior district officials e.g. ADC in Jacobabad. However this allowed government departments to become familiarised with the PATS approach as well as to the issues on the ground in the villages. In certain parts where government was more responsive water supply schemes have been rehabilitated, whilst in other areas provincial government allocation in its budget to WASH sector has increased due to the engagement with SPSP. Government‟s direct contribution has been to the linkages output, with community-government, non-profit –government department links being formed to deliver PATS. IPs (range of non-profit organisations) contribution has been across the results framework in all output areas. The extent, degree as well as success varies across the IPs, however all have been able to create and sustain demand for sanitation, while also ensure supply facilitation, health and hygiene promotion and total sanitation results were also attained. Specific areas have been highlighted in the findings section as well as more specific detail is provided in the lessons and recommendations section to help improve IPs delivery and impact for the next programme phase. In summary the required number of activities and outputs were delivered, benefitting a larger number of villages and beneficiaries than targeted, with many result indicators showing positive trends i.e. increased access to sanitation facilities (for men, women, children as well as an improvement in the equity context), similarly ODF declaration, verification and certification has also been achieved though slippage will need to be monitored and controlled. Community stakeholders (CRPs, VSCs, Children – School WASH Clubs, Teachers as well as masons, entrepreneurs and sanitation mart owners) have played the most vital part in helping achieve the results. The various community stakeholders stated have actively participated in the programme activities, given their valuable time to the interventions, facilitated IPs, APEX and UNICEF. Moreover their willingness and acceptance of wanting to improve their and their community‟s lives helped make this

76

programme an effective one that also achieved its targets. Their contribution can be mapped across the range of the outputs.

5.2.5. Gender Equality, Equity & Disability In line with UNICEF policies, the programme design and interventions have incorporated principles relating to gender equality, equity and disability. In terms of gender the programme was effectively able to demonstrate focus across men, women, girls and boys (including having a staff mix (of men and women) in the field, creation of women/men VSCs and CRPs, focus on boys/girls schools, appropriate BCC strategies etc.)expressed focus on equity is evident with the use of PRA techniques to identify the poor as well as inclusion of subsidy to ensure access to sanitation facilities. However the integration of disability remained weak as evidenced in the construction and access aspects of WASH facilities. The selection of schools (boys and girls), Social Organizers, Community Resource Persons (CRPs) and Village Sanitation Committees (VSCs) ensured participation of men and women in the programme.To address differential information needs, access to information sources and physical accessibility -- mainly for cultural seclusion between sexes -- the programme evolved responsive BCC strategies in terms of content, design and medium of transmission, e.g. female CRP sessions, theatre performances for children. The programme was also in line with the needs identified by women, including children‟s health due to the unhygienic conditions as well as hazards of open defecation, difficulties faced by women particularly due to privacy needs which forces them to defecate either very early in the morning or to hold until it is evening causing health complications. It was also mentioned that going at night exposes them to a range of risks including being bitten by insects or attacked by animals, to even being harassed and physically attacked by men. The programme design therefore clearly identified gender considerations such as appropriate design of latrines for women and girls to ensure privacy and safety, appropriately designed water collection points as well as role in collecting water. Gender disaggregated data was effectively collected and used in the program. The programme design also effectively addressed equity considerations by including provision of low-cost environment friendly and culturally appropriate demonstration latrines for the poorest of the poor communities. The programme had a 3% (for the extremely vulnerable) and 7% (for the vulnerable) approach to address the sanitation needs, whereby the extremely vulnerable received subsidy for the whole latrine, whilst the vulnerable had to construct the super structure using their own resources. SPSP design also considered addressing the needs of the disabled; however, this approach lackedclarity in terms of expectation and implementation.

44%34%

27% 32%

13%7% 9%

66% 64%

40% 37%

12%5% 6%

0%10%20%30%40%50%60%70%

Disabled Having Access

to Latrine

Women Having Access

to Latrine

Lowest 2nd 3rd 4th Highest 5th

Income Quintile

Gender, Disability & Equity Based Access to Latrines

Baseline Evaluation

77

From a gender perspective the results of the post-KAP survey demonstrate that women‟s access to latrines has increased by 30%. Moreover in numerous interviews and meetings with a range of stakeholders (both men and women) the evaluators were told about instances of violence as well as harassment towards women due to their vulnerable situation while going out at night for defecation. In the gravest of situations false allegations of sexual indiscretion have even led to instances of karo kari. In terms of equity, the lowest and the 2nd Quintile seems to have benefitted most in terms of access to latrines as intended by the programme. However as highlighted above household subsidy as a method to respond to equity needs will need to be revised or replaced with innovative methods. Lastly in terms of the disabled, access to household latrine has increased to 66% during the evaluation as compared to 44% during the baseline, however whether these latrines cater to the special needs of the disabled is not confirmed, given this component remained weak during the design as well as implementation stages. Moreover, there is substantial evidence that suggests that the programme‟s successes, as secondary impact, have also contributed to reinforcing a gender divide. For instance, the programme interventions have added to the workload of women and children being responsible for cleaning latrines and fetching additional water required due to the introduction of flush latrines. There has been some feedback on the DRR designs of the latrines as well as water facilities which have been designed to be elevated, and which have reduced the privacy factor for some. This is of particular issue where such facilities have been constructed without taking community into confidence about the design aspects of the facilities, otherwise an alternate site for the facility could have been facilitated by the community. Also there is however a strong perception by a range of stakeholders (CRPs, VSCs, Government Officials, that having access to latrines has reduced instances and opportunities for harassment and violence against women that was always a possibility when women were defecating in the open particularly given the privacy needs at night time.

Figure 28: Gender, Disability and Equity Based Access to Latrines

78

5.3. Efficiency11

Evaluation Q3: To what extent have the outputs of the project been achieved (in terms of quality and quantity) with the allocated resources/inputs (such as funds, time, and procedures)? The evaluators have looked at the element of efficiency in terms of progress or performance against key results, adequacy of resources (time, human and financial), coherence to the set/available quality standards. The evaluators did do value for money analysis with respect cost per beneficiary (for varied results) and basic analysis around costs for partnership management. The programme has achieved efficiency in terms of performance against target indicators, Cost per beneficiary and villages achieving ODF declaration stage demonstrates efficiencies against the planned costs. However some indicators underperformed, while concern around planning, target setting and best use of resources is raised on indicators that substantially overachieved their targets (this leads to the understanding that either there was a cost and resource implication or a quality impact). Project results indicate adequacy of human resources, however the results exceeded by a number of times pointing to gross miscalculation as to planned targets and resources required (financial and human). Instances such as contractors charging the same amount of funds for building new and rehabilitation old school wash facilities suggests areas of financial inefficiencies. Highly variable cost of latrines raises the question of affordability for the vulnerable across SPSP programme areas. Performance Against Target Indicators The efficiency analysis with respect to performance against key indicators suggeststhat the IPs have been able to achieve several of major targets. The Programme Output Matrix (updated version available as of March 2014) shows that out of 60 output indicators, the programme exceeded targets for 41 indicators. Furthermore, within these there are 10 output indicators, where results exceeded by over 300%. This however raises concerns around targets being reasonable, over-budgeting and given these were achieved in shorter duration, the quality and sustainability of the results. However several target indicators did not do so well, for example outreach to beneficiaries through media, theatres for health and health and hygiene messages, provision of rewards and incentives (with the target to provide 158 critical community infrastructure schemes as incentives/rewards, only 2 were delivered). Furthermore two programme components indicate little progress i.e. provision of loan and provision of underground drainage. This points to limited success with fostering partnerships (with relevant public authorities) and mobilizing public resources.

11Note:The evaluators’ ability to conduct a detailed analysis was constrained by the fact that project expenditure sheets were unavailable (not compiled by the time evaluation was in progress) The calculations and analysis done below relies on budget estimates rather actual expenses, drawn either from PCAs or UNICEF submissions to DFID (UK).

79

Financial, Human & Time Resources The programme has largely performed well where it has achieved the set targets points to the adequacy of resources to carry out tasks. However, the fact that the programme has exceeded most of the targets by 20% to 100%, raises concerns around realistic costing by the IPs. (refer Annex 13: Programme Target versus Achievements). The evaluators noted some costing oversights where they found that contractors (for rehabilitation of WASH services) charging the same amounts for building new facilities as for the rehabilitation of old ones at school and health outlets. Some IPs communicated that the umbrella arrangements contributed to raising operational expenses (where both umbrella and implementing partners charged operational expenses), which points to the inefficient use of resources. The staffing arrangements were found to be adequate by and large. The field staff especially SOs did raise concerns of the availability of time for quality work as mobilization and exit planning. To them, this contributed to pushing them work extra hours to meet the targets. Some IPs referred to this as key factor for high staff turnover. SPSP Phase 1 was designed for a period of one year. The field level implementation in some cases started with a delay of up to 4 months (PCAs with IPs were signed between February to June 2013)which allowed some IPs implementation period of only 6-8 months against the planned one year. Despite the fact that IPs were able to achieve most of the set targets, there are concerns with respect to work planning&time availability (to IPs for implementation). Reportedly, there were oversights with respect to timing activities with community life-cycle (despite the fact that the information was available from village PRA exercises) and with those of public services outlets such as schools and health centres. This to a degree pushed IPs staff to use short-cuts to complete activities e.g. BCC, and left them with little time to plan exit and leave behind arrangements to sustain the achievements and take it further. WASH Facilities Cost Comparison Across IPs and Geographic Spread:

As per UNICEF and IPs PCAs the cost of demo latrines varies significantly between Rs. 9,500 to Rs. 20,000 and similarly for hand pumps from Rs. 18,000 to Rs. 69,590 (Inaccessibility to actual expenditure details limits detailed analysis).Whilst a price differential is expected across various regions, it raises question on the definition of low-cost in terms of latrines. For instance the costs range between Rs. 9,500 and Rs. 20,000 across kashmore and Jaffarabad respectively, though the monthly household income of the lowest quintile across the two districts varies only by Rs. 292, thus questioning the appropriateness as well as cost efficiency for demonstration latrines for the vulnerable in Jaffarabad and likewise for other districts.

80

Cost Analysis This section offers cost/beneficiary analysis with respect to different output/outcome level results. Also, it offers comparative analysis with regional countries. Cost Per Beneficiary The budgeted figures indicate that for SPSP Phase 1, a budget of $13.6 million was set aside to reach 1.3 million people (men, women, boys, and girls) in 13 target districts/agencies. Reportedly (refer Indicator No. 2 in Annex 13), the project reached out to 1.2 million peoplewith sanitation messages. Hence, the cost per beneficiary comes out to be US$11.4 (against a planned cost of US$10.5) for demand creation through health and hygiene promotion. Total Budget US$ 13,690,427

Target/ Planned Beneficiaries 581,700

Actual Beneficiaries Reached 711,467

Per Capita Costs for ODF Declaration

Planned Beneficiaries Actual Beneficiaries

$23 $19 Table 17: Cost Per ODF Declared Beneficiary

The cost per beneficiary for achieving ODF declaration for actual number of project beneficiaries is US$19, whereas the planned cost for the targeted beneficiaries was US$ 23, thus demonstrating efficiency gains of $4 within the given project budget. Cost Per Community/Village Another key result was for communities/villages to achieve ODF status. These include those who declared (self declared) ODF and those that were verified.

Total Programme Budget US$ 13,690,427

No. of Planned Villages* 1,385

No. of Actual Villages Reached* 1,878

No. of Target ODF Villages 831 No. of Declared ODF Villages 1436 No. of Verified ODF Villages 1250

Cost per village to total target no. of ODF villages

Cost per village to declare it as ODF village

Cost per village to Verify it as ODF Village

US$ 16,474 US$ 9,533 US$ 10,952 Table 18: Cost per ODF Declared and Verified Village

In this context the programme has overachieved both targets i.e. number of people living in ODF settlements, as well as the number of ODF declared villages. This has resulted in a substantial efficiency of 42% in the cost of having a village declared as ODF. In comparison to SPSP, the Total Sanitation and Sanitation Marketing (TSSM) programme in India which was implemented in two states Himachel Pradeshand

81

Madya Pradesh12.with an actual budget of $1.9 million had a cost per person living in an ODF declared village of US$9 only. The programme documents describe the costs for building a new latrine into three categories equalling $38 for category A toilet, $70 for category B toilet and $99 for category C toilet. Using the average cost of these 3 categories, the average cost of constructing a new latrine equals $69. A study conducted by WaterAID for CLTS programmes across Bangladesh, Nepal and Nigeria suggests that the latrine (lined pit, concrete slab and concrete/ceramic pan, brick or concrete superstructure) costs $62-100 in the plain areas and US$150-187 in the hills. As such the three latrine designs are well within comparable costs. In sum (based on information made available) SPSP needs to address the areas mentioned above to improve its programme and cost efficiency.Overall the work planning, target setting and duration of the programme seems to have been planned and executed with weaknesses, while cost for people and village to achieve ODF declaration stage comes across to have created efficiencies against the planned costs.

12Water, Sanitation and Hygiene Portfolio Review; Department for International Development

(DFID); March 2012, available at https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/67433/DFID_20WASH_20Portfolio_20Review.pdf

82

5.4. Sustainability Evaluation Q 4: To what extent various stakeholders including the public sector and civil society are likely to sustain the behaviour change related to the goals of the programme after it has ended Programme design excludes any articulated and expressed exit plan or sustainability strategy. Knowledge, attitudinal and behavioural changes that this programme has been able to bring amongst men, women and children, are the most significant and likely results that may sustain. Whilst the community structures and mechanisms, such as CRPs and VSCs are likely to sustain but with limited success. However IPs found no/little time to prevent the process of slippage following ODF declaration and certification. Linkages with duty bearers component could achieve little in terms of fostering ownership amongst public agencies. The sustainability of sanitation mart owners as well as entrepreneurs created/promoted during the programme is also uncertain due to lack of access to loans/finance as well as low demand issues. Lack of having instituted an O&M mechanism for rehabilitated WASH facilities at schools and health outlets suggests a high probability for them to become defunct in time. According to the PATS approach achievement of an ODF environment (declared or certified) is not sufficient, it needs to be sustained. Moreover there needs to be an enabling environment at local, provincial as well as national level to ensure long term sustainability. The evaluators have therefore assessed sustainability across a range of aspects i.e. sustainability as part of the design, level of commitment amongst stakeholders to sustain the ODF status in project areas,contribution of SLTS in sustaining ODF status in communities, partnerships established during programme implementation (with government, donors, civil society organizations, and other stakeholders) and lastly government commitment for PATS scale-up. The design excludes any articulated and expressed exit plan or sustainability strategy. Some references are made to sustainability in PCAs, though they appear more of a requirement, as none of the IPs, the evaluators interacted with,shared evolution of exit plan and its implementation. The exit strategy is not defined nor possible stakeholders identified for sustainability, possible scale-up or even moving up the sanitation ladder. The implication of lack of an expressed sustainability plan or strategy is that there is no standard approach that can be followed and put in place for and by the various IPs, moreover interventions and project aspects that could otherwise be brought to sustain may be affected by the omission of a focused and planned effort which would have been followed given a strategy/plan had been articulated, put in contract and monitored over the course of the programme. A further implication is leaving the community with unanswered questions e.g. who will continue the work? How will those people get access to a facility who are poor but did not receive it in the current phase? The programme has however contributed to development of training and IEC materials, latrine guides etc., which are available with partners. The applied

83

In response to a question on sustainability and continuity of

WASH Club work, one of the students from Dera Ghazi Khan said

‘ I am graduating to the 6th Standard and will be going to a new school but will continue to share my

learning with the children over there and also volunteer to be part

of their Wash Club’

In another instance children imposed on their mother to wash her hands before she kneaded the

wheat dough for making roti.

knowledge acquired through practical application of PATS approaches; principles and standards will likely sustain and be used to further build on. In terms of stakeholders,several IPs have expressed interest to continue working in sanitation sector, though this is at their own individual level and not attributed to efforts for sustainability through the programme. Moreover continuity and scale of interventions will be dependent on resources available to these IPs. Most of the Programme Staff and SOs have also moved on to other programmes, however they will remain available in the market for other WASH programmes, drawing on their SPSP experience. CRPs are referred to as a cadre of human resource for sensitizing communities on the adoption of improved sanitation and safe hygiene practices by PATS. This cadre has been successfully created by the programme, and while all may not continue, feedback from most is that they will continue to raise awareness as it is a social good and in the benefit of their communities. With time this cadre of local resource persons may either themselves or through functional VSCs work on larger community development agenda, including water and sanitation. VSCs have developed CAPs and there is some sense of communal benefit for VSCs to continue beyond end of programme, their ability to sustain as VSCs/CBOs has not been strengthened and most of these VSCs are likely to either disband or become dormant. However the knowledge and experience gained by individual members will still be there, and future interventions may be able to revive or strengthen them for developmental purposes. Sustainability of entrepreneurs and sanitary marts is not very likely, this is due to several reasons, first the way they were identified and mobilised (initial concept was to provide loans/grants which was dropped by UNICEF), secondly their innate nature to conduct business activities which don‟t carry a possibility of risk, and lastly access to required finance. In cases where post, end of programme the demand continues to be high in the initial months the sanitary mart or entrepreneur may find sufficient incentive to continue provision of services. SLTS has played a complementary role to CLTS, however it is too early to say whether it will help in sustaining ODF status. However the change that has been evidenced in children‟s behaviour, increased knowledge and practices will continue as well as be passed on from student to student, to household members and to community in general.Also as observed in the field they will convince their household members as well as community of the significance of health and hygiene as well as maintaining an ODF status. Longer lasting contribution will come through sustainability of Children‟s WASH Clubs which depends on continued

84

and proactive involvement of teachers (as WASH Club Supervisors).Therefore it is important that teachers remain effective and receive periodic refreshers. In the longer term the importance of health and hygiene needs to be incorporated into the curriculum, this can be raised by UNICEF and IPs as a policy need in the next phase. Sustainability of physical infrastructure (school latrines and wash facilities, household latrines and community water schemes) is going to have mixed results. School WASH facilities as well as community water schemes depend on continuous operation and maintenance (O&M) provisions without which they are likely to become defunct in time. At school level O&M provision should have been institutionalised or linked with use of SMC funds through a mechanism, whilst a community based O&M system was necessary for sustainability of communal water supply schemes. Household latrine facilities are likely to sustain until their expected life span (depending on how soon the pit fills up and whether the household is able to afford another pit or cleaning of the first one). The behavioural changes amongst men, women and children, as also noted in health and hygiene section from the post KAP results, are likely to not only sustain but have a multiplier effect on the community that they interact with (including hand washing practices, construction of latrines for family (particularly women). ODF status is likely to sustain but not at the current level, evaluators field observation as well as feedback from several FGDs suggests slippage (already visible in some villages), this is due to the fact that programme delays in the beginning left little margin for supporting the sustainability of ODF status. Most of the partnerships developed during the implementation phase are not going to sustain beyond project end, without a continued funding relationship. The implication of this will be that several of the IPs may end up not continuing their work on PATS and WASH sector unless they are able to secure funds from elsewhere, reducing the efforts to a projectised approach which lasted only one year, where as the situation that SPSP is trying to address i.e. access to sanitation and achievement of ODF status is a much longer term effort as also reflected in the Results Framework, therefore it should be UNICEF‟s effort to secure a minimum funding commitment for the entire period of the programme, moreover partnerships should continue with the IPs subject to continued funding. A further implication of operating this way is duplication of effort and costs of training IP staff each time a phase ends since the trained staff from previous phase is laid off by the IPs at the end of each phase of the programme given the uncertainty of partnership and funding. However key partnerships that were developed during SPSP Phase 1 include, DSI‟s action research work that contributed to PATS learning for UNICEF, RSPN‟s role as master trainer which has been effective and may likely continue in the next phase, though by bringing on board some improvements. In terms of Government commitment to scale-up only Punjab as a province has shown commitment and has taken on the responsibility of sanitation sector with the intent of achieving ODF status.

85

Therefore even in the absence of a stated sustainability strategy several critical areas are likely to sustain. The programme sustainability could have improved had there been more paced time to see through the post ODF status, also involvement of schoolteachers should‟ve been more and instead of SO‟s providing the training in schools, teachers should have been used. As also highlighted in the PATS, CRPs could have a sustained role to play, particularly where they can pick up entrepreneurship or sanitation mart, given their own incentives, this area should be explored in the next phase.

5.5. Unforeseen Events Evaluation Q 5: To what extent was the project implementation affected and/or influenced by unstable, conflict or emergency situation? Programme design, TOC and PCAs lacked planning or comprehensive risk assessment of the security environment in the programme areas. Fortunately nothing significant happened that would have derailed the programme. IPs knowledge of local communities and recruitment of local staff helped in smooth running of activities. However there is a need to have a Risk/Disaster Strategy in place in future programme design/PCA. Although the programme was focusing on working in flood and security affected areas, the programme design, TOC as well as PCAs lacked any planning or comprehensive risk assessment of the programme areas and overall security environment. There was no detailed natural disaster risk assessment done, however, the latrine construction guidelines contain safety measures to counter impacts of natural disasters such as floods and there is a section detailing DRR in WASH. The PCAs also did not have a risk assessment or mitigation measurement in the event of securityproblem. Fortunately nothing significant happened during the course of implementation and the programme interventions were not derailed. There were a few instances reported where IPs staff was threatened or attacked e.g. one of the Prepared staff was shot at, in another incident a contractor in District Kohat was unable to complete the latrine construction job due to security situation, and there was also an issue with Muslim Aid not being able to obtain NOC from the government. Other unforeseen issues noticed were the delays in supplies and release of funds. Generally, even without having any strategy in place the IPs managed to respond to the situation in a manner to reduce any significant impact on the programme implementation. Feedback from the IPs suggests that they used the knowledge of local communities to make decisions on day to day basis, and in most cases having local staff also helped in ensuring smooth running of activities. IPs who had prior presence in district were better able to deal with any such incidences. Considering the above mentioned issues there is a need to have a Risk/Disaster Strategy in place in the programme design/PCAs so that it could be used at the time of implementation. The strategy should have clear protocols to respond to situations.

86

The strategy should be made clear to all and applied from field to office. Similarly, there should be a clear policy on IP staff security, possible insurance and clarity of responsibility and duty of care.

87

5.6. Human Rights Based Approach Evaluation Q 6: To what extent did the project observe HRBA and/or contributed to observance of Human Rights in Pakistan as an implicit objective of the project? Ethical Programme Design The design and implementation did not formally articulated ethical considerations though they were adhered to in practice, including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community. It is however recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. The evaluation TORs did not specify conducting an analysis of how ethical the programme design was, rather the focus for evaluators was limited to ensuring that the evaluation was carried out under the UNEG ethical guidelines. However for completeness of the evaluation process consultants have reviewed the programme in line with ethical considerations both in terms of design and implementation. At the design stage there is clear focus on addressing the needs of the vulnerable and extremely vulnerable. The programme also ensures that the range of primary stakeholders i.e. the beneficiaries of the programme are reached and their needs addressed through specific mechanisms and approaches for instance a mix use of CLTS and SLTS focusing specifically on the needs of the community and directly on the needs of children through school led approach. Similarly interventions and operational methods were designed so that women, men and children were approached directly so that their views and input as well as their specific needs and concerns could be addressed directly. The programme also demonstrates focus on specific issues such as disability and MHM in design though less so during the implementation stage. Process of using PRA to ensure vulnerability assessment is carried out as well as approaching community through Men and Women Social Organisers is a good practice and considerate of local context and cultural sensitivities. Moreover the use of local resource in terms of CRPs, Mart Owners and Masons ensured the involvement and ownership of the community in the programme process. Children‟s involvement is clear in the SLTS process and their views and participation in the process has been ensured. There was no particular mention of violations or breach in terms of ethical considerations during interaction with community, schools and children. However some operational oversights have occurred particularly in the areas of constructing DRR specific elevated water supply facilities and latrines, in which case certain privacy aspects (particularly for girls in schools) have been infringed, this could be addressed in the future through better planning and involvement of respective stakeholders.

88

Having reviewed both the design and implementation the evaluators conclude that though not formally articulated ethical considerations were adhered to in practice,including protection of the dignity, rights, and welfare of human subjects, particularly children, and respect for the values of the beneficiary community.However it is recommended that ethical considerations be made a formal and written part of design documents as well as implementation arrangements including in PCA documents with partners for future programmes. Programme Design and Implementation UNICEF uses a human rights based approach (HRBA) and works in partnership with communities, especially women and children (in planning, implementing and maintaining water and sanitation systems).The design of SPSP does not explicitly state this, however the design elements comply mostly with the HRBA, this was made part of the programme following recommendations from the RusFAD evaluation. The design and PCA refer to gender considerations as well as demonstrated equity focus At the contractual stage all IPs were required to follow Millennium Declaration, Core Commitments for Children (CCCs), the Convention on the Rights of the Child (CRC) and the Convention on the Elimination of All Forms of Discrimination against Women (CEDAW). At the implementation stage expected HRBA approaches and techniques are observed as strengths of the programme, including,

- participatory rural appraisal, - participatory situational assessment, - community dialogues (within villages, and with district officials) - Village Sanitation Committees (allowing to build village consensus, promotion

of hygiene and responsibility of liaising with officials) - CAP (articulating community‟s development needs and creating an action

plan to move towards these). - Joint activity with programme IPs for ODF declaration and certification. - Children‟s WASH Clubs, helping them take charge of their health and hygiene

conditions in the school. As already noted above this was not done through an explicit statement of HRBA, therefore there are certain weak areas where the programme could have done more and better. The programme does not use a rights based language, therefore the articulation of water and sanitation as a right is not made, nor is the emphasis clear on the roles and responsibilities of the right holder as well as the duty bearer who is responsible for provision and upholding of the right. Also the programme could have done more in linking the VSCs with Duty Bearers and could have tried to access resources (water and sanitation) in an effort to demonstrate the HRBA. Instead the programme provides most of the water and sanitation facilities through IPs and school facilities through independent contractors, in many ways not in line with HRBA principles. Moreover the staff, and IPs do not seem to be fully aware of HRBA as an approach and therefore areas which could have been stronger, come across as not being addressed. One such example is the lack of policy influencing and push on providing WASH facilities in government schools through the Education department. Another is

89

the lack of jointly with community advocating for rehabilitation of water schemes. HRBA in full spirit would also have required more full on participation of the community in the monitoring of the programme including being involved in transparent bidding of community level physical infrastructure activities, which was not the case. Monitoring Arrangements UNICEF’s approach in this programme was to mainstream HRBA across the programme design and implementation framework, including even in the context of monitoring. Third Party Monitors were therefore required to not only monitor the quality of processes but also on progress in activities and outputs under the framework including aspects such as proper identification of extremely vulnerable in the community. Similarly monitoring that the PRA process was followed as per the UNICEF checklist to ensure inclusion of marginalized in the planning process. The monitoring checklist included other HRBA related indicators, which was further strengthened through the Programme Results Framework which mainstreamed income quintile based desegregation as well as gender based desegregation. Other aspects including a monitoring focus on access for the disabled. In addition to this UNICEF also conducted the WASH Equity study which helped informed UNICEF of the public spending in all districts of Pakistan on WASH as well as access to WASH facilities in terms of income quintiles, which could be compared with baseline and end of phase evaluation figures to demonstrate increased access to water and sanitation (declared as basic rights in the various treaties that Pakistan has ratified and is a signatory of). Contribution towards MDGs PATS is the formal strategy by Government of Pakistan towards attaining MDGs related to sanitation. The programme has been successful in increasing the proportion of population having access to sanitation facility to 65% from 44% at the time of baseline in the programme areas. Moreover this achievement is higher than the current national coverage of 50% (which includes rural and urban areas), thus one can confidently state that there has been some attributable increase in access to sanitation facility resulting from this programme even at the national level. Child Friendly Facilities In line with the Islamabad Commitment, 2010which was signed by all provinces relating to Child Friendly Inclusive Schools, the programme has been successful in providing child friendly WASH facilities.

95%

64%

42%

30%

73%

77%

97%

77%

77%

0% 20% 40% 60% 80% 100% 120%

Well Lit Classrooms

Ventilation in Classrooms

Dustbins in Classrooms

Availability of Soap

Is Latrine In Usable Condition

Water Availability in Latrines

Availability of Latrine Facility

Handwashing Facilities

Availability of Drinking Water Points

School Facilities Attributes

90

The programme has been able to provide 97% of the schools with latrine facilities, with 77% having access to water for latrines (including piped as well as through bucket). In addition to this SPSP was able to form WASH Clubs, train teachers, reactivate SMCs which should help in addressing issues to ensure the school facilities remain child friendly. Disability Addressing of disability remained a weak aspect of the programme at both community as well as school level in terms of sensitisation as well as actual infrastructure designs and sanitation supplies. At the proposal stage the programme clearly stipulates that all activities will be designed and carried out with due regard to disability. The baseline study also highlighted the need to collect desegregated data for the disabled, additionally Third Party Monitoring mechanisms also included monitoring of facilities including assessing their suitability for the disabled. Lastly, under the SLTS component the facilities were to be child friendly including accessibility to children who are disabled. However this emphasis has not been translated into action during programme implementation. None of the school or household latrines meet the requirements for being accessible to the disabled. The hand washing points are based on traditional design which means it can be limiting for a child having physical disability. Due to DRR concerns both water as well as Sanitation facilities have been designed to be elevated from the ground, however no arrangementswere made to provide suitable ramps, instead both facilities have stairs. While visiting some hand pump facilities which did have ramps (although very steep), the evaluators were told that these ramps are provided so that community can bring wheelbarrows/trolleys upto the hand pump so that it is easier to carry the water, failing again to highlight any possible link to disability. Duty Bearers Capacity Building The programme has carried out orientation and capacity building activities with duty bearers (including district budget seminar). However they have not been focused on HRBA principles and they do not formally and explicitly hold duty bearers responsible for respecting people’s rights, or identify delivery of basic services such as sanitation and access to water as a basic right for the people. Moreover, whilst the programmecreated community structures such as the VSC as well as the CRPs, there has been little effort and progress in connecting them to duty

Figure 29: School Facilities Attributes

91

bearers or political individuals to ensure a statement of rights is placed in front of them, or resources and services are demanded of them. Formal and Informal Mechanisms In some cases IPs with the cooperation of district government were able to set up District Committees to oversee the progress, implementation of the SPSP programme, however in most casesthe relationship remained informally structured with IPs picking and choosing who to interact and deal with given the responsiveness of the duty bearer (interaction varies significantly from District Commissioner down to Union Council Secretaries).

The programme however encourage Duty Bearers to interact and visit the communities through the orientation sessions, opportunity to visit and certify ODF Villages, engagement with IP field staff vis-a-vis Union Council Secretaries, BHU Staff, Govt Field Extension Workers. At the community level CRPs and VSCs (men and women) have been developed as the informal mechanisms for articulating community needs and rights. Similarly at the school level Children‟s WASH clubs (for boys and girls) help raise health and hygiene as well as WASH issues. However the process of articulating and voicing rights needs more capacity building to ensure these mechanisms remain effective as well as sustain. Similarly more work is required in building duty bearers understanding and ability to respect, protect as well as provide for people‟s rights.

92

5.7. Partnership and Financial Management Evaluation Q 7: To what extent the project has observed and applied UNICEF’s standard financial management practices with Implementing Partners as well as with Government? UNICEF guidelines for partner selection were complied with,though the process could have been made more clear in terms of decisions and the process used to achieve them. The partner selection nevertheless was appropriate with partners having relevant WASH experience and geographical presence. The umbrella arrangement posed operational and implementation challenges and needs to be reassessed. Consultation with Government Departments and their involvement in the programme remained weak, while issues identified during the bottleneck analysis seem not to have been adequately addressed. UNICEF guidelines for partner selection were complied with, however the guideline does not highlight how the committee should have made a decision, on what basis to shortlist and select partners particularly in the event of qualification of greater number of organizations than required. The proceedings of committee deciding on final partners was not made available for evaluation. Evaluators also noted inconsistencies in scoring vis-a-vis final selection of partners. However information was not available to determine how these inconsistencies affected the programme during implementation. To a large extent appropriate partners were selected (though no specific capacities beyond an element of WASH experience and geographical presence/outreach in the area were assessed) while filtering for partners. Moreover evaluators were able to identify an exception to geographical presence criteria in one province. Evaluators were able to identify through TPM reporting that there were operational and programme delivery challenges with the umbrella arrangement in Sindh (NRSP as lead agency with three sub-IPs) as well as in the partnership in Baluchistan (NCBP with WDCO). Micro assessment methodology comes across having its own limitation, however through it, UNICEF did identify the areas of improvement.Though it seems UNICEF ended up picking and choosing among the various risk areas for interventions and no record was available to explain this selective approach and the circumstances that contributed to it. Some basic level capacity building was done which helped to mitigate the project related financial risks (including training of the finance teams, introducing independent internal audit functions). Government Partners Coordination and Management Arrangements UNICEF provincial offices were not able to produce enough documents where coordination meetings at provincial and district levels and issues related thereof could have been reviewed. Evaluators had to rely on the interactions with UNICEF staff and Provincial Government department‟s representatives.

93

Provincial Government departments were not adequately consulted and were not taken onboard in the programme design even to the extent that details of roles and responsibilities expected from them had not been properly shared. For example, under the SLTS component, the education department at provincial level was not on board in terms of selection of schools, teachers training and formation of Wash clubs as well as details of operation and maintenance of latrines that were constructed. While orientation on PATS was provided in the beginning of the programme, government departments were not fully aware of key programme activities nor was an approach agreed with them to ensure the sustainability of the programme after exit phase. Evaluators found that situation at district level was relatively better where CSPs were more active in engaging with district management. Though government departments had little knowledge and involvement on incentive and rewards component. The engagement level at district level varied from district to district and was more dependent on personal and organizational reputation of the implementing CSP. Capacity development plans for provincial level departments in 2013 were not rolled out with allocation of budgets to carry out the activities nor the bottleneck analysis that were conducted with government departments in four provinces were fed into the programme or reflected fully during programme implementation. Financial Risk Management PCAs contain sufficient clauses on anti corruption and bind the NGO partners to follow the UNICEF policies. However evaluators do not have any mechanism to verify what alignment initiatives have been taken to ensure that NGO partners and Government partners are on board and fully understand these policies and comply with them in their routine operations. This fact is also not checked during spot checks. Limited sharing of further information makes a conclusive assessment difficult. Procurement Procedures for Goods and Services The evaluators neither have the mandate nor access to key information on procurements by UNICEF and NGO partners as this comes under the financial audit domain. However as a process, supply management assessments were carried out for NGOs as mentioned on PCA-Review Committee checklist before signing off the PCA.

94

5.8. Unexpected Findings The unexpected findings resulting from the implementation of SPSP Phaseinclude unravelling of the critical link between open defecation exposes women to accusations of illicit relations and can lead to being subjected to extreme violence. Similarly Access To Sanitation Facilities and Reduction in Violence Against Women. The evaluation findings have contributed to unravelling a critical link between open defecation and extreme cases of gender violence. A significant number of respondents (including Government Officials, CRPs and Women) across different geographical locations (Sindh and Punjab) shared that the construction of latrines have reduced the incidences of violence against women such as karo kari (honour killing). To them, the construction of latrines has reduced the need for women to step out, and hence reduced the probability of being accused of illicit relations and being subjected to violence. In a narrated instance a mother married off her daughter to a mentally handicapped person in order to save her daughter from Karo Kari. There is other emerging evidence for instance in May 2014, two girls were gang-raped and hanged in Uttar Pradesh, India, they had gone out to look for a place to defecate. A WaterAid study in the slums of Lagos in Nigeria also showed that a quarter of women who lacked access to sanitation had first or second-hand experience of harassment, threats of violence or actual assault linked to their lack of a safe, private toilet. Amnesty International has released similar studies from Kenya and the Solomon Islands.Results of the SPSP Phase 1 post-KAP survey suggest that 83% of the respondents feel it is unsafe for women to go out to defecate in the open. Though an indirect result, it is a significant aspect and therefore should be incorporated in a cognizant way in future programmes in BCC strategy and demand creation interventions. Shame, Shock and Disgust versus Pride, Self Respect and Dignity There is an ongoing debate on which approach to use, it must be noted that the purpose of both the approaches is to trigger the CLTS process, with a larger objective of saving children‟s lives and prevention from diseases. Experience of SPSP and feedback from several stakeholders (IPs, CRPs and VSCs) suggests that to bring about a change in people‟s behaviour they resorted to the use of both approaches within the same programme and resultantly ensure more effective and wider reaching triggering. Business Incentive and Opportunity Is a Larger Motive for Response. Another unexpected finding that was highlighted, comes from the action research being carried out by DSI which was commissioned under SPSP by UNICEF. In their assessment on a comparative basis identifying an individual from the community as a CRP or Sanitation Mart Owner and training that person for 3-5 days to carry out sanitation promotion/marketing activities or conducting business is neither sustainable nor assures continued interest and incentive. They adopted an approach of directly interacting with regional manufacturers (for instance one producing in Peshawar with a supply chain to around 4-5 districts) exposing them to the potential of business in the rural areas where demand had been created through

95

CLTS, Once the manufacturer realises the opportunity he takes over the process of both marketing as well as ensuring supplies are made available, this in their assessment is more sustainable as well as ensures continued interest from the manufacturer.

6. CONCLUSIONS & LESSONS LEARNED The evaluation concludes that the SPSP Phase 1 was implemented successfully though with varied levels achievements in terms of programme components, interventions and results. This conclusion is made based on observations from field visits, discussions held with beneficiaries in visited villages, government staff and IP‟s in the four provinces and FATA as well as from consultations at national level, desk reviews and comparison of the Pre-Post KAP. Under the programme the poor and extremely poor communities in the rural areas benefitted from access to sanitation services, improved access to water, health and hygiene promotion and through supply side sanitation interventions. The programme maintained a gender and vulnerability focus which is demonstrated in operations and results. CLTS and SLTS were found to be complimentary to each other while community mobilisation training and capacity building proved effective in achieving ODF declaration status. The field interaction and involvement of government representatives during the ODF verification/certification process comes across as a strong point. However the programme could have developed stronger and better engagement with government departments with particular focus on drawing down public sector funds from provincial to district level and ensuring that communities received awards/ incentives following ODF verification/certification. The presence of a well-defined monitoring and evaluation system since the beginning of the Programme resulted in improved quality of progress reporting in terms of adequacy and consistency of information as well as informing management decisions and course correction during implementation. The programme also contributed in developing WASH training manuals, health and hygiene promotion material and conducting action research. Moreover the baseline and evaluation studies commissioned during the programme have created evidence and provided data that can be used to inform future programme direction as well as contribute to review of PATS. Future programme would however need to focus more on modality of subsidy provision, methods of demonstration latrines, sanitation marketing (particularly developing sanitation marts and entrepreneurs), and availability of low-cost technologies for sanitation. The evaluation identifies lessons and proposes recommendations that the programme can take into consideration while designing the next phase or future programme. Overall the experience of SPSP Phase I confirms that PATS can be implemented on a large scale with key requisites such as increased capacity of and advocacy with government, sector stakeholders and communities to apply PATS approach effectively.

97

Lessons Learned Following four key lessons emanate from having implemented the programme, which have a broader implication and also help add to general understanding and body of knowledge around these issues, both in the context of future WASH programming by UNICEF or others. Reflecting and incorporating these into the PATS approach and informing UNICEF‟s Global CATS approach, or simply across other programmatic contexts where these may apply or overlap. Access To Sanitation Facilities Has a Role in Reducing Violence Against Women. As noted above in the section on unexpected findings, several stakeholders have expressed their views on the benefits of having access to sanitation facility which not only provides sanitary benefit but also offers privacy and even more importantly security for women, limiting the chances of being exposed to harassment or false accusations of sexual misconduct which may lead to incidences such as karo kari. Results of the KAP survey suggest that 83% feel it is unsafe for women to go out to defecate in the open, while only 7.6% feel it is safe. The lesson that can be derived from this instance is that though indirect, the benefit of contributing to a reduction in violence is a substantially important aspect and should be learnt from for similar as well as other interventions which may have the potential of reducing or limiting harm, harassment even violence against women. Shame, Shock and Disgust versus Pride, Self Respect and Dignity There is an ongoing debate on which approach to use, during RusFAD UNICEF decided to observe the Shame, Shock and Disgust approach, whilst based on findings and evaluation recommendations it was decided that the next programme i.e. SPSP Phase 1 will implement the programme using Pride, Self respect and Dignity approach. However in practice as commented by the IPs, in many cases they resorted to a mix of the approaches. It must however be noted that the purpose of both the approaches is to trigger the CLTS process, and the larger picture entails saving children‟s lives and preventing from disease. Therefore for a change to be brought about in people‟s behaviour if Pride, Self Respect and Dignity approach does not trigger it resorting to Shame, Shock and Disgust should be resorted to, rather than fixating on an either or situation. Business Incentive and Opportunity Is a Larger Motive for Response. DSI‟s action research (already highlighted above) is suggestive that tapping into the interests of manufacturers and those who already are having business risk bearing attributes will facilitate supply for their own incentive and expansion of market. What is important is to bring a realisation to the market opportunity that is provided by serving these underserved areas, whether it is sanitation supplies or projects with other similar offerings (for instance supply of medicine vs doling out free medicines). Clearly the continuity and sustainability is self evident in those stakeholders who hold an incentive versus community workers that may be put through a 3-5 day training, to deliver services during the life of the programme, but ultimately not sustaining continuity of those services.

98

Using Subsidy to Address Equity Can Lead to Dependency The purpose of the subsidy was to ensure equity aspects were addressed i.e. a fixed percentage of people considered vulnerable and extremely vulnerable (established on the basis of PRA with villages) would received subsidy for construction of household latrines. While this concept does give some comfort the some of the vulnerable are also receiving sanitary facilities, it also leads to problems when.

d) there are more vulnerable and extremely vulnerable families with the programme area but are not recipient of similar subsidy,

e) families who may have considered access to such facilities useful and beneficial but decide to hold back putting their own resources into it, given free handout in the form of subsidy was being made available, thus acting as a detrimental incentive.

f) Lastly by the end of the project those who have not received this subsidy, remain dependent and wishful for similar other interventions that may bring with them benefits such as subsidies, while delaying the good that they could have benefitted from had more emphasis been on self and community based help mechanisms.

Therefore using subsidy to incentivise or address equity needs to factor in people‟s response behaviour and possible negative impact that it may have in the longer run.

7. RECOMMENDATIONS Following are key recommendations along with priority and identified stakeholder(s). The recommendations were drawn through a comprehensive and systematic process, including drawing on lessons learnt by the stakeholders (UNICEF, DSI, RSPN, Range of IPs, other development partners, government representatives as well as community representative) and their propositions that could improve the programme. Following this process the evaluators were able to consolidate key recommendations and present to the Evaluation Reference Group in a plenary session (including UNICEF representative from Regional Office, Islamabad and Provincial Offices, Government Officials as well as Donor Representative) during the two day workshop. The recommendations were then discussed within smaller groups, arguing their respective importance, clarifying their objective and then agreed through a mutual process with agreement of the Evaluation Reference Group. No. Recommendations Priority Stakeholder

1. Informing PATS Policy UNICEF has garnered PATS related experience through implementation of RusFAD and SPSP (for instance whether to use Respect and Dignity or Shock, Shame and Disgust method), moreover lessons and findings are also coming out of the Action Research carried out under SPSP by DSI (suggestive that involving manufacturers with clearly identified incentives and opportunities goes a longer way in facilitating sanitation supply). Similarly other development partners such as WaterAID have tried other variations (for instance not offering household level subsidy) while interpreting and implementing the PATS approach. It is therefore recommended that UNICEF commission a joint research to study the various PATS models being implemented in Pakistan and the adaptations that have been brought about to ensure ODF status. The research should also highlight the differences in approaches due to contextualising of cultural and geographical considerations and sensitivities such as found across the Provinces. This research should thus be used to inform and revise the PATS policy and depending on scheduling be showcased in the next PACOSAN. Dissemination of the study would also help bring attention to PATS and the effort to make progress towards MDG targets.

Medium Term UNICEF, Development Partners, Government

2. Knowledge Management It is clear from the SPSP evaluation as well as from interacting with the various stakeholders that there is substantial and good work being done on PATS (e.g. WaterAID, plans of WSSCC as well as work being carried out by Punjab). This work is also being documented and adding to the existing knowledge base.

Immediate UNICEF (along with Development Partners)

100

No. Recommendations Priority Stakeholder

It is therefore recommended that UNICEF takes a lead role (along with other development partners) to ensure that an online repository is created, which makes programme documents, evaluations, action research etc. relevant to PATS programmes (including those generated during SPSP Phase 1, as well as from RusFAD) widely available. The repository should provide open access, be user-friendly and allows authorised people to upload documents.

3. Harmonisation & Coordination From the interaction with various stakeholders as well as based on the evaluation findings it is clear that the sanitation sector critically lags behind other sectors in terms of harmonisation as well as coordination mechanisms. It is recommended that UNICEFshould use its stature and leverage its relations with other development partners, federal and provincial government to ensure sector wide coordination forums at national and provincial levels are formed/made active for advocacy, policy making and resource mobilisation/allocation. Furthermore for coordination at project level it is recommended that Provincial and District Coordination Committees be formed/notified through government with PHED/LGRDD as lead to leverage provincial and district level support and ownership.

Medium Term UNICEF, Development Partners, Government

4. Policy Influencing & Advocacy for Government Allocations Based on the evaluation findings following things are evident,

a) drawdown of government resources remained weak even though it is considered intrinsic to PATS that communities are linked to Government Departments at district and sub-district level for continued work in achieving and maintaining ODF status, for which government resources should also be mobilised,

b) also the scale and magnitude of the sanitation challenge in Pakistan necessitates the involvement of Government to ensure scale-up and sustainability.

Immediate UNICEF

101

No. Recommendations Priority Stakeholder

It is therefore recommended13 that UNICEF should use its position and influence and work top to bottom from province to ensure budgetary provision for awards, incentives and rewards to departments at the district level. A standardised approach and methodology should be adopted by UNICEF in Islamabad as well as the Provincial Office for policy influencing at provincial level and implementation at district levels (with variations to suit provincial or geographic context). This should be based on written MOU or agreement between UNICEF and Government Departments. In SPSP Phase II, UNICEF should pilot PATS implementation with Public Stakeholders in at least one Province (which is receptive and committed to addressing sanitation issues).

5. Integration with Other Sectors It is evident from the evaluation as well as through interaction at the ground level with Government Departments that to achieve sanitation targets engagement and committed action on a multi-stakeholder basis will be required. There is a clear need to advocate mainstreaming of sanitation across other sectors (including Health, Nutrition, Education and Government Departments). This should include cross sector convergence on WASH policy and programmes, which can include advocating for hygiene to be led by Health Department, while being mainstreamed into the education curriculum. Moreover it can look into options for improved usage of SMC funds while ensuring regular monitoring of Schools including monitoring of WASH facilities to identify and address defunct facilities. It is therefore recommended as a minimum that UNICEF engages with these departments both at the Provincial and District level, while ensuring that they come together (it is notified as part of SPSP Phase 2 agreement with government) at the provincial level as a Coordination and Steering Committee headed preferably by PHED, and at district level an Implementation Committee headed and reporting to the District Administrative head. This would also allow other stakeholders to use similar government structures for better coordination and implementation of their respective WASH programmes.

Medium Term UNICEF, Development Partners and Government

13 The evaluators note that this is an ambitious recommendation, however it is considered necessary for scale-up and sustainability so that in the longer term progress can be made at the national level in meeting MDG Target.

102

No. Recommendations Priority Stakeholder

6. Duration The evaluation findings and the two-day workshop with the Evaluation Reference Group highlighted that there was no specific time framework (minimum incubation period) for PATS. Examples from different provinces as well as IPs ranged from as short as 3-4 months to 6-9 months. However based on substantial and in-depth discussion a clear recommendation for future programming was agreed which should take into account and provide adequate duration for the following phases of the programme,

- Start-up/inception phase (PCA signing, hiring of staff, IP inception workshops etc) - Implementation phase of 12 months or more for CLTS and SLST (based on time required for

effective community mobilisation, as well as factoring in community life/work cycle aspects, school holidays etc)

- Exit phase (should include exit planning, sustainability of interventions as per programme strategy, and post ODF monitoring).

Immediate UNICEF

7. Gender Equality, Equity, HRBA (Disability) While SPSP Phase 1 was clear in mandating and focusing on gender equality, equity and disability, evaluation findings suggest more clarity, working mechanisms and definitions need to be articulated, along with describing the process that will ensure these aspects are factored into the programming and implementation. It is therefore recommended that clear processes need to be articulated in the programme design, the TOC should explain the change process and results, while PCA should include sections detailing specific interventions, criteria and processes that IPs have to implement ensuring inclusiveness (such as for PRA, School WASH Facilities, more targeted BCC, disability integration in Latrines) in line with UNICEF guidelinesto address gender equality, equity and vulnerability along with special needs of the disabled.

Immediate UNICEF, Government

8. Exit Strategy & Sustainability

The evaluation highlights the lack of clarity about the mechanisms and time duration that is required to ensure a smooth exit and put in place sustainability mechanisms.

It is recommended that for SPSP Phase 1 implementing partners a minimal operational support for

Immediate UNICEF

103

No. Recommendations Priority Stakeholder

six months is provided in the districts of operation to ensure gains of the implementation phase (ODF Status - declaration and certification) are not lost to slippage. For the next phase and future programmes it is then recommended that a Theory of Change needs to be articulated, and a Sustainability Strategy drawn in line with it, focusing on sustaining ODF status (for which a study can also be commissioned examing what needs to be done to sustain ODF status). PCAs should include specific IP exit plans, and those stakeholders who will be ultimately responsible for sustainability should be clearly informed of their role along with their consent to delivery it (CRPs, VSCs, Wash Clubs as well as role expected from Government Departments and their extension workers such as Community Development Officers, Union Council Secretaries, LHWs and School Teachers etc).

9. Training & Mentoring Findings suggest that using a central training organisation proved effective, though with some constraints which if addressed should help make it even more effective. It is therefore recommended that the programme should continue the use of a central training organisation, however the level of resource provision, staffing should be according to the magnitude of activities over the duration of the programme. Mentoring should be need based and resourced from the training organisation based on IPs need or as identified through progress monitoring. Training events should be announced in a timely manner to allow maximum attendance and should benefit all who are relevant, while taking into account aspects such as community work cycle and availability.

Immediate UNICEF and Training Partner Organisation

10. Supply Facilitation Sanitation Mart Owner/Entrepreneur Distinction between Sanitation Mart Owners and Entrepreneurs should be reassessed and if required dropped. Innovative results emanating from Action Research carried out by DSI need to be examined to evolve an effective role for entrepreneur (including the element of risk taking, market forces of demand and supply, as well as issue of financing). Community Resource Persons (CRPs) The concept of service fee for Community Resource Persons (CRPs) worked and should continue

Ongoing UNICEF (with advice of DSI, RSPN)

104

No. Recommendations Priority Stakeholder

for the period of mobilisation. Options should be considered to utilise these resources for sustained focus on sanitation by transforming them into small entrepreneurs and help expand the capacity of the market to supply services and products (this could be made possible by giving them a reward once they have achieved their agreed targets as an incentive. Household Subsidy Household subsidy should be withdrawn and other mechanism such as support with material through voucher system in post humanitarian crisis situation and collective incentives can be tested. For equity purposes and the needs of the vulnerable revolving credit/contributory fund be considered as an option to be considered managed by the VSC to ensure the sanitation requirements of vulnerable are also met, this can be done through options such as setting up revolving credit with the VSC.

11. Low-Cost Sanitation and Drainage More options for low-cost sanitation need to be explored. This should include options for low cost pits (possibly using low cost pit lining, increased use of local materials), as well as for superstructures (which should make use of local materials while ensuring usability of the facility by the target community). It is recommended that options for low cost end of pipe solutions should be identified (if required a study be conducted to inform options)and new ideas piloted in some locations given the demand for drainage articulated by community during SPSP Phase 1.

Ongoing UNICEF

Table 19: Evaluation Recommendations