Evaluating Community-Demand Driven Water & Sanitation Approaches
in Rural Maharashtra
…. a story of taps, toilets, and behaviors
Priti Kumar 1 and J Murty 2
S. K. Pattanayak 3, Sumeet Patil 4, K. Lvovsky 5, N. Raghava 6, J-C Yang 7, C. Poulos 7
1, 2, 5, 6 The World Bank
3 Duke University, USA (formerly RTI International)
4 Neerman, Mumbai (formerly RTI International)
7 RTI International, USA
2
Overview of Presentation
Context
Study Objectives
Study Methodology and Implementation Details
Results
Lessons Learnt
3
Context for Study
Child Diarrhea - key underlying link for India’s MDG
targets
11th 5yr Plan targets on WSS are ambitious, exceed the MDG goals,
and require multisectoral approaches
Significant success and scaling up RWSS reforms– (e.g.,
Jalswarajya, TSC)
Variety of settings, delivery mechanisms, and interventions
Focus on creating demand at the community level for increasing use of IHL; Incentives to the poor.
Need to understand how to improve health and non-health benefits of WSS programs
Limited knowledge in this area. Households behaviors or adaptation
modify the impacts, but very little is known
4
Context for Study
Importance of Water and Sanitation for health is not disputed. BUT…
Major knowledge gaps. Scientific literature is thin and inconclusive Limited knowledge on effectiveness of EH interventions (e.g.,
private/public taps, water quality treatment, long term behavior change etc.)
Until now more emphasis on clinical interventions (ORS, vaccines) for diarrhea reduction
Past evaluations do not meet quality standards Cross-sectional assessments (lacking baseline, control, statistical
power)
Insufficient designs for measuring broad range of program impacts (e.g.: equity)
5
Study Objectives
First scientific study of this kind in India to examine the
benefits of community demand driven approaches in RWSS
programs that are supported by GoI / GoM policies
Address question in a scientifically rigorous manner:
Do participatory water and sanitation projects deliver health, and socio-economic outcomes? (Maharashtra)
What factors affect / modify these impacts?
Contribute to global knowledge on health and WSS.
6
Specific Objectives
Determine whether WSH interventions due to Jalswarajaya Project cause differences in water, sanitation and health outcomes in villages that participate in the Project
Measure broader impacts of WSH interventions on rural livelihoods (e.g.: time savings, investments in coping activities, improvements in convenience/privacy and indirect benefits to caregivers)
Develop proxy indicators for monitoring that are correlated to health impact (water quality at point of use/source, retention of hygiene knowledge, use of latrines, condition of public stand posts, etc.)
Learn lessons for future implementation and evaluation
7
Why undertake the study in Maharashtra?
The Jalswarajaya Project was an ongoing Bank operation under implementation, with the schedule most suitable for the study.
A large RWSS intervention with a comprehensive range of WSS service packages, good geographical coverage.
Strong and committed government counterpart in GoM.
8
Study Team Partners
Rajiv Gandhi Drinking Water Mission, Govt of India –
Department of Water Supply and Sanitation, Govt of Maharashtra –
Multidisciplinary technical advisory group – WHO, USAID, UNICEF, Indian Council of Medical Research, National Institute of Epidemiology, CMC Vellore in GoI (reviewed/advised during initial phases of study design and development of study protocols)
RTI International (USA) – Principal investigating agency– study design, management, analysis and reporting.
TNS Mode (India) –field and data collection support
9
Jalswarajya – water self-rule (Maharashtra)
10
Interventions under Jalswarajya
Philosophy
community demand driven (Panchaayati Raj institutions)
decentralized delivery
cost sharing (10% of K, and 100% of O&M)
resources for capacity building and local institutional strengthening
2800 villages from 26 districts (400 households per village)
Pilot:30 villages from 3 districts
Phase I: Batch 1 - 225 villages from 9 districts
Phases I (Batch 2), and Phase II (17 additional districts)
Timeline
pre-planning – 1 month (IEC, community mobilization - VWSC)
planning – 6 months (VAP, appraisal and sanction)
implementation – 9 months
operation & maintenance – on-going
Process: inform, apply, select, plan, finance, construct, monitor
11
Study Design
Not a controlled experiment but evaluation of a real life program
Cannot randomly select treatments and control
Cannot dictate selection and timeline of the project. Evaluation had to be non-intrusive
Jalswarajya in Maharashtra: Quasi-experimental (matched) sample of 242 villages
Treatment: 95 from Phase 1, Batch 1 from 4 dry districts
Control: 147 matched controls from the above districts.
Treatment and controlled had similar probability of being selected in Jalswarajya Project
District selection Focus on rural, dry, and drought-prone (exclude coastal and urban
districts) and 4 geographic regions of Maharasthra – Buldana, Nashik, Osmanabad, Sangli.
Total: ~10,000 households
12
Study Design
Baseline (May & Aug 2005) and Endline (May & Aug 2007) data collection for intervention and control villages
Select controls using a matching technique to estimate counterfactual outcome and removes bias (Propensity Score Matching)
Difference in difference (before/after and with/without) estimation of impacts : Estimates population average treatment effect. Advantageous and robust compared to:
Only Before-After: Cannot demonstrate specific success of the program without controls
Only With-Without: Cannot demonstrate “progress” from a baseline. Cannot deal with different starting points.
Multiple observations from the same cluster (community) – ~50 HHs per village (randomly select from list and map all U5 hhs)
13
Data: Measurements
Outcomes:
Household pit latrines (IHL): constructed, operational and in-use
Diarrhea frequency & severity (> 3 episodes in 24-hr, 2-week recall)
Arm circumference
Additional parameters:
Individual - sex, age, education, anthropometric, health
Household - family size and composition, housing conditions, asset holdings, occupation and expenditures, SEC, services etc
Community – roads, electricity, environmental sanitation, employment, clinics, schools, credits, markets
Institutional - main governmental and NGO programs, local government size and composition
Water quality (E. coli & total coliform) – lab tested, spread plate method
14
Household Survey
Respondent - Primary Care Giver for U5 child
Water samples collected from approx 50% of surveyed households
Modular questionnaire
Knowledge, Attitudes
Household demographics
Sanitation Behaviors – outputs and outcomes
Hygiene Behaviors
Water Sources and their use
Water Treatment/safety behaviors
Food safety behaviors
Environmental conditions – HH and community
SEC & Budget constraints
Community Participation
15
Community Survey
Approx 150 minutes. Respondent – sarpanch, GP member, Informal leader, Doctor, etc
Water samples collected from up to 10 in-use drinking water sources
Modular questionnaire design
Background: population, households, area, arable land, major crop grown
Public infrastructure: roads, water supply, sanitation, hygiene, electricity, clinics, schools, STD booths, telegraph offices, post offices, credits and markets
Environmental sanitation: general cleanliness, drainage, animal and household waste, use of water sources, open defecation practices
WSS scheme: Jalswarajya, VWSC, NGO/SO, scheme details etc
Development Programs: Health, education, women support etc
Economy: employment opportunities, major governmental and NGO programs, prices
Local government: structure, composition, activities
16
Results-1: Outputs
Access to water supply and sanitation
increased in two years across programmes
Clear indication that investments are
reaching villages
17
% Households Using Taps
(by intervention, season, and year)
I indicates the 95% confidence interval.
35%
45%41%
44% 42%
51%
40%
47%
0%
10%
20%
30%
40%
50%
60%
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = 13%*** DID = ~
18
% Households Using Toilets
(by intervention, season, and year)
I indicates the 95% confidence interval.
16%
34%
12%
22%19%
34%
14%
21%
10%
15%
20%
25%
30%
35%
40%
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID= 8*** DID= 6***
19
Results-2: Outcomes-Water
30% reduction in time spent in walking to and waiting at water source
(Access) across the board
Dry Season: 34 minutes in 2005 dropped to 22 minutes in 2007
Rainy Season: 23 minutes in 2005 dropped to 16 minutes in 2007
Across the board increase in LPCD consumption by households (Approx
30% jump)
10 to 11 LPCD increase from baseline levels depending upon season and
type of villages. (baseline levels: 27-30 LPCD increased to 38-40 LPCD)
Coping costs (water collection, treatment and storage, walking to
defecation site) (40% reduction)
Dry Season: Reduced from Rs 976 in 2005 to Rs. 590 in 2007
Rainy Season: Reduced from Rs 857 to Rs. 490 in 2007
20
Monthly coping costs of inadequate WSS (in Rs.)
(by intervention, season, and year)
I indicates the 95% confidence interval.
DID= -202* DID= ~
926
605
836
458
869
513562
1064
0
200
400
600
800
1000
1200
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
21
Household water consumption in LPCD
(by intervention, season, and year)
I indicates the 95% confidence interval.
30
41
31
39
27
38
27
38
25
30
35
40
45
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = 4.5** DID = ~
22
Results-3: Outcomes on Behavior
Change
Not much impact seen on behavior change
Measured indicators are:
Water treatment
Safe water handling at hh level
Hand Washing
23
# of Critical Times a Child Washes Hands
(by intervention, season, and year)
I indicates the 95% confidence interval.
1.1
1.3
1.2
1.3
1.7
1.2
1.7
1.2
0.0
0.3
0.6
0.9
1.2
1.5
1.8
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = ~ DID = ~
24
% Households Treated Water before Drinking
(by intervention, season, and year)
I indicates the 95% confidence interval.
63%68%
63%67%
75%69%
76%68%
0%
20%
40%
60%
80%
100%
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = ~ DID = ~
25
# of Safe Water Handling Techniques HH Practiced
(by intervention, season, and year)
I indicates the 95% confidence interval.
1.41.3
1.41.3 1.2 1.2 1.2
1.1
0.0
0.4
0.8
1.2
1.6
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = ~ DID = 0.1*
26
E. Coli in Household Drinking Water
(by intervention, season, and year in log10)
I indicates the 95% confidence interval.
0.6
0.8
0.5
0.8
1.1
0.7
1.2
0.7
0.0
0.3
0.6
0.9
1.2
1.5
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = ~ DID = 0.23*
27
Diarrhea Prevalence among Children under 5
(by intervention, season, and year)
I indicates the 95% confidence interval.
11%
9%10%
9%
12%
8%
13%
7%
0%
3%
6%
9%
12%
15%
2005 2007 2005 2007 2005 2007 2005 2007
Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages
Dry Season Rainy Season
DID = ~ DID = 2%*
28
Finer Analysis
Two types of finer analysis were carried out
Impacts due to higher coverage
Impact due to Jalswarajaya approach
29
Subgroup Analysis
1. Group 1: 80 control villages with >50% Taps, Toilets or Both
2. Group 2: 21 JS villages <50% taps or toilets.
3. Group 3: 10 JS Villages with >50% of HH with toilets but <50% using taps
4. Group 4: 31 JS villages >50% of HH using taps, but <50% using toilets
5. Group 5: 17 JS villages >50% of HHs using taps and toilets.
Control: 83 villages with <50% using taps and toilets
30
Subgroup Analysis Continued…
In dry season:
Group 5 have less e coli in their HH water
Lower COI and diarrhea in group 5
In rainy season:
Coping costs decrease in group 5 (more accessibility)
in home WQ worse in 1, 3 and 4. In group 1 and 3
WQ may worsen may be because WS is not
adequate enough to support suitable sanitation
Group 3 has higher cost of illness, diarrhea rates and
smaller arm circumference .. maybe because of
decline in water quality
Group 1 (80 non JS):
>50% Taps and/or
Toilets
Group 2 (21 JS):
<50% taps or toilets.
Group 3 (10 JS):
>50% toilets but <50%
taps
Group 4 (31 JS):
>50% taps but <50%
Group 5 (17 JS):
>50% taps and toilets
Control (83 non JS):
<50% taps and toilets
31
Jalswarajya Impact
Supply Side Support (definitely more structured than non-JS)
Jalswarajya villages have 21% higher likelihood of a proper water
supply scheme being implemented
Contribution and collaboration
JS villages were 50% more likely to contribute to the WSS
scheme (implication for sustainability and O&M)
JS villages were 25% more likely to have an active VWSC
(implication for capacity building, ownership, accountability and
sustainability)
Positive outlook for sustainable approaches…
32
Jalswarajya Impact
Compare BPL and SC/ST in Jalswarajya villages with those in
non-JS villages
less coping costs (water collection, walking to defecation site)
(Rs 200 in rainy and ~Rs 2000 in dry)
WQ better in dry season but worse in rainy season
Higher reported use of taps in dry season (weak but significant
evidence)
Differences not significant in APL and open caste HHs
targeting of “poor” OR interplay between program and “poor”
is happening
33
Lessons Learnt: Policy Implication
Promote both taps and toilets in higher number to see significant health impacts (integration of programs, coverage etc)
LG and community capacity building (like in JS) for sustainability, across programs
Improved IEC/ awareness for behavior change (very crucial)
Continuous and timely water quality management approaches
M&E systems to include indicators linked with impacts/outcomes (actual use, water quality, etc)
Lessons For Future IE…1
34
Early and sustained dialogue with counterpart and stakeholders is very important
Need a strong counterpart throughout the IE exercise..if not, spend time in building their interest .. and knowledge..
Do not outsource training of enumerators to external agencies. Should be the primary responsibility of PI and Co-PI, intensify efforts by reviewing early results of training
Simplify the study design, survey instruments and keep the sample size manageable
Study design should allow for mid-course corrections
Lessons For Future IE…2
35
Baseline assessments should be done very early on for a true picture’
Logistics related to sampling and data collection should be manageable - in this case WQ sample collection, transportation has become highly resource and time intensive
There needs to be a balance between the research cycle and time required to provide results to counterparts to influence sector programs/ polices.