7/30/2019 How to design a survey for international aid
1/29
How to design a surveyfor international aid?
Dr Mark Griffin
Australian Development
Agency for Statistics andInformation Systems
www.adasis-oz.com
7/30/2019 How to design a survey for international aid
2/29
Statistics in international aid
In time gone by financial donorsmight have been happy to givemoney to Africa, with little
thought as to how the moneywas spent and little expectationof feedback.
Today there are strong expectations of a clear
knowledge of the current development context, aclearly identified strategy for achieving potentialchange, a clear method for evaluating the potentialsuccess of an intervention strategy, and careful
reports to supporting donors all of this comesdown to statistics.
7/30/2019 How to design a survey for international aid
3/29
Why is statistics needed?
To understand the extent ofproblems within developingnations
To identify possible causesof these problems
To evaluate how well international aid programs
work
To argue the worth of a cause with internationalgovernments and financial sponsors
7/30/2019 How to design a survey for international aid
4/29
ADASIS and meI am a survey methodologist,statistician, and computer engineer
After a long career at the University of QueenslandI established ADASIS in mid-2011
I still maintain strong research collaborations withstaff from the School of Population Health, University
of Queensland
I also do a wide range of consultancy work,primarily with clients working in international
development (such as World Vision)
7/30/2019 How to design a survey for international aid
5/29
World Vision human trafficking
When World Vision conducts anyof their projects they will typicallyconduct a baseline evaluation at
the start of the project and afollow-up evaluation at projectclose (typically 3 years later).
Since February 2013 I have been involved in thestudy design for a baseline survey for a humantrafficking project in Cambodia, China, Laos,Myanmar, Thailand Vietnam.
7/30/2019 How to design a survey for international aid
6/29
World Vision human trafficking
A major risk factor for a person toparticipate in human trafficking isa sense of over-optimism.
This survey will assess an expectedimprovement in knowledge andattitude towards human trafficking in local residents.
I am currently analyzing data from 10,000households from these countries.
7/30/2019 How to design a survey for international aid
7/29
World Vision gender-based violence
I have also been involvedin a baseline evaluation for aproject on gender-based
violence with World Vision inEurope.
Some of the indicators that we will be measuringthrough these surveys include the prevalence of
gender-based violence and the steps that individualstake to avoid gender-based violence.
This project will be conducted in Bosnia, Georgia,Jerusalem, and Lebanon.
7/30/2019 How to design a survey for international aid
8/29
Papua New Guinea
Condom distributionWith 1.28 percent of the adultpopulation estimated by UNAIDS
to be HIV-positive in 2006, PapuaNew Guinea has one of the mostserious HIV/AIDS epidemics in the Asia-Pacificsubregion.
Papua New Guinea accounts for 70 percent of thesubregion's HIV cases and is the fourth country(after Thailand, Cambodia, and Burma) to beclassified as having a generalized HIV epidemic.
7/30/2019 How to design a survey for international aid
9/29
Papua New Guinea
Condom distributionWithin this project I have beenworking with consultants working
with the PNG Aids Council.
I have been involved in the study design for the pilotstage of this project, and we are currently planning
the second stage.
7/30/2019 How to design a survey for international aid
10/29
Papua New Guinea
Condom distributionIn the pilot stage 100 store ownerswere asked about whether they
distributed condoms and theirattitude towards the distributionof condoms.
98 of the store owners indicated that condoms
were never or rarely available in their stores.
The most commonly cited reason for why storeowners do not distribute condoms is never thoughtabout it.
7/30/2019 How to design a survey for international aid
11/29
Survey Design
One approach used within theseevaluations is the use of surveys
Designing a good survey is a catch-22process.
You wont be able to develop an idealsurvey until after you have conductedthe survey.
7/30/2019 How to design a survey for international aid
12/29
Survey Design
In the mean time you need to makesome educated guesses based uponprior experience, data, and gut
feeling.
Within a survey we will learn bothabout the topic of the survey as well asabout how to conduct a survey of thattopic.
7/30/2019 How to design a survey for international aid
13/29
A good survey questionMisinterpretation of questions
Tourangeau (2000) identifies seven types
of misinterpretation:
Grammatical ambiguity
Excessive complexity
Faulty presupposition
The question makes an assumption about thestate of the world, and then asks theparticipant about this assumption
7/30/2019 How to design a survey for international aid
14/29
A good survey questionMisinterpretation of questions
Vague concepts
What is an alcoholic drink?
Vague quantifiers
How often is often ?
Unfamiliar terms
7/30/2019 How to design a survey for international aid
15/29
A good survey questionMisinterpretation of questions
False inferences
Participants make an assumption about whatthe question is trying to ask.
They then have two possible questionstheliteral interpretation, and the assumedinterpretation.
7/30/2019 How to design a survey for international aid
16/29
Survey sampling
How do we choose a group ofparticipants from a population whichwe think is representative of a larger
population?
Two key techniques
Stratification Clustering
7/30/2019 How to design a survey for international aid
17/29
Survey sampling - stratification
In the human trafficking projectperhaps people of a different socio-economic status are at a greater risk?
We ensure that there is a goodrepresentation of people at high andlow SES within our survey?
But how do we choose which variablesto stratify against?
7/30/2019 How to design a survey for international aid
18/29
Survey sampling - stratification
We need to think carefully about thesevariables.
A good choice of variables willimprove the accuracy of our results.
A poor choice of variables will reducethe accuracy of our results.
Only stratify against those variableswhere we can justify their use.
7/30/2019 How to design a survey for international aid
19/29
Survey sampling - clustering
Instead of collecting data from eachvillage within a population we cansave money by collecting more data
from a few carefully chosen set ofvillages.
How many villages and how many
households within a village?
We need to know the variation thatwe would expect between villages,
and the variation within a village.
7/30/2019 How to design a survey for international aid
20/29
Piloting the survey
There are a large number of things wemight learn from piloting the survey
We might have a better understanding of
our population, stratification andclustering.
Any problems with the logistics involved in
collecting the data or anymisunderstandings that the interview staff
had.
Also want pilot data prior to determining
the sample size in the follow-up survey.
7/30/2019 How to design a survey for international aid
21/29
Focus groups with interviewers
Focus groups with in-countryinterviewers after both the pilot surveyand the main household survey to
learn about any questions thatparticipants had problems with, or anyproblems with the logistics in
conducting the survey.
7/30/2019 How to design a survey for international aid
22/29
Comparison groups
Comparison groups (collect data onprograms where World Vision is notworking).
There are often reasons for why WorldVision is working in one village and notanotherthese effects can bereduced through the use of PropensityScore Matching.
7/30/2019 How to design a survey for international aid
23/29
Are all villages equal?
World Vision might spend moreresources in some areas compared toothers.
We might expect more communitychange within those areas wheremore effort was spent.
Can we include measures of resourcesspend within the data analysis?
7/30/2019 How to design a survey for international aid
24/29
Analysis Mixed Effects Models
Within an evaluation we typically datafrom a number of households within anumber of villages.
We might expect households from thesame village to be more similar thanhouseholds from different villages.
We need to account for this clusteringwithin our data analysiswecommonly used a mixed effects model
instead of using standard regression.
7/30/2019 How to design a survey for international aid
25/29
Analysis spatial maps
Publicly available data of unexplodedordnance in Laos.
Which is easier to interpret?
7/30/2019 How to design a survey for international aid
26/29
Missing data
Missing data can happen for a varietyof reasons
We cant contact selected people
People refuse to be in the survey
People might physically not be able to
provide the answers we want (memory
problems, literacy, etc)
Are those people that we dont havedata on different to those whose data
we have?
7/30/2019 How to design a survey for international aid
27/29
Missing data
We can have missing data on twolevels
Individual people (a person is completely
missing)
Individual questions (a person is partly
missing)
Analysis methods
Weighting
Mixed effects models
Multiple imputation
7/30/2019 How to design a survey for international aid
28/29
Conclusion
Statistics in international aid
Question design
Survey sampling Piloting the survey
Spatial mapping
Missing data
7/30/2019 How to design a survey for international aid
29/29
Conclusion
I have purposefully chosen within thisseminar to take a small superficial lookat a very large number of topics that
have come up in my work with WorldVision to date.
Each of those topics could easily be a
one hour seminar in their own right. I would be very happy to explore any
of those topics in a lot more depth with
anyone that is interested