How to design a survey for international aid

7/30/2019 How to design a survey for international aid

1/29

How to design a surveyfor international aid?

Dr Mark Griffin

Australian Development

Agency for Statistics andInformation Systems

www.adasis-oz.com


2/29

Statistics in international aid

In time gone by financial donorsmight have been happy to givemoney to Africa, with little

thought as to how the moneywas spent and little expectationof feedback.

Today there are strong expectations of a clear

knowledge of the current development context, aclearly identified strategy for achieving potentialchange, a clear method for evaluating the potentialsuccess of an intervention strategy, and careful

reports to supporting donors all of this comesdown to statistics.


3/29

Why is statistics needed?

To understand the extent ofproblems within developingnations

To identify possible causesof these problems

To evaluate how well international aid programs

work

To argue the worth of a cause with internationalgovernments and financial sponsors


4/29

ADASIS and meI am a survey methodologist,statistician, and computer engineer

After a long career at the University of QueenslandI established ADASIS in mid-2011

I still maintain strong research collaborations withstaff from the School of Population Health, University

of Queensland

I also do a wide range of consultancy work,primarily with clients working in international

development (such as World Vision)


5/29

World Vision human trafficking

When World Vision conducts anyof their projects they will typicallyconduct a baseline evaluation at

the start of the project and afollow-up evaluation at projectclose (typically 3 years later).

Since February 2013 I have been involved in thestudy design for a baseline survey for a humantrafficking project in Cambodia, China, Laos,Myanmar, Thailand Vietnam.


6/29

World Vision human trafficking

A major risk factor for a person toparticipate in human trafficking isa sense of over-optimism.

This survey will assess an expectedimprovement in knowledge andattitude towards human trafficking in local residents.

I am currently analyzing data from 10,000households from these countries.


7/29

World Vision gender-based violence

I have also been involvedin a baseline evaluation for aproject on gender-based

violence with World Vision inEurope.

Some of the indicators that we will be measuringthrough these surveys include the prevalence of

gender-based violence and the steps that individualstake to avoid gender-based violence.

This project will be conducted in Bosnia, Georgia,Jerusalem, and Lebanon.


8/29

Papua New Guinea

Condom distributionWith 1.28 percent of the adultpopulation estimated by UNAIDS

to be HIV-positive in 2006, PapuaNew Guinea has one of the mostserious HIV/AIDS epidemics in the Asia-Pacificsubregion.

Papua New Guinea accounts for 70 percent of thesubregion's HIV cases and is the fourth country(after Thailand, Cambodia, and Burma) to beclassified as having a generalized HIV epidemic.


9/29

Papua New Guinea

Condom distributionWithin this project I have beenworking with consultants working

with the PNG Aids Council.

I have been involved in the study design for the pilotstage of this project, and we are currently planning

the second stage.


10/29

Papua New Guinea

Condom distributionIn the pilot stage 100 store ownerswere asked about whether they

distributed condoms and theirattitude towards the distributionof condoms.

98 of the store owners indicated that condoms

were never or rarely available in their stores.

The most commonly cited reason for why storeowners do not distribute condoms is never thoughtabout it.


11/29

Survey Design

One approach used within theseevaluations is the use of surveys

Designing a good survey is a catch-22process.

You wont be able to develop an idealsurvey until after you have conductedthe survey.


12/29

Survey Design

In the mean time you need to makesome educated guesses based uponprior experience, data, and gut

feeling.

Within a survey we will learn bothabout the topic of the survey as well asabout how to conduct a survey of thattopic.


13/29

A good survey questionMisinterpretation of questions

Tourangeau (2000) identifies seven types

of misinterpretation:

Grammatical ambiguity

Excessive complexity

Faulty presupposition

The question makes an assumption about thestate of the world, and then asks theparticipant about this assumption


14/29


Vague concepts

What is an alcoholic drink?

Vague quantifiers

How often is often ?

Unfamiliar terms


15/29


False inferences

Participants make an assumption about whatthe question is trying to ask.

They then have two possible questionstheliteral interpretation, and the assumedinterpretation.


16/29

Survey sampling

How do we choose a group ofparticipants from a population whichwe think is representative of a larger

population?

Two key techniques

Stratification Clustering


17/29

Survey sampling - stratification

In the human trafficking projectperhaps people of a different socio-economic status are at a greater risk?

We ensure that there is a goodrepresentation of people at high andlow SES within our survey?

But how do we choose which variablesto stratify against?


18/29

Survey sampling - stratification

We need to think carefully about thesevariables.

A good choice of variables willimprove the accuracy of our results.

A poor choice of variables will reducethe accuracy of our results.

Only stratify against those variableswhere we can justify their use.


19/29

Survey sampling - clustering

Instead of collecting data from eachvillage within a population we cansave money by collecting more data

from a few carefully chosen set ofvillages.

How many villages and how many

households within a village?

We need to know the variation thatwe would expect between villages,

and the variation within a village.


20/29

Piloting the survey

There are a large number of things wemight learn from piloting the survey

We might have a better understanding of

our population, stratification andclustering.

Any problems with the logistics involved in

collecting the data or anymisunderstandings that the interview staff

had.

Also want pilot data prior to determining

the sample size in the follow-up survey.


21/29

Focus groups with interviewers

Focus groups with in-countryinterviewers after both the pilot surveyand the main household survey to

learn about any questions thatparticipants had problems with, or anyproblems with the logistics in

conducting the survey.


22/29

Comparison groups

Comparison groups (collect data onprograms where World Vision is notworking).

There are often reasons for why WorldVision is working in one village and notanotherthese effects can bereduced through the use of PropensityScore Matching.


23/29

Are all villages equal?

World Vision might spend moreresources in some areas compared toothers.

We might expect more communitychange within those areas wheremore effort was spent.

Can we include measures of resourcesspend within the data analysis?


24/29

Analysis Mixed Effects Models

Within an evaluation we typically datafrom a number of households within anumber of villages.

We might expect households from thesame village to be more similar thanhouseholds from different villages.

We need to account for this clusteringwithin our data analysiswecommonly used a mixed effects model

instead of using standard regression.


25/29

Analysis spatial maps

Publicly available data of unexplodedordnance in Laos.

Which is easier to interpret?


26/29

Missing data

Missing data can happen for a varietyof reasons

We cant contact selected people

People refuse to be in the survey

People might physically not be able to

provide the answers we want (memory

problems, literacy, etc)

Are those people that we dont havedata on different to those whose data

we have?


27/29

Missing data

We can have missing data on twolevels

Individual people (a person is completely

missing)

Individual questions (a person is partly

missing)

Analysis methods

Weighting

Mixed effects models

Multiple imputation


28/29

Conclusion

Statistics in international aid

Question design

Survey sampling Piloting the survey

Spatial mapping

Missing data


29/29

Conclusion

I have purposefully chosen within thisseminar to take a small superficial lookat a very large number of topics that

have come up in my work with WorldVision to date.

Each of those topics could easily be a

one hour seminar in their own right. I would be very happy to explore any

of those topics in a lot more depth with

anyone that is interested

Date post:	14-Apr-2018
Category:	Documents
Upload:	studioq
View:	212 times
Download:	0 times

How to design a survey for international aid

Documents