IHSN Question Bank- supporting quality and consistency
in survey data
Olivier Dupriez World BankFrançois Fonteneau PARIS21/OECDGeoffrey Greenwell PARIS21/OECDMark McConaghy DFID - UK
International Household Survey Network
Evidence informed policy
“Trusted, high quality statistics are a necessary condition for developing and maintaining a strong
economy and society that functions well”
(Hon. Prof. Peter Anyang’ Nyong’o former Minister for Planning and Development, Kenya)
“Comparing groups, cultures, nations or continents is an essential means of
distinguishing between local conditions and universal regularities”
(Janet Harkness; Cross-Cultural Survey Methods; 2003:3)
Concepts
Latent constructs
Indicators
Questions
Questions and concepts
Improved/ unimproved drinking
water sources
Access to clean/safe
water
What is the main source of drinking water for your household?
Disease prevention
….but different building blocks
Joint Monitoring program
What is the main source of drinking water for members of your household?
11 Piped – to dwelling
12 Piped – to yard or plot
13 Piped – public tap/stand pipe
21 Tubewell/borehole
31 Dug well – protected
32 Dug well – unprotected
41 Water from spring – protected
42 Water from spring – unprotected
51 Rainwater collection
61 Tanker truck
71 Cart with small tank/drum
81 Surface water
91 Bottled water
96 Other (specify)
Typical surveyIn the rainy season what is the main source of
water for drinking and cooking for your household?
1 Private connection to pipeline
2 Private well
3 Public tap/standpipe
4 Public well
5 Neighbors
6 Water vendor
7 Spring
8 River, stream, lake, pond
9 Rainwater
10 Other (specify)
11 Bottled
Changes to questions
Joint Monitoring Program
What is the main source of drinking water for members of your household?
11 Piped – to dwelling
12 Piped – to yard or plot
13 Piped – public tap/stand pipe
21 Tubewell/borehole
31 Dug well – protected
32 Dug well – unprotected
41 Water from spring – protected
42 Water from spring – unprotected
51 Rainwater collection
61 Tanker truck
71 Cart with small tank/drum
81 Surface water
91 Bottled water
96 Other (specify)
Typical surveyIn the rainy season what is the main source of water for
drinking and cooking for your household?
1 Private connection to pipeline
2 Private well
3 Public tap/standpipe
4 Public well
5 Neighbors
6 Water vendor
7 Spring
8 River, stream, lake, pond
9 Rainwater
10 Other (specify)
11 Bottled
Deprivation estimates in same country
74
63
17 15
6 5
14
70
46
15
26
613
17
0
10
20
30
40
50
60
70
80
90
100
Shelte
r
Water
Sanita
tion
Info
rmat
ion
Educa
tion
Health
Food
per
cen
tag
e
DHS 1998 MICS 2000
Different ways of treating household and communal information
Different illnesses included- untreated diarrhoeal
illness verses ARI
?
All aged 5 or over
Has (name) ever attended school
<If YES>
What was the highest school level attended?
Within that school level, what was the highest grade
completed?
Can you read and write?
<If YES and age greater than 13>
What was the highest school level you attended?
Within that level, what was the highest grade completed?
Measuring educational attainment
Educational attainment outputs
37.6
26.5 27
9
51.8
14.8
32.3
1.10
10
20
30
40
50
60
70
80
90
100
No schooling Primary Secondary Above secondary
pe
rce
nt
DHS 2004 HIES 2000
Biased up and down by those who cannot read & write but did
attend some schooling?
Documentation for replication
“The only way to understand and evaluate an empirical analysis fully is to know the exact process by which the data were collected and the analysis produced… The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party could replicate the results without any additional information from the author”
(Gary King in Political Science and Politics Vol. XXVIII No 3 443-449)
Literal question –What kind of toilet facility do members of your household usually use?
Answer categories Flush/pour flushFlush to piped sewer system 11Flush to sceptic tank 12….
Ventilated Improved Pit latrine 21Pit latrine with slab 22Pit latrine without slab/open pit 23….
Post question text If ‘flush’ or ‘pour flush’ probe – Where does it flush to? IF NECESSARY ASK PERMISSION TO OBSERVE THE FACILITY
Example DDI elements
Notes for Interviewer Pit latrines are the best conventional sanitation option when there is no source of water on or close to the plot so that water use is low (typically less than 25 liters per person per day). They may also be used when water consumption is higher, provided that separate provision is made for sullage (“gray” water) disposal. However, they may be less attractive to users of pour-flush toilets in such situations. Simple unimproved pit latrines may smell. VIPs should be better but users may not appreciate the need for a dark interior and may modify the superstructure to provide more light, thus undermining the basic rationale behind the design. The VIP design is very dependent on a durable fly screen and may give rise to insect problems if the screen fails for any reason. High water table and/or flooding are obvious problems for pit latrines.
Universe All households
Source of information Responsible household member
DDI helps with interoperability
Inclusion of technical material
Ongoing and planned work, so far…
UNICEF/WHO immunization, anthropometry, nutrition etc
Joint Monitoring Program water & sanitation
FAO/ World Food Program Agricultural census and surveys
ILO economic activity, Int. labour migration
World Bank/ ILO consumption, savings & assets, remittances
World Bank/ ONS social capital
UIS/UNESCO educational surveys
IHFAN health facility surveys
• Using DDI to develop a meta data structure for data collection a priori, link concepts, indicators & questions
• Central repository for survey guidelines - custodians for topic areas (labour, health, food, education …) ‘a living resource’
• Generation of interviewing, editing and coding instructions
• Standards for coding and classification• Transparency with derived variables/ indicators creation• Tabulation plans for a primary analysis
Question Bank – key components
Question Bank – possible extensions
• Question Bank to include discussion forum• Facilitate production of questionnaires and data
input programs including CAI?• Provide the application to NSOs to help
establish Question Banks at country level• Introduce different language versions,
translation issues and cognitive testing/piloting• Develop information about relationship between
concepts and questions across countries and cultures
Technical aspects of the Question Bank
• XML schema has been developed to store all information (built on DDI and Dublin Core standards, plus additional elements)
• Q-Bank will be a user-friendly application that makes re-use of metadata easy (built on Flex/Air). It will be web-enabled and also work as a stand-alone application.
• user-friendly "metadata editor" developed to ensure easy maintenance of the system
• all applications are open source, and will be distributed free of charge
• XML schema and the application support multi-lingual metadata
• application can serve as a repository for international or national classifications
“Careful documentation of design decisions and implementations will enable cross-cultural researchers to begin accumulating a knowledge base of experience that will promote greater awareness of the relative advantages of each approach and move the community closer to establishing ‘best practice’ recommendations with confidence”
(Janet Harkness; Cross-Cultural Survey Methods; 2003: 34)
Question Bank important because
Thank you
www.surveynetwork.org
International Household Survey Network