Date post: | 01-Jan-2016 |
Category: |
Documents |
Upload: | robyn-white |
View: | 214 times |
Download: | 0 times |
Statistics Networking Day
Question design
Jacki Schirmer
Senior Research FellowHealth Research Institute & Institute for Applied Ecology
Designing survey questionsWhether you are using existing measures, adapting existing measures, or designing new
measures, you need to ensure your questions produce valid, reliable, accurate data
• Questions do not ‘lead’ people to a specific answer
• Result is close to real life, generates statistically useful data
• Produces similar results under consistent conditions
• Measures what it is intended to measure
Valid Reliable
UnbiasedAccurate
Ideal survey question design process1. Conceptualise
a. Clearly define end uses of datab. Design topics that will achieve objectives
2. ‘Straw man’ drafta. Draft initial measures – but expect them to change
3. Qualitative testinga. Focus group with survey usersb. They attempt the survey and then discuss
What was confusing, hard, offensive, unclear Where does wording need to change
c. You revise the questions
4. Expert reviewa. Get a statistician or survey expert to review questions
Advise on response options, direction of question
5. Pilot testa. Sample of participants complete survey under exact conditions in which it will be deliveredb. Analyse results from pilot sample to identify problem areasc. Revise
6. Deliver your survey
Key considerations (common pitfalls)
• What language/terminology does your target survey population use?• What response option formats are best?• Literacy and numeracy level of target survey population• Eyesight, deafness prevalence
Statistics Networking Day
Reweighting
Jacki Schirmer
Senior Research FellowHealth Research Institute & Institute for Applied Ecology
Why reweight?
• Your sampling technique was not based on a specific sample frame– Online surveys
promoted via social media
– Self-select surveys
• Your sample is not representative of the population you sampled– Due to deliberate over-
sampling of some groups– Due to bias in who
responded to your survey
Responses by region – 2014 Regional Wellbeing Survey
State Total no. survey participants
NSW 3,403
VIC 5,289
QLD 728
SA 1224
WA 791
TAS 564
How should you analyse representativeness?
• Common approach: analyse demographics– Gender– Age– Education– Occupation
• Assume that if you have similar distribution to population, survey is OK
• But bias is often related to salience, not demographics– Are you biased to people
with a greater interest in your topic (who may have different views to those with less interest)
– Is there a general dataset you can compare to which asks questions that let you assess this?
Comparison of survey response to population benchmarks
• All surveys have biased samples
• Ours is no exception • In 2014, we deliberately
oversampled farmers, Victorians
• We also have a bias to older, female respondents (common across many surveys)
• Our large sample enables us to weight data so we can make statements about the population as a whole
Characteristic Rural and regional Australia, 20111
Regional Wellbeing Survey, 2014
State NSW & ACTVICQLDSAWA & NTTAS
35.7%17.4%23.5%8.1%11.3%4.1%
28.4%43.7%6.0%10.1%7.0%4.7%
Gender FemaleMale
50.7%49.3%
57.9%42.1%
Age 18-3940-5455-6465+
33.2%28.0%17.3%21.5%
13.7%29.8%28.6%27.9%
Working as a farmer FarmerNon-farmer
4.5%95.5%
30.6%69.4%
1Data source: Australian Bureau of Statistics Census of Population and Housing 2011. Data accessed via TableBuilderPro. Data were calculated for rural and regional Australia and exclude people living in cities with >100,000 population.
ABS GregWT procedure used to weight data, using Census data as benchmarks. This ensures we use national best practice in analysis to correct for biases in our sample
If it’s not representative, how do you weight?
• Simple multiplicative approach– I have 15% young people in my survey compared to 30% in the
population. Every young person will be assigned a weight of ‘2’ to ensure they are representative
– OK if weighting by simple demographics and not many of them– Problematic when you weight by multiple attributes, as requires all cells
to be populated, and simple approach does not represent interactions well
• Regression and other statistical modelling approaches– Use regression modelling to assign weighting to each respondent– E.g. GREGWT used by ABS– See Tanton et al. (2014) discussion http://
microsimulation.org/IJM/V7_1/4_IJM_7_1_Tanton_Williamson_Harding.pdf
Challenges• Cell sizes – when they become small, there is
greater potential to ‘weight the bias’ and get an extreme result
• Number of benchmark variables used
Scale (population) weights vs proportional weights
• Scale weights scale to population size• Proportional weights preserve your n• Following examples are from
file://stafffiles.win.canberra.edu.au/Homes$/s427944/Windows%20Profile/Desktop/Creating%20Weights%20to%20Improve%20Estimates.pdf
Problems with weighting
• Problematic to calculate standard error, confidence intervals for weighted data
• Other statistical tests may be non-robust• Some programs assign population n, rather than sample n,
to weighted data – with resulting problems with statistical analysis (SPSS in particular)
• Weighting does not correct all types of survey error• Weighting is only as good as your benchmarks – pick the
wrong benchmarks and it has limited utility • Non-response effects – weighting does not correct for
these