The CSES Questionnaire Design Process
David HowellUniversity of [email protected]
Jessica FortinGESIS – Leibniz Institute for the Social Sciences
European Survey Research Association (ESRA) Lausanne, Switzerland
July, 2011
CSES Questionnaire
• Over 60 countries participate• Common module of survey questions
— new theme every five years— some questions the same— some different according to new theme— to be run “as is”
• Administrative, demographic and voting variables to national standards
Questionnaire Design Process
Questionnaire Design Process
• A lengthy development process— Over a three year period, 2008-2011
• Iterative• As transparent as possible• Collaborative, with broad input
Planning Committee Formation
A Planning Committee, comprised of, selected by, and informed by collaborators, designs and oversees each Module.
• Public call for nominations• Nominating Committee recommends
members• Plenary Session confirms list and
suggests additional members• Result = 20 members from 15 countries
Theme Development
Every five years a new module is developed to specifically address a big question in science.
• Multiple public calls for theme proposals • Task Force collects proposals• Task Force presents and evaluates
proposals for Plenary Session and Planning Committee
• Theme(s) tentatively decided upon before questionnaire design begins
Questionnaire Development
After a theme is arrived at, attention turns to developing the questionnaire for it.
• Subcommittees formed to pursue the new theme(s) and improvements to past questions
• Subcommittees return to proposers for revisions to their original proposals
• Draft questionnaire produced• Planning Committee discussion, revision• Repeat as necessary
Pretesting
The draft questionnaire is pretested cross-nationally in different contexts.
• Partial pretests in Brazil, Germany, UK• Full pretests in Canada, Ireland• Reports received from pretest countries
Questionnaire Finalization
After having received extensive feedback, the questionnaire can now be finalized.
• Subcommittees formed to address sections outside of the new theme(s)
• A Plenary Session provides final feedback— Review of pretest results,
subcommittees• Planning Committee makes final changes• Final questionnaire disseminated
Challenges
Design Challenges• 10-15 minutes is not much questionnaire time
… but, pressure to reduce collaborator burden— competing with other content on surveys— competing with other comparative studies
• Existing, validated questions preferred• Priority to cross-national questions
— must be applicable in most all countries— must work well across most all countries— must be feasible in a variety of study
designs
Information Challenges• A lot of people are involved
— Managing and tracking lots of input— Reconciling many opinions and approaches— Consensus not always possible
• Making decisions with imperfect information— More testing in more contexts would always
be better, but relies on volunteers who...• Have an election coming up (optimally)• Have a funded survey with available space• Can produce reports in a timely fashion
— Eventually decisions must be finalized
Collective Challenges• A lot of countries are involved
— Cross-national and cross-cultural applicability and equivalence
— Accommodating the variety of institutional arrangements in the many countries
— Difficult to predict or account for all situations, even with broad cross-national participation in the process
Where to go from here
Where to go from here• A formal post-process evaluation
— We’ve done this a number of times now— What worked well, what didn’t— Consider improvements— Formalize and document process for next time
Some possibilities:• More extensive pretesting
— Allow more time for pretesting— Test in more cultures and contexts— Possible use of focus groups
Where to go from here• More focus on cross-national/cultural issues
... concept equivalence
... collaborator/interview instructions
... translation issues• More methodological working groups• Learning from the approaches of other cross-
national studies
Thank you for your time!
To learn more about CSES, or to download data:www.cses.org
...or email your questions to:[email protected]
