The CSES Questionnaire Design ProcessDavid HowellUniversity of [email protected]
Jessica FortinGESIS Leibniz Institute for the Social [email protected]
European Survey Research Association (ESRA) Lausanne, SwitzerlandJuly, 2011
CSES QuestionnaireOver 60 countries participate
Common module of survey questions new theme every five years some questions the same some different according to new theme to be run as is
Administrative, demographic and voting variables to national standards
2Questionnaire Design Process
Questionnaire Design ProcessA lengthy development process
Over a three year period, 2008-2011
As transparent as possible
Collaborative, with broad input
Planning Committee FormationA Planning Committee, comprised of, selected by, and informed by collaborators, designs and oversees each Module.
Public call for nominationsNominating Committee recommends membersPlenary Session confirms list and suggests additional membersResult = 20 members from 15 countries
Theme DevelopmentEvery five years a new module is developed to specifically address a big question in science.
Multiple public calls for theme proposals Task Force collects proposalsTask Force presents and evaluates proposals for Plenary Session and Planning CommitteeTheme(s) tentatively decided upon before questionnaire design begins
Questionnaire DevelopmentAfter a theme is arrived at, attention turns to developing the questionnaire for it.
Subcommittees formed to pursue the new theme(s) and improvements to past questionsSubcommittees return to proposers for revisions to their original proposalsDraft questionnaire producedPlanning Committee discussion, revisionRepeat as necessary
PretestingThe draft questionnaire is pretested cross-nationally in different contexts.
Partial pretests in Brazil, Germany, UKFull pretests in Canada, IrelandReports received from pretest countries
Questionnaire FinalizationAfter having received extensive feedback, the questionnaire can now be finalized.
Subcommittees formed to address sections outside of the new theme(s)A Plenary Session provides final feedback Review of pretest results, subcommitteesPlanning Committee makes final changesFinal questionnaire disseminated
Design Challenges10-15 minutes is not much questionnaire time but, pressure to reduce collaborator burden competing with other content on surveys competing with other comparative studies
Existing, validated questions preferred
Priority to cross-national questions must be applicable in most all countries must work well across most all countries must be feasible in a variety of study designsInformation ChallengesA lot of people are involved Managing and tracking lots of input Reconciling many opinions and approaches Consensus not always possible
Making decisions with imperfect information More testing in more contexts would always be better, but relies on volunteers who...Have an election coming up (optimally)Have a funded survey with available spaceCan produce reports in a timely fashion Eventually decisions must be finalized
Collective ChallengesA lot of countries are involved Cross-national and cross-cultural applicability and equivalence Accommodating the variety of institutional arrangements in the many countries Difficult to predict or account for all situations, even with broad cross-national participation in the process
Where to go from here
Where to go from hereA formal post-process evaluation Weve done this a number of times now What worked well, what didnt Consider improvements Formalize and document process for next time
More extensive pretesting Allow more time for pretesting Test in more cultures and contexts Possible use of focus groupsWhere to go from hereMore focus on cross-national/cultural issues... concept equivalence... collaborator/interview instructions... translation issues
More methodological working groups
Learning from the approaches of other cross-national studies
Thank you for your time!
To learn more about CSES, or to download data:www.cses.org
...or email your questions to:[email protected]
Click here to load reader