Survey Research Operations Survey Research Center Institute for Social Research Design Issues in an...

transcript

Survey Research OperationsSurvey Research CenterInstitute for Social Research

Design Issues in an Academic Survey Organization

Beth-Ellen Pennell

Director, Survey Research Operations

Survey Research Center

Institute for Social Research

University of Michigan

SRO – SRC – ISR

Outline

• What’s different about an academic survey research center?

• Introduction to survey research operations• Project management 101• Communication• The triple constraints in project management• A note about quality• Communication• Can we make it more complex?

SRO – SRC – ISR

What’s different about an academic survey research

center?

SRO – SRC – ISR

Institute for Social Research

SRO – SRC – ISR

• Established in 1946 • Productivity: In the past ten years faculty and staff have:

produced 59 books, 232 book chapters, and 1116 scientific articles

produced datasets that have been downloaded by thousands of researchers worldwide resulting in a new scientific article every three days

submitted ~3 proposals per week achieved higher than average success rate on proposal funding

from NIH and NSF

SRO – SRC – ISR

• Research Volume: ISR approximately ~$100M + per year in sponsored

research; SRC about 70-80% of volume• Smallest of large-scale survey research

organizations but largest of academic-based • ~80% from US Federal government, predominately

National Institutes of Health, National Science Foundation

Investigator initiated grants (very few contracts)• Increased flexibility in design and execution

SRO – SRC – ISR

• Faculty and Staff 70 faculty, most with joint appointments

in academic departments 275 staff ~up to 2000 interviewers ~100 students

• Teaching mission of University

SRO – SRC – ISR

Educational Programs

• Summer Institute in Survey Research Techniques

• Michigan Program in Survey Methodology Masters PhD Certificate

• Joint Program in Survey Methods

• Joint Institute with Peking University

SRO – SRC – ISR

Introduction to Survey Research Operations

SRO – SRC – ISR

Survey Research Operations

• Complete about ~100 projects per year with staff of ~130

• Process ~100 proposals per year

• Historically, ~ half of SRO clients are outside of ISR

SRO – SRC – ISR

Types of Projects

• Interviewer-administered CAPI CATI (both centralized and decentralized) CASI and A-CASI as components of CAPI

instruments

• Self-administered Web Mail

• Many projects use a combination of modes, attempting to optimize quality and cost

SRO – SRC – ISR

Types of Projects

• Our volume is comprised of: large, national CAPI studies regional CAPI studies smaller telephone, mail or web-based studies

• Our organizational infrastructure accommodates large projects that can be modified for smaller or more specialized efforts

SRO – SRC – ISR

Project Management 101

SRO – SRC – ISR

Project Manager/Team Roles

• The project manager is the person responsible for accomplishing the project objectives

• The project team is comprised of the people who have assigned roles and responsibilities for completing aspects of the project

SRO – SRC – ISR

COMMUNICATION

SRO – SRC – ISR

Communication

“. . . 75 to 90 percent of the project manager's time on a project is spent

communicating with one or more stakeholders” (Corpedia)

SRO – SRC – ISR

Project Management

Managing a project includes:

• Identifying requirements

• Establishing clear and achievable objectives

• Adapting/balancing approach with various stakeholders

• Managing the team

• Balancing the competing demands (scope (includes quality), time & cost)

SRO – SRC – ISR

Project Management

• Application of knowledge, skills, tools and techniques to project activities with following phases: Initiation Planning Risk assessment/ Plan B

• Example Executing and controlling Closing

SRO – SRC – ISR

Stakeholders

• Individuals and organizations actively involved in the project, or whose interest may be affected by the project

• Project management team’s job is to: Identify stakeholders Determine requirements and expectations Manage stakeholders’ influence

SRO – SRC – ISR

Key Stakeholders

• Sponsor

• Client (if different from sponsor)

• Users (if different from above)

• Project management organization

• Performing organization

• Project manager and team members

• Respondents

Example

SRO – SRC – ISR

Stakeholder Influence

SRO – SRC – ISR

Management Plan

• Purpose: Documents actions necessary to define, prepare,

integrate, and coordinate project activities May include several subsidiary plans

• Progressively elaborated

• Multiple authors

• Version control important (hard to do after the fact)

SRO – SRC – ISR

Management Plan Broad Outline

• Background

• Initiation activities

• Project planning/risk assessment

• Implementation and control

• Closing

SRO – SRC – ISR

Launching a Project

• Project design also involves: Creating a team Developing a team Managing a team

SRO – SRC – ISR

Team Management

• Complexities of managing resources:

Delineate roles & responsibilities

Acquire staff with needed skills

Develop and build the team

Manage individual performance

Task Team Model

SRO – SRC – ISR

The Triple Constraints

SRO – SRC – ISR

Constraint #1: Scope (read: Design)

• Scope planning

• Scope definition

• Work breakdown structure

• Scope control

SRO – SRC – ISR

Design Specifications (1)• Study Purpose

Objectives Funding source(s) Timing Production plan milestones/deliverables Overall design summary

• Project Coordination Plan Lead team Task teams Working groups Communication/coordination

SRO – SRC – ISR

Design Specifications (2)

• Sample Design Type (sample frame availability and coverage

properties) Size Accessibility of population Goals (response rate, number of cases in

various domains, etc.) Replicate design Responsive design

SRO – SRC – ISR

• Mode Feasibility of mode

• Infrastructure• Social desirability bias (what is sensitive)• Literacy• Interviewer safety

Mixed/multi-mode – increasing in use• Sequential – cheapest first• Driven by coverage/cost

SRO – SRC – ISR

• Data Collection Ethical consideration/review

• Confidentiality/privacy• Informed consent• Sensitive topics or vulnerable populations• Age of majority• Literacy• Incentives/coercion• Local laws, regulations• Special cases: biomarkers

SRO – SRC – ISR

Design Specifications (5)• Data Collection (continued)

Field Staffing Plan• Assess local infrastructure• Languages • Sample assignment model• Supervisory structure/reporting• Production schedule • Interviewer remuneration guidelines• Mechanisms for reporting progress• Result code definitions and response rate calculation

guidelines

SRO – SRC – ISR

• Instrument /Application Development Mode Languages Special printing requirements Hardware and software requirements Local adaptation requirements

SRO – SRC – ISR

Testing• Pretests• Applications/systems testing• Sample requirements

Training Plans• Supervisor/Team Leader training• Interviewer training• Special training, e.g., biomarkers

SRO – SRC – ISR

• Quality Checks Application requirements Training Paradata: dashboards, control charts Verifications/validations Editing and coding Data review Site visits

SRO – SRC – ISR

Design Specifications (9)• Data Processing Plans

Coding Data entry Staffing Production schedule Mechanism for reporting

• Sample Post-processing Weights, nonresponse or other adjustments,

imputation, variance estimation

SRO – SRC – ISR

• Data Processing Quality Control Testing Training plans

• Supervisor training• Editor guide• DDE training• Coding/Special function training

SRO – SRC – ISR

• Materials Development (Deliverables) Datasets

• Sampling variables (weighting, nonresponse adjustments) • Sample file (result codes to check response rate calculations)• Call records• Data/Application

Respondent materials• Coversheets• Questionnaires/ applications• Questionnaire supplements/aids (e.g., respondent booklets,

event history calendars)• Contact materials

SRO – SRC – ISR

• Materials Development (Deliverables) - continued Training

• Materials (e.g., PowerPoint slides, exercises)• Manuals (supervisor/interviewer, other)• Agendas• Location/organization

Protocols• Incident/problem reporting• Technical support/back-up

SRO – SRC – ISR

Design Specifications (13)• Materials Development (Deliverables) - continued

Codebook Reports/Quality Profile

• Sample design description• Ethics review verification• Translation reports• Pretest reports• Field progress reports• Quality control audits• Field procedures reports

SRO – SRC – ISR

Constraint #2: Time

• Project Time Management

Activity Definition

Activity Sequencing

Activity Resource Estimation

Schedule Development

Schedule Control

SRO – SRC – ISR

Schedule Development and Control

• Identify schedule activities

• Establish optimum project work flow

• Determine resources needed

• Determine time needed for each activity

• Lay out activities in calendar time

• Keep track of progress, take corrective action

SRO – SRC – ISR

TASK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Kickoff Meeting *Develop Questionnaire * * * * * * * *Programming Questionnaire * * * * *Translation of Questionnaire * * *1" Round Instrument Testing & Modifications * * * * * *Add Translation to Programming * *Sample Preparation & Listing * * * * * * * * *Native Speaker Review & Testing of Programmed, Translated Modules * *Instrument Translation Modifications * * *Instrument Programming Modifications * * *Recruit/Hiring of Production Interviewers * * * *Integrate & Test Full Instrument * *Develop & Test Sample Management * *Finalize Pretest Training/Project Materials *Duplicate Training Materials/Prepare Laptops *Pretest I Interviewer Training & Practice *Pretest I *Debrief Pretest I *Instrument & Materials Modifications, Testing *Pretest II Interviewer Training & Practice *Pretest II *Debrief Pretest II *Instrument & Materials Final Modifications *Duplicate Training Materials/Prepare Laptops *Draw Sample *Sample Assignments * *Train the Trainer *Interviewer Training I *Production Interviewing * * * * * * * * * * * *Coding * * * * * * * * * * * * *Interviewer Training II *Data Preparation * * * * * * * *Data Delivery *Documentation Preparation * * * * * * * * * * * * * * * * * * * * * * *Documentation Delivery *

Month of Survey

Project Work Plan Example

SRO – SRC – ISR

Constraint #3: Cost

• Project Cost Management Primarily concerned with the cost of

resources needed to complete schedule activities

SRO – SRC – ISR

Cost Monitoring

• Cost management processes include Cost estimating Cost budgeting Cost control

SRO – SRC – ISR

Design Considerations in Tenders, Bids, and Contracts

• Purpose: to thoroughly document and communicate roles, responsibilities, design details, expectations and outcomes in order to adequately budget the survey effort

• Organizational Considerations Locus of control Balancing standardization and localization Balancing funding versus contractual obligations versus expertise Methodological, substantive, local expertise Legal expertise ‘One Voice’/Point of contact

SRO – SRC – ISR

Budgeting and Bid Assessment

• Benchmark, where possible

• Seek multiple bids, if feasible

• Ask for project experience and references

• Ask for sample materials

• Ask for staff and organizational capacity

• Ask for facilities and equipment capacity

• Allow time for clarification and several iterations of bids

SRO – SRC – ISR

Budgeting and Bid Assessment

• Potential Problems Sample frame access Difficulty with concept of probability sampling Inadequate quality control procedures Not familiar/sensitive to/barrier to nonresponse reduction

techniques (number of attempts to contact households, refusal conversion techniques, inconsistent contact result coding schemes)

Inadequate interviewer training/interviewer incentives Inadequate facilities (hardware, software, Web access,

etc.) Inadequate budget detail Research ‘traditions’

SRO – SRC – ISR

Budget TemplateNumber Number of Months Cost per Unit Total Cost

Base Salaries Project Manager Data Manager Fieldwork Manager Accountant Assistants Supervisors Interviewers Data entry operators Drivers Translators Computer programmers Overtime Incentive payments

Travel Allowances Project Manager Data Manager Fieldwork Manager Assistants Supervisors Interviewers Drivers

SRO – SRC – ISR

Number Number of Months Cost per Unit Total CostMaterials Vehicle purchase Fuel and maintenance Computers Printers, etc. Publicity Computer/printer supplies Photocopier/Fax machine Office supplies Communications (phone, fax, postage, etc) Equipment maintenance Printing Costs Questionnaires Training Manuals Reports Miscellaneous (maps, listings, manuals, etc)

Consultant Costs International consultants International per diem International travel Local consultants Local per diem Local travel

Contingency (10%)

TOTAL COST

Budget Template (continued)

SRO – SRC – ISR

Contracting

• Specific to: Funding source Contracting organizations Employee contracts and remuneration policies Countries Type of contract

• Must have local legal advice• Commercial versus academic versus government

pros / cons

SRO – SRC – ISR

One Mechanism to Control Scope and Costs: Responsive Design

• Interviewer labor dominates costs on most surveys

• Surveys inherently uncertain (e.g. estimated costs and completed interviews)

• Maximize cost benefit by adapting design features based on incoming survey process data (make data driven decisions)

SRO – SRC – ISR

Responsive Design Implementation Process (Groves et al., 2009)

1) Pre-identify features potentially affecting costs and errors (risks)

2) Identify indicators

3) Wherever possible, devise benchmarks for each indicator

4) Create decision rules that alter features of the design if benchmarks are not met

5) Develop effective visual displays of indicators

6) When benchmarks are not met, alter the survey design based on the decision rules

SRO – SRC – ISR

Creating decision rules: Example

• Risk: Interviewers unable to convince eligible respondents to participate, potentially increasing non response bias

• Indicator: Cumulative average interview cooperation rate by day by interviewer by area

• Benchmark: Similar study had a 71% interview cooperation rate after 3 weeks of data collection

• Decision rule: If not at a 70% coop rate by the end of the 3rd week, identify source of problem: is it systematic across all interviewers and sample types or limited to a few interviewers or sample types, area

SRO – SRC – ISR

A note about quality

SRO – SRC – ISR

Quality Management

• Components Quality Planning and Assurance Quality Control Quality Profile

SRO – SRC – ISR

COMMUNICATION

SRO – SRC – ISR

Communication

• Communications Planning Information Distribution Performance Reporting Manage Stakeholders

SRO – SRC – ISR

Communication

• Frequent

• Detailed

• Mixed (face-to-face, e-mail, conference call, newsletters, Webcasts)

• Document Sharing Critical for timely review Version control

SRO – SRC – ISR

Data Documentation eRoom

SRO – SRC – ISR

Can we make this more complex?

SRO – SRC – ISR

We could try to do this across multiple countries…

SRO – SRC – ISR

Mode of Administration

SRO – SRC – ISR

Cross-national Considerations (1)

• Structure dynamics Leadership Levels Number of organizations Locus of control Level of control Experience at various levels Funding

SRO – SRC – ISR

• Degree of standardization: Often determined by level, flow, and source of

funding Experience and research infrastructure at central

and local levels Control of data Decisions need to be made at every stage of the

survey lifecycle

SRO – SRC – ISR

• Laws/ethical standards• Lingua Franca• Time zones• Terminology• Research traditions and infrastructure• Technology (Web: e-mail, VOIP, web

casts, document sharing)

SRO – SRC – ISR

Danke!