Date post: | 28-Mar-2015 |
Category: |
Documents |
Upload: | sarah-ingram |
View: | 216 times |
Download: | 2 times |
Survey Research OperationsSurvey Research CenterInstitute for Social Research
Design Issues in an Academic Survey Organization
Beth-Ellen Pennell
Director, Survey Research Operations
Survey Research Center
Institute for Social Research
University of Michigan
SRO – SRC – ISR
Outline
• What’s different about an academic survey research center?
• Introduction to survey research operations• Project management 101• Communication• The triple constraints in project management• A note about quality• Communication• Can we make it more complex?
2
SRO – SRC – ISR
What’s different about an academic survey research
center?
3
SRO – SRC – ISR
Institute for Social Research
4
SRO – SRC – ISR
Survey Research Center
• Established in 1946 • Productivity: In the past ten years faculty and staff have:
produced 59 books, 232 book chapters, and 1116 scientific articles
produced datasets that have been downloaded by thousands of researchers worldwide resulting in a new scientific article every three days
submitted ~3 proposals per week achieved higher than average success rate on proposal funding
from NIH and NSF
•
5
SRO – SRC – ISR
Survey Research Center
• Research Volume: ISR approximately ~$100M + per year in sponsored
research; SRC about 70-80% of volume• Smallest of large-scale survey research
organizations but largest of academic-based • ~80% from US Federal government, predominately
National Institutes of Health, National Science Foundation
Investigator initiated grants (very few contracts)• Increased flexibility in design and execution
6
SRO – SRC – ISR
Survey Research Center
• Faculty and Staff 70 faculty, most with joint appointments
in academic departments 275 staff ~up to 2000 interviewers ~100 students
• Teaching mission of University
7
SRO – SRC – ISR
Educational Programs
• Summer Institute in Survey Research Techniques
• Michigan Program in Survey Methodology Masters PhD Certificate
• Joint Program in Survey Methods
• Joint Institute with Peking University
8
SRO – SRC – ISR
Introduction to Survey Research Operations
9
SRO – SRC – ISR
Survey Research Operations
• Complete about ~100 projects per year with staff of ~130
• Process ~100 proposals per year
• Historically, ~ half of SRO clients are outside of ISR
10
SRO – SRC – ISR
Types of Projects
• Interviewer-administered CAPI CATI (both centralized and decentralized) CASI and A-CASI as components of CAPI
instruments
• Self-administered Web Mail
• Many projects use a combination of modes, attempting to optimize quality and cost
11
SRO – SRC – ISR
Types of Projects
• Our volume is comprised of: large, national CAPI studies regional CAPI studies smaller telephone, mail or web-based studies
• Our organizational infrastructure accommodates large projects that can be modified for smaller or more specialized efforts
12
SRO – SRC – ISR
Project Management 101
13
SRO – SRC – ISR
Project Manager/Team Roles
• The project manager is the person responsible for accomplishing the project objectives
• The project team is comprised of the people who have assigned roles and responsibilities for completing aspects of the project
14
SRO – SRC – ISR
COMMUNICATION
15
SRO – SRC – ISR
Communication
“. . . 75 to 90 percent of the project manager's time on a project is spent
communicating with one or more stakeholders” (Corpedia)
16
SRO – SRC – ISR
Project Management
Managing a project includes:
• Identifying requirements
• Establishing clear and achievable objectives
• Adapting/balancing approach with various stakeholders
• Managing the team
• Balancing the competing demands (scope (includes quality), time & cost)
17
SRO – SRC – ISR
Project Management
• Application of knowledge, skills, tools and techniques to project activities with following phases: Initiation Planning Risk assessment/ Plan B
• Example Executing and controlling Closing
18
SRO – SRC – ISR
Stakeholders
• Individuals and organizations actively involved in the project, or whose interest may be affected by the project
• Project management team’s job is to: Identify stakeholders Determine requirements and expectations Manage stakeholders’ influence
19
SRO – SRC – ISR
Key Stakeholders
• Sponsor
• Client (if different from sponsor)
• Users (if different from above)
• Project management organization
• Performing organization
• Project manager and team members
• Respondents
Example
20
SRO – SRC – ISR
Stakeholder Influence
21
SRO – SRC – ISR
Management Plan
• Purpose: Documents actions necessary to define, prepare,
integrate, and coordinate project activities May include several subsidiary plans
• Progressively elaborated
• Multiple authors
• Version control important (hard to do after the fact)
22
SRO – SRC – ISR
Management Plan Broad Outline
• Background
• Initiation activities
• Project planning/risk assessment
• Implementation and control
• Closing
23
SRO – SRC – ISR
Launching a Project
• Project design also involves: Creating a team Developing a team Managing a team
24
SRO – SRC – ISR
Team Management
• Complexities of managing resources:
Delineate roles & responsibilities
Acquire staff with needed skills
Develop and build the team
Manage individual performance
25
Task Team Model
26
SRO – SRC – ISR
The Triple Constraints
27
SRO – SRC – ISR
Constraint #1: Scope (read: Design)
• Scope planning
• Scope definition
• Work breakdown structure
• Scope control
28
SRO – SRC – ISR
Design Specifications (1)• Study Purpose
Objectives Funding source(s) Timing Production plan milestones/deliverables Overall design summary
• Project Coordination Plan Lead team Task teams Working groups Communication/coordination
29
SRO – SRC – ISR
Design Specifications (2)
• Sample Design Type (sample frame availability and coverage
properties) Size Accessibility of population Goals (response rate, number of cases in
various domains, etc.) Replicate design Responsive design
30
SRO – SRC – ISR
Design Specifications (3)
• Mode Feasibility of mode
• Infrastructure• Social desirability bias (what is sensitive)• Literacy• Interviewer safety
Mixed/multi-mode – increasing in use• Sequential – cheapest first• Driven by coverage/cost
31
SRO – SRC – ISR
Design Specifications (4)
• Data Collection Ethical consideration/review
• Confidentiality/privacy• Informed consent• Sensitive topics or vulnerable populations• Age of majority• Literacy• Incentives/coercion• Local laws, regulations• Special cases: biomarkers
32
SRO – SRC – ISR
Design Specifications (5)• Data Collection (continued)
Field Staffing Plan• Assess local infrastructure• Languages • Sample assignment model• Supervisory structure/reporting• Production schedule • Interviewer remuneration guidelines• Mechanisms for reporting progress• Result code definitions and response rate calculation
guidelines
33
SRO – SRC – ISR
Design Specifications (6)
• Instrument /Application Development Mode Languages Special printing requirements Hardware and software requirements Local adaptation requirements
34
SRO – SRC – ISR
Design Specifications (7)
Testing• Pretests• Applications/systems testing• Sample requirements
Training Plans• Supervisor/Team Leader training• Interviewer training• Special training, e.g., biomarkers
35
SRO – SRC – ISR
Design Specifications (8)
• Quality Checks Application requirements Training Paradata: dashboards, control charts Verifications/validations Editing and coding Data review Site visits
36
SRO – SRC – ISR
Design Specifications (9)• Data Processing Plans
Coding Data entry Staffing Production schedule Mechanism for reporting
• Sample Post-processing Weights, nonresponse or other adjustments,
imputation, variance estimation
37
SRO – SRC – ISR
Design Specifications (10)
• Data Processing Quality Control Testing Training plans
• Supervisor training• Editor guide• DDE training• Coding/Special function training
38
SRO – SRC – ISR
Design Specifications (11)
• Materials Development (Deliverables) Datasets
• Sampling variables (weighting, nonresponse adjustments) • Sample file (result codes to check response rate calculations)• Call records• Data/Application
Respondent materials• Coversheets• Questionnaires/ applications• Questionnaire supplements/aids (e.g., respondent booklets,
event history calendars)• Contact materials
39
SRO – SRC – ISR
Design Specifications (12)
• Materials Development (Deliverables) - continued Training
• Materials (e.g., PowerPoint slides, exercises)• Manuals (supervisor/interviewer, other)• Agendas• Location/organization
Protocols• Incident/problem reporting• Technical support/back-up
40
SRO – SRC – ISR
Design Specifications (13)• Materials Development (Deliverables) - continued
Codebook Reports/Quality Profile
• Sample design description• Ethics review verification• Translation reports• Pretest reports• Field progress reports• Quality control audits• Field procedures reports
41
SRO – SRC – ISR
Constraint #2: Time
• Project Time Management
Activity Definition
Activity Sequencing
Activity Resource Estimation
Schedule Development
Schedule Control
42
SRO – SRC – ISR
Schedule Development and Control
• Identify schedule activities
• Establish optimum project work flow
• Determine resources needed
• Determine time needed for each activity
• Lay out activities in calendar time
• Keep track of progress, take corrective action
43
SRO – SRC – ISR
TASK1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Kickoff Meeting *Develop Questionnaire * * * * * * * *Programming Questionnaire * * * * *Translation of Questionnaire * * *1" Round Instrument Testing & Modifications * * * * * *Add Translation to Programming * *Sample Preparation & Listing * * * * * * * * *Native Speaker Review & Testing of Programmed, Translated Modules * *Instrument Translation Modifications * * *Instrument Programming Modifications * * *Recruit/Hiring of Production Interviewers * * * *Integrate & Test Full Instrument * *Develop & Test Sample Management * *Finalize Pretest Training/Project Materials *Duplicate Training Materials/Prepare Laptops *Pretest I Interviewer Training & Practice *Pretest I *Debrief Pretest I *Instrument & Materials Modifications, Testing *Pretest II Interviewer Training & Practice *Pretest II *Debrief Pretest II *Instrument & Materials Final Modifications *Duplicate Training Materials/Prepare Laptops *Draw Sample *Sample Assignments * *Train the Trainer *Interviewer Training I *Production Interviewing * * * * * * * * * * * *Coding * * * * * * * * * * * * *Interviewer Training II *Data Preparation * * * * * * * *Data Delivery *Documentation Preparation * * * * * * * * * * * * * * * * * * * * * * *Documentation Delivery *
Month of Survey
Project Work Plan Example
44
SRO – SRC – ISR
Constraint #3: Cost
• Project Cost Management Primarily concerned with the cost of
resources needed to complete schedule activities
45
SRO – SRC – ISR
Cost Monitoring
• Cost management processes include Cost estimating Cost budgeting Cost control
46
SRO – SRC – ISR
Design Considerations in Tenders, Bids, and Contracts
• Purpose: to thoroughly document and communicate roles, responsibilities, design details, expectations and outcomes in order to adequately budget the survey effort
• Organizational Considerations Locus of control Balancing standardization and localization Balancing funding versus contractual obligations versus expertise Methodological, substantive, local expertise Legal expertise ‘One Voice’/Point of contact
47
SRO – SRC – ISR
Budgeting and Bid Assessment
• Benchmark, where possible
• Seek multiple bids, if feasible
• Ask for project experience and references
• Ask for sample materials
• Ask for staff and organizational capacity
• Ask for facilities and equipment capacity
• Allow time for clarification and several iterations of bids
48
SRO – SRC – ISR
Budgeting and Bid Assessment
• Potential Problems Sample frame access Difficulty with concept of probability sampling Inadequate quality control procedures Not familiar/sensitive to/barrier to nonresponse reduction
techniques (number of attempts to contact households, refusal conversion techniques, inconsistent contact result coding schemes)
Inadequate interviewer training/interviewer incentives Inadequate facilities (hardware, software, Web access,
etc.) Inadequate budget detail Research ‘traditions’
49
SRO – SRC – ISR
Budget TemplateNumber Number of Months Cost per Unit Total Cost
Base Salaries Project Manager Data Manager Fieldwork Manager Accountant Assistants Supervisors Interviewers Data entry operators Drivers Translators Computer programmers Overtime Incentive payments
Travel Allowances Project Manager Data Manager Fieldwork Manager Assistants Supervisors Interviewers Drivers
50
SRO – SRC – ISR
Number Number of Months Cost per Unit Total CostMaterials Vehicle purchase Fuel and maintenance Computers Printers, etc. Publicity Computer/printer supplies Photocopier/Fax machine Office supplies Communications (phone, fax, postage, etc) Equipment maintenance Printing Costs Questionnaires Training Manuals Reports Miscellaneous (maps, listings, manuals, etc)
Consultant Costs International consultants International per diem International travel Local consultants Local per diem Local travel
Contingency (10%)
TOTAL COST
Budget Template (continued)
51
SRO – SRC – ISR
Contracting
• Specific to: Funding source Contracting organizations Employee contracts and remuneration policies Countries Type of contract
• Must have local legal advice• Commercial versus academic versus government
pros / cons
52
SRO – SRC – ISR
One Mechanism to Control Scope and Costs: Responsive Design
• Interviewer labor dominates costs on most surveys
• Surveys inherently uncertain (e.g. estimated costs and completed interviews)
• Maximize cost benefit by adapting design features based on incoming survey process data (make data driven decisions)
53
SRO – SRC – ISR
Responsive Design Implementation Process (Groves et al., 2009)
1) Pre-identify features potentially affecting costs and errors (risks)
2) Identify indicators
3) Wherever possible, devise benchmarks for each indicator
4) Create decision rules that alter features of the design if benchmarks are not met
5) Develop effective visual displays of indicators
6) When benchmarks are not met, alter the survey design based on the decision rules
54
SRO – SRC – ISR
Creating decision rules: Example
• Risk: Interviewers unable to convince eligible respondents to participate, potentially increasing non response bias
• Indicator: Cumulative average interview cooperation rate by day by interviewer by area
• Benchmark: Similar study had a 71% interview cooperation rate after 3 weeks of data collection
• Decision rule: If not at a 70% coop rate by the end of the 3rd week, identify source of problem: is it systematic across all interviewers and sample types or limited to a few interviewers or sample types, area
55
SRO – SRC – ISR
A note about quality
56
SRO – SRC – ISR
Quality Management
• Components Quality Planning and Assurance Quality Control Quality Profile
57
SRO – SRC – ISR
SRO – SRC – ISR
COMMUNICATION
59
SRO – SRC – ISR
Communication
• Communications Planning Information Distribution Performance Reporting Manage Stakeholders
60
SRO – SRC – ISR
Communication
• Frequent
• Detailed
• Mixed (face-to-face, e-mail, conference call, newsletters, Webcasts)
• Document Sharing Critical for timely review Version control
61
SRO – SRC – ISR
Data Documentation eRoom
62
SRO – SRC – ISR
Can we make this more complex?
63
SRO – SRC – ISR
YES!
We could try to do this across multiple countries…
64
SRO – SRC – ISR
Mode of Administration
65
SRO – SRC – ISR
Cross-national Considerations (1)
• Structure dynamics Leadership Levels Number of organizations Locus of control Level of control Experience at various levels Funding
66
SRO – SRC – ISR
Cross-national Considerations (2)
• Degree of standardization: Often determined by level, flow, and source of
funding Experience and research infrastructure at central
and local levels Control of data Decisions need to be made at every stage of the
survey lifecycle
67
SRO – SRC – ISR
Cross-national Considerations (3)
• Laws/ethical standards• Lingua Franca• Time zones• Terminology• Research traditions and infrastructure• Technology (Web: e-mail, VOIP, web
casts, document sharing)
68
SRO – SRC – ISR
Danke!
69