Post on 16-Dec-2015
transcript
April 18, 2023
Strategies for Evaluation Data Collection
Eric Graig, Ph.D.
Slide 2Data Collection for Evaluation
Session Objectives
Participants will understand the factors to consider when selecting data collection methods
Participants will become familiar with the data collection methods most often used in evaluation studies
Participants will leave with an understanding of the resources Innovation network and LSC are providing to help develop effective data collection tools
Slide 3Data Collection for Evaluation
More Information
http://www.innonet.org/services/LSC_Train
Slide 4Data Collection for Evaluation
Starting Points
Evaluation questions Evaluation plan
Slide 5Data Collection for Evaluation
Choosing a Data Collection Strategy
Process evaluation Outcome evaluation Organizational capacity
Budget Expertise Staff time
Desired / Required N
Experiments Surveys Standardized instruments Interviews Focus groups Review of existing records
Slide 6Data Collection for Evaluation
Integrate Data Collection into Operations
The best data collection strategy is one that is seamlessly integrated into you day-to-day operations.
Think of your organization as an information system. Where in your workflow can program evaluation information be best captured?
Slide 7Data Collection for Evaluation
Be Systematic
Effective evaluation requires a close attention to detail. You needn’t be scientific but you should strive to be systematic.
Record keeping is primary. Encourage your left-brain to create forms and
protocols. Write everything down.
Slide 8Data Collection for Evaluation
Example 1
Slide 9Data Collection for Evaluation
Example 2
Slide 10Data Collection for Evaluation
When to Use Surveys
When your evaluation questions are well defined and the range of possible answers is limited
When you need to collect data from a fairly large number of people and you have access to them and can reasonably expect them to cooperate with the survey
When you have the ability to process and analyze the relatively large amount of data you will collect
Slide 11Data Collection for Evaluation
Kinds of Survey Questions
Knowledge questions Attitude questions Behavior questions Subject characteristics
Slide 12Data Collection for Evaluation
Generating Survey Questions
Evaluation QuestionsEvaluation Sub-questionsOperational QuestionsIndividual Survey Items
Slide 13Data Collection for Evaluation
Generating Survey Questions
Evaluation Questions
Evaluation Sub-questions
Operational Questions
Individual Survey Items
•What was learned?•How was identity affected?
•Were students satisfied?
•Satisfied w/ registration proc.•Satisfied with support received•Satisfied w/ course component
•Relevance of content to visits•Quality of the instructors
•Convenience of class schedule
•Right days•Right time
•Right class length
Slide 14Data Collection for Evaluation
Writing Survey Questions
Question formats Multiple choice Ordinal scales Multi-punch questions
Number of choices provided Even number for a scale
Strive for clarity Use appropriate language Avoid words with multiple meanings Be careful around dates and amounts
Slide 15Data Collection for Evaluation
Creating Neutral Questions
Dealing with obviously sensitive topics Dealing with less obviously sensitive topics
How often do you attend religious services?
1. Once a week or more2. Several times a month3. About once a month4. Several times a year5. Never
Some people attend religious services once or several times each week. Others attend only several times a year or not at all. Thinking back over the last year, how often would you say you attended religious services?
Slide 16Data Collection for Evaluation
Survey Delivery
In-person paper and pencil Mail paper and pencil Internet Telephone
Slide 17Data Collection for Evaluation
Questions
Slide 18Data Collection for Evaluation
When to Use Interviews
When you are trying to understand someone else’s subjective experience.
When your evaluation questions are more open-ended and the range of possible answers is large.
When you don’t need to generalize your findings to a larger group.
Slide 19Data Collection for Evaluation
Interview Questions
Avoid questions that lead to ‘motive talk’ Try to use:
Could you tell me about that Please tell me more about that How did that work / come about / feel
Ask interviewees to provide a narrative account of what happened or how they felt
Slide 20Data Collection for Evaluation
Tips for Interviews
Spend time on small talk in order to make the interviewee feel comfortable
Use a written set of questions with probes Follow the protocol but remain open to
exploring other themes as they arise Record if at all possible
Slide 21Data Collection for Evaluation
When to Use Focus Groups
When you are trying to understand subjective experience
When your evaluation questions are more open-ended
When you sense that respondents may be uncomfortable in a one-on-one interview
When your evaluation questions require you to understand a group dynamic
Slide 22Data Collection for Evaluation
Tips for Focus Groups
Same as for interviews Plan on 8 to 10 participants and accept that one
or two may not show up Get a good mix of participants Learn how to prevent particular participants from
dominating the group and how to encourage wall flowers to speak up
Consider an introductory exercise to get the people comfortable with one another
Slide 23Data Collection for Evaluation
Questions
Slide 24Data Collection for Evaluation
When to Program Records
When you have them and can access them When you can vouch for their accuracy and
reliability and when you know they are complete
When you can understand them
Slide 25Data Collection for Evaluation
Tips for Analyzing Program Records
Very their accuracy Find out what, if anything, is missing Be systematic Plan on making recommendations about how
to improve them to make them more useful for evaluation
Slide 26Data Collection for Evaluation
Questions
Slide 27Data Collection for Evaluation
Additional Resources
InnoNet website www.innonet.org Asynchronous material Synchronous course One-on-one technical assistance