April 18, 2023

Strategies for Evaluation Data Collection

Eric Graig, Ph.D.

Slide 2Data Collection for Evaluation

Session Objectives

Participants will understand the factors to consider when selecting data collection methods

Participants will become familiar with the data collection methods most often used in evaluation studies

Participants will leave with an understanding of the resources Innovation network and LSC are providing to help develop effective data collection tools

Slide 3Data Collection for Evaluation

More Information

http://www.innonet.org/services/LSC_Train

Slide 4Data Collection for Evaluation

Starting Points

Evaluation questions Evaluation plan

Slide 5Data Collection for Evaluation

Choosing a Data Collection Strategy

Process evaluation Outcome evaluation Organizational capacity

Budget Expertise Staff time

Desired / Required N

Experiments Surveys Standardized instruments Interviews Focus groups Review of existing records

Slide 6Data Collection for Evaluation

Integrate Data Collection into Operations

The best data collection strategy is one that is seamlessly integrated into you day-to-day operations.

Think of your organization as an information system. Where in your workflow can program evaluation information be best captured?

Slide 7Data Collection for Evaluation

Be Systematic

Effective evaluation requires a close attention to detail. You needn’t be scientific but you should strive to be systematic.

Record keeping is primary. Encourage your left-brain to create forms and

protocols. Write everything down.

Slide 8Data Collection for Evaluation

Example 1

Slide 9Data Collection for Evaluation

Example 2

Slide 10Data Collection for Evaluation

When to Use Surveys

When your evaluation questions are well defined and the range of possible answers is limited

When you need to collect data from a fairly large number of people and you have access to them and can reasonably expect them to cooperate with the survey

When you have the ability to process and analyze the relatively large amount of data you will collect

Slide 11Data Collection for Evaluation

Kinds of Survey Questions

Knowledge questions Attitude questions Behavior questions Subject characteristics

Slide 12Data Collection for Evaluation

Generating Survey Questions

Evaluation QuestionsEvaluation Sub-questionsOperational QuestionsIndividual Survey Items

Slide 13Data Collection for Evaluation

Generating Survey Questions

Evaluation Questions

Evaluation Sub-questions

Operational Questions

Individual Survey Items

•What was learned?•How was identity affected?

•Were students satisfied?

•Satisfied w/ registration proc.•Satisfied with support received•Satisfied w/ course component

•Relevance of content to visits•Quality of the instructors

•Convenience of class schedule

•Right days•Right time

•Right class length

Slide 14Data Collection for Evaluation

Writing Survey Questions

Question formats Multiple choice Ordinal scales Multi-punch questions

Number of choices provided Even number for a scale

Strive for clarity Use appropriate language Avoid words with multiple meanings Be careful around dates and amounts

Slide 15Data Collection for Evaluation

Creating Neutral Questions

Dealing with obviously sensitive topics Dealing with less obviously sensitive topics

How often do you attend religious services?

1. Once a week or more2. Several times a month3. About once a month4. Several times a year5. Never

Some people attend religious services once or several times each week. Others attend only several times a year or not at all. Thinking back over the last year, how often would you say you attended religious services?

Slide 16Data Collection for Evaluation

Survey Delivery

In-person paper and pencil Mail paper and pencil Internet Telephone

Slide 17Data Collection for Evaluation

Questions

Slide 18Data Collection for Evaluation

When to Use Interviews

When you are trying to understand someone else’s subjective experience.

When your evaluation questions are more open-ended and the range of possible answers is large.

When you don’t need to generalize your findings to a larger group.

Slide 19Data Collection for Evaluation

Interview Questions

Avoid questions that lead to ‘motive talk’ Try to use:

Could you tell me about that Please tell me more about that How did that work / come about / feel

Ask interviewees to provide a narrative account of what happened or how they felt

Slide 20Data Collection for Evaluation

Tips for Interviews

Spend time on small talk in order to make the interviewee feel comfortable

Use a written set of questions with probes Follow the protocol but remain open to

exploring other themes as they arise Record if at all possible

Slide 21Data Collection for Evaluation

When to Use Focus Groups

When you are trying to understand subjective experience

When your evaluation questions are more open-ended

When you sense that respondents may be uncomfortable in a one-on-one interview

When your evaluation questions require you to understand a group dynamic

Slide 22Data Collection for Evaluation

Tips for Focus Groups

Same as for interviews Plan on 8 to 10 participants and accept that one

or two may not show up Get a good mix of participants Learn how to prevent particular participants from

dominating the group and how to encourage wall flowers to speak up

Consider an introductory exercise to get the people comfortable with one another

Slide 23Data Collection for Evaluation

Questions

Slide 24Data Collection for Evaluation

When to Program Records

When you have them and can access them When you can vouch for their accuracy and

reliability and when you know they are complete

When you can understand them

Slide 25Data Collection for Evaluation

Tips for Analyzing Program Records

Very their accuracy Find out what, if anything, is missing Be systematic Plan on making recommendations about how

to improve them to make them more useful for evaluation

Slide 26Data Collection for Evaluation

Questions

Slide 27Data Collection for Evaluation

Additional Resources

InnoNet website www.innonet.org Asynchronous material Synchronous course One-on-one technical assistance