Qualitative & Survey Methods for Research & Evaluation ...€¦ · Overview of qualitative methods ...

transcript

Qualitative & Survey Methods for Research & Evaluation

March 20, 2019

Orientation1. Intro to research & evaluation2. Designing projects + developing capacity3. Qualitative & survey methods4. Quantitative methods5. Synthesis, comms + decision-making

Course update1. Feedback: feeling more concrete; keep

focusing on exercises + practicalities2. Capstone update: projects are coming

together, nice work!3. Looking ahead:Ø Quantitative methods (04/03)Ø Capstone calls (04/11 – 04/12)Ø Capstone development periodØ Final session, inc. presentation (06/12)

Agenda for todayTime Topic

10:15am – 11:00amOverview of qualitative methods<<<Exercise: developing an interview guide>>>

11:00am – 11:15am Break

11:15am – 12:15pm Guest presentation: Bonnie Hough & Kim Tyda

12:15pm – 1:00pm Lunch

1:00pm – 2:15pm Overview of survey methods

2:15pm – 2:30pm Break

2:30pm – 3:45pm <<<Exercise: survey design workshop>>>

3:45pm – 4:00pm Wrapping up

Qualitative Methods

Listening Exercisehttps://www.legalaidsmc.org/videos/

From the client’s perspective, what is most important about the help they received from legal aid?

Why use qualitative methods?For capturing the human dimensionFor characterizing and understandingFor centering client voices

Opinions

Beliefs

Feelings

Reasons

Decisions

Experiences

ReflectionsPreferences

Key features of qualitative methods

1. Semi-structured2. Controlled exploration3. Active listening

Example: exploring self-representation Justice DivertedExperiences of tenants in rent court

Justice Diverted report:http://legalaidresearch.org/pub/4682/justice-diverted-how-renters-are-processed-in-the-baltimore-city-rent-court/

“The only question they ask you is, “So you agree with the amount that’s owed?” Like they don’t, you know, take the time to get your

side of the story or to see….why you haven’t paid anything, or if there’s anything wrong with the property.”

Example: value of representationVera Evaluation Experiences of NYIFUP clients

Vera Evaluation of New York Immigrant Family Unity Project:

https://storage.googleapis.com/vera-web-assets/downloads/Publications/new-

york-immigrant-family-unity-project-evaluation/legacy_downloads/new-york-

immigrant-family-unity-project-evaluation.pdf

“The legal system is a different kind of comprehension. They use

different kinds of words with legal definitions. So a regular word

would not be a regular word when applied legally…. If I wasn’t

provided a lawyer, I couldn’t stand a chance. I didn’t know the

law. Everybody in court needs a lawyer. ”

Capstone applications

Understanding client

experiences

Exploring staff and stakeholder

perceptions

Understanding how services

Developing + testing surveys

Planning qualitative projects1. Defining objectives2. Designing sample3. Selecting and tailoring a methodology4. Interview guide5. Fieldwork6. Analysis

Defining objectives

What topic(s) do you want to explore? (e.g.

benefits of representation to clients)

Try to identify the different strands of this

topic, e.g.

• Perceptions – lawyer’s role

• Experiences – relationship w/ attorney

• Attitudes – reflection on experience

Designing a sample• Goal is to capture the range or variety• Aim for a mix of known characteristics• Common sample sizes in range 10-30• Sample until responses are less varied• Create sub-samples (e.g. clients, judges)• Some variables only known at recruitment

stage (e.g. satisfied vs dissatisfied)• Think critically and consider biases

Selecting a qualitative methodology

Individual Group

ü Group dynamicü Efficient

ü Individual focusü Flexible

Tailoring a qualitative methodology

What is the best time and situation to

gather the feedback you want?

Credit: Margaret Hagan

https://medium.com/legal-design-and-innovation/a-service-design-approach-to-the-legal-system-ef01ae0c2b78

Developing an interview guideSemi-structured exploration of topicAiming for a intuitive flowAddress topic in multiple ways

Intro: welcome, consent, confidentiality

Personal & family background

Experience of contact with legal aid

Perceived strengths & weaknesses

Outro: final reflections & thanks

Questions & exercises

• Go for simple, open-ended, value-

neutral questions

• Don’t be afraid to probe (“Can you tell

me a bit more about that…?”)

• Use practical exercises to break up the

interview (e.g. journey mapping)

“How did you decide to contact Legal Aid for help?”

Recruitment & fieldwork

Early communicationRecruitment processIncentives

LocationPrivacyAccommodations

Interviews and data collection

Interviewers

• Staff or volunteers

• Language & cultural competence

• Consider trauma/traumatization

• Briefing, training & oversight

Options for data collection

Content analysis

Participant Strengths of service Weaknesses of service

1 Attorney good listener Irregular case updates

2 Attorney’s expertise Irregular case updates

3 Free representation Attorney poor listener

Presenting findings

Case studies & vignettes

Summarizing key themes

Interview Guide Exercise

Time for lunch….

Survey Methods

Why use survey methods?For gathering info or feedback from peopleFor measuring, comparing & tracking

Strongly agree

Strongly disagree

“The attorney listened carefully to what I was saying.”

Key features of survey methods

1. Using samples to understand population2. Structured topic coverage via a

questionnaire (e.g. listening behavior) 3. Measurement of characteristics (e.g. %

agree attorney listens carefully)

Example: Justice Gap study

LSC 2017 Justice Gap Study: https://www.lsc.gov/media-center/publications/2017-justice-gap-reportCalifornia’s forthcoming Justice Gap study: http://www.calbar.ca.gov/Portals/0/documents/accessJustice/Justice-Gap-Study-fact-sheet-final.pdf

71% of low-income Americans experienced at least 1 civil legal problem in last year

Example: tracking outcomes

Cleveland Legal Aid’s Texting Project: https://medium.com/innovations-in-legal-aid/cleveland-legal-aid-learned-the-outcomes-of-its-brief-services-through-texting-4c24202d7a7b

“You recently attended a legal clinic for help filing a divorce. Did you file your papers with the court? Text

Y or N to let us know.”

Capstone applications

Client intake surveys and

baseline measures

Client exist surveys and

follow-up measures

Assessing learning and

behavior change

Surveying program

characteristics

Planning survey projects• Defining objectives• Sampling• Tailoring methodology• Questionnaire• Fieldwork• Analysis

Defining objectivesWhat question(s) do you want to answer? What information do you need?

Try to distinguish between priority and contextual and supplemental information

Sampling terminology

• Target population: a group of interest; often also defined by time and place

• Census: full population measured• Sample: a subset of population used to

make inferences about the whole • Sample stratification: splitting sample

into subgroups (e.g. by case type)

Sample size• Consider aiming for n = 50 as a minimum• Sample can be acquired over time (e.g.

quarterly data collection)• Margin of error is a function of variability

in sample and population size• Increase sample size if you want to• Reduce margin of error • Enable more robust comparisons

• Try using sample size calculators: e.g. https://www.surveysystem.com/sscalc.htm

Sampling procedure• Prepare your contact list (if necessary,

separate into ‘strata’)• Randomly select a large group to

attempt to contact (only some respond)• Work through sample in batches;

attempt each contact at least 2-3 times• Record status of (a) contact and (b)

participation and (c) completion• Monitor response rate and patterns

Sampling exampleTime period Clients opening

new casesContact list(10% of clients)

Responded (assuming 25% RR)

Q1 1,000 100 25

Q2 1,000 100 25

Q3 1,000 100 25

Q4 1,000 100 25

Total 4,000 1,000 100

A sample of 100 for population of 4,000

gives +/- 10% margin of error

Managing contact listsName Intake

DateLegal area Home Phone Mobile

phoneEmail Consent

Client A 12/01/19 Consumer 000-000-0000 000-000-0000 clienta@gmail.com Yes

Client B 12/01/19 Housing 000-000-0000 clientb@gmail.com

Client C 12/02/19 Housing clientc@gmail.com No

Client D 12/03/19 Housing 000-000-0000 000-000-0000 clientd@gmail.com Yes

ü Deduplicated ü Contact info providedü Consent given

Are my results reliable?• All surveys contain error; goal is to avoid

erroneous conclusions• Best practice is to try to estimate error• Sampling errors (sample size)• Other biases (response bias)

• Importance depends on how data is used, esp. for decision-making

• Avoid assuming too much precision when sample size is small

Survey modes

Structuring a questionnaire

Introduction

Contextual info (if needed)

*** Priority section ***

Supplemental info (if needed)

Conclusion + exiit

Crafting questions

Binary QuestionsUse yes/no questions when:• It is an inherently binary situation• You want a very simple measure

Have you completed a training course on this subject before?

Did you file your papers with the court? Text Y or N to let us know.

Multiple select questionsUse when you want respondents to select all that apply (not mutually exclusive)

What kind of legal help did you receive? Check all that apply.

1. I was referred to legal information online2. I got assistance filling out a legal documents or forms3. I got legal advice4. A legal professional helped me negotiate with the

other person [...]5. I was represented by a legal professional in court

Multiple choice (unordered)Use when you want a response in only one of a mutually exclusive set of categories

Which of these statements best describes the status of your legal issue?

1. The issue is ongoing, and I’m still dealing with it

2. The issue is not resolved, but I haven’t been dealing with it lately

3. The issue is resolved

Multiple choice (ordered)Use these when you want a response that falls on a scale or range (unipolar or bipolar)

How much did this legal issue personally affect you?

1. Not at all2. Slightly 3. Moderately4. Very Much5. Severely

How strong is your peer support network?

1. Very weak2. Weak3. Fair4. Strong5. Very strong

Likert scalesUse these when you want to gauge agreement with a number of statements

“The attorney listed carefully to what I was saying.”

1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree

“The attorney communicated with me regularly.”

1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree

Open questionsUse these when you do not want to narrow the possible range of response

What was most helpful about the service you received from Legal Aid?

In your opinion, how could the service be improved?

User testingUse qualitative interviews to test the survey with respondents (esp. if complex)

Fieldwork• Internal and external communications

• Changes to organizational processes

• Training and managing staff / volunteers

• During fieldwork, monitoring

• Response rates

• Non-response to specific questions

• Interviewer consistency

• Data management in spreadsheets

• Managing follow-up

Preparing survey data for analysis

Resp. ID Age Office Legal Area

Q1. Listening

Q1 Numeric

Q3. Filed

1 20 A Family S. Agree 5 Filed

2 30 A Family Agree 4 Filed

3 40 B Family Neither 3 Not Filed

4 50 A Family Disagree 2 Refused

5 40 B Family S. Disagree 1 Filed

Analysis of survey dataFrequencies + proportions

Yes filed Not filed Missing Total

Frequency 35 13 2 50

Percentage 70% 26% 4% 100%

Valid % 73% 27% -- 100%

Cross-tabulationsYes filed Not filed Missing Total

Office A 23 (77%) 6 (20%) 1 (3%) 30 (100%)

Office B 12 (60%) 7 (35%) 1 (5%) 20 (100%)

Total 35 (70%) 13 (26%) 2 (4%) 50 (100%)

Analysis of survey data3. Grouping ordered responses

Agree/strongly agree

Neither Disagree/Strongly disagree

Frequency 30 5 15 50

Percentage 60% 10% 30% 100%

4. Ordered responses as numericStatistic Listening

Mean 4.2

SD 1.2

Survey Design Workshop

Thank you! Wrapping upReflections from today

Qualitative & Survey Methods for Research & Evaluation ...€¦ · Overview of qualitative methods ...

Documents