Qualitative & Survey Methods for Research & Evaluation ...€¦ · Overview of qualitative methods ...

Copyright © 2018 by OneJustice. All rights reserved.

Qualitative & Survey Methods for Research & Evaluation

March 20, 2019


Orientation1. Intro to research & evaluation2. Designing projects + developing capacity3. Qualitative & survey methods4. Quantitative methods5. Synthesis, comms + decision-making


Course update1. Feedback: feeling more concrete; keep

focusing on exercises + practicalities2. Capstone update: projects are coming

together, nice work!3. Looking ahead:Ø Quantitative methods (04/03)Ø Capstone calls (04/11 – 04/12)Ø Capstone development periodØ Final session, inc. presentation (06/12)


Agenda for todayTime Topic

10:15am – 11:00amOverview of qualitative methods<<<Exercise: developing an interview guide>>>

11:00am – 11:15am Break

11:15am – 12:15pm Guest presentation: Bonnie Hough & Kim Tyda

12:15pm – 1:00pm Lunch

1:00pm – 2:15pm Overview of survey methods

2:15pm – 2:30pm Break

2:30pm – 3:45pm <<<Exercise: survey design workshop>>>

3:45pm – 4:00pm Wrapping up


Qualitative Methods


Listening Exercisehttps://www.legalaidsmc.org/videos/

From the client’s perspective, what is most important about the help they received from legal aid?


Why use qualitative methods?For capturing the human dimensionFor characterizing and understandingFor centering client voices

Opinions

Beliefs

Feelings

Reasons

Decisions

Experiences

ReflectionsPreferences


Key features of qualitative methods

1. Semi-structured2. Controlled exploration3. Active listening


Example: exploring self-representation Justice DivertedExperiences of tenants in rent court

Justice Diverted report:http://legalaidresearch.org/pub/4682/justice-diverted-how-renters-are-processed-in-the-baltimore-city-rent-court/

“The only question they ask you is, “So you agree with the amount that’s owed?” Like they don’t, you know, take the time to get your

side of the story or to see….why you haven’t paid anything, or if there’s anything wrong with the property.”

http://legalaidresearch.org/pub/4682/justice-diverted-how-renters-are-processed-in-the-baltimore-city-rent-court/

http://legalaidresearch.org/pub/4682/justice-diverted-how-renters-are-processed-in-the-baltimore-city-rent-court/


Example: value of representationVera Evaluation Experiences of NYIFUP clients

Vera Evaluation of New York Immigrant Family Unity Project:

https://storage.googleapis.com/vera-web-assets/downloads/Publications/new-

york-immigrant-family-unity-project-evaluation/legacy_downloads/new-york-

immigrant-family-unity-project-evaluation.pdf

“The legal system is a different kind of comprehension. They use

different kinds of words with legal definitions. So a regular word

would not be a regular word when applied legally…. If I wasn’t

provided a lawyer, I couldn’t stand a chance. I didn’t know the

law. Everybody in court needs a lawyer. ”

https://storage.googleapis.com/vera-web-assets/downloads/Publications/new-york-immigrant-family-unity-project-evaluation/legacy_downloads/new-york-immigrant-family-unity-project-evaluation.pdf

https://storage.googleapis.com/vera-web-assets/downloads/Publications/new-york-immigrant-family-unity-project-evaluation/legacy_downloads/new-york-immigrant-family-unity-project-evaluation.pdf


Capstone applications

Understanding client

experiences

Exploring staff and stakeholder

perceptions

Understanding how services

work

Developing + testing surveys


Planning qualitative projects1. Defining objectives2. Designing sample3. Selecting and tailoring a methodology4. Interview guide5. Fieldwork6. Analysis


Defining objectives

What topic(s) do you want to explore? (e.g.

benefits of representation to clients)

Try to identify the different strands of this

topic, e.g.

• Perceptions – lawyer’s role

• Experiences – relationship w/ attorney

• Attitudes – reflection on experience


Designing a sample• Goal is to capture the range or variety• Aim for a mix of known characteristics• Common sample sizes in range 10-30• Sample until responses are less varied• Create sub-samples (e.g. clients, judges)• Some variables only known at recruitment

stage (e.g. satisfied vs dissatisfied)• Think critically and consider biases


Selecting a qualitative methodology

Individual Group

ü Group dynamicü Efficient

ü Individual focusü Flexible


Tailoring a qualitative methodology

What is the best time and situation to

gather the feedback you want?

Credit: Margaret Hagan

https://medium.com/legal-design-and-innovation/a-service-design-approach-to-the-legal-system-ef01ae0c2b78




Developing an interview guideSemi-structured exploration of topicAiming for a intuitive flowAddress topic in multiple ways

Intro: welcome, consent, confidentiality

Personal & family background

Experience of contact with legal aid

Perceived strengths & weaknesses

Outro: final reflections & thanks


Questions & exercises

• Go for simple, open-ended, value-

neutral questions

• Don’t be afraid to probe (“Can you tell

me a bit more about that…?”)

• Use practical exercises to break up the

interview (e.g. journey mapping)

“How did you decide to contact Legal Aid for help?”


Recruitment & fieldwork

Early communicationRecruitment processIncentives

LocationPrivacyAccommodations


Interviews and data collection

Interviewers

• Staff or volunteers

• Language & cultural competence

• Consider trauma/traumatization

• Briefing, training & oversight

Options for data collection


Content analysis

Participant Strengths of service Weaknesses of service

1 Attorney good listener Irregular case updates

2 Attorney’s expertise Irregular case updates

3 Free representation Attorney poor listener


Presenting findings

Case studies & vignettes

Summarizing key themes


Interview Guide Exercise


Interview Guide Exercise


Time for lunch….


Survey Methods


Why use survey methods?For gathering info or feedback from peopleFor measuring, comparing & tracking

Strongly agree

Strongly disagree

“The attorney listened carefully to what I was saying.”


Key features of survey methods

1. Using samples to understand population2. Structured topic coverage via a

questionnaire (e.g. listening behavior) 3. Measurement of characteristics (e.g. %

agree attorney listens carefully)


Example: Justice Gap study

LSC 2017 Justice Gap Study: https://www.lsc.gov/media-center/publications/2017-justice-gap-reportCalifornia’s forthcoming Justice Gap study: http://www.calbar.ca.gov/Portals/0/documents/accessJustice/Justice-Gap-Study-fact-sheet-final.pdf

71% of low-income Americans experienced at least 1 civil legal problem in last year

https://www.lsc.gov/media-center/publications/2017-justice-gap-report

http://www.calbar.ca.gov/Portals/0/documents/accessJustice/Justice-Gap-Study-fact-sheet-final.pdf


Example: tracking outcomes

Cleveland Legal Aid’s Texting Project: https://medium.com/innovations-in-legal-aid/cleveland-legal-aid-learned-the-outcomes-of-its-brief-services-through-texting-4c24202d7a7b

“You recently attended a legal clinic for help filing a divorce. Did you file your papers with the court? Text

Y or N to let us know.”

https://medium.com/innovations-in-legal-aid/cleveland-legal-aid-learned-the-outcomes-of-its-brief-services-through-texting-4c24202d7a7b


Capstone applications

Client intake surveys and

baseline measures

Client exist surveys and

follow-up measures

Assessing learning and

behavior change

Surveying program

characteristics


Planning survey projects• Defining objectives• Sampling• Tailoring methodology• Questionnaire• Fieldwork• Analysis


Defining objectivesWhat question(s) do you want to answer? What information do you need?

Try to distinguish between priority and contextual and supplemental information


Sampling terminology

• Target population: a group of interest; often also defined by time and place

• Census: full population measured• Sample: a subset of population used to

make inferences about the whole • Sample stratification: splitting sample

into subgroups (e.g. by case type)


Sample size• Consider aiming for n = 50 as a minimum• Sample can be acquired over time (e.g.

quarterly data collection)• Margin of error is a function of variability

in sample and population size• Increase sample size if you want to• Reduce margin of error • Enable more robust comparisons

• Try using sample size calculators: e.g. https://www.surveysystem.com/sscalc.htm

https://www.surveysystem.com/sscalc.htm


Sampling procedure• Prepare your contact list (if necessary,

separate into ‘strata’)• Randomly select a large group to

attempt to contact (only some respond)• Work through sample in batches;

attempt each contact at least 2-3 times• Record status of (a) contact and (b)

participation and (c) completion• Monitor response rate and patterns


Sampling exampleTime period Clients opening

new casesContact list(10% of clients)

Responded (assuming 25% RR)

Q1 1,000 100 25

Q2 1,000 100 25

Q3 1,000 100 25

Q4 1,000 100 25

Total 4,000 1,000 100

A sample of 100 for population of 4,000

gives +/- 10% margin of error


Managing contact listsName Intake

DateLegal area Home Phone Mobile

phoneEmail Consent

Client A 12/01/19 Consumer 000-000-0000 000-000-0000 [email protected] Yes

Client B 12/01/19 Housing 000-000-0000 [email protected]

Client C 12/02/19 Housing [email protected] No

Client D 12/03/19 Housing 000-000-0000 000-000-0000 [email protected] Yes

Client D 12/03/19 Housing 000-000-0000 000-000-0000 [email protected] Yes

ü Deduplicated ü Contact info providedü Consent given


Are my results reliable?• All surveys contain error; goal is to avoid

erroneous conclusions• Best practice is to try to estimate error• Sampling errors (sample size)• Other biases (response bias)

• Importance depends on how data is used, esp. for decision-making

• Avoid assuming too much precision when sample size is small


Survey modes


Structuring a questionnaire

Introduction

Contextual info (if needed)

*** Priority section ***

Supplemental info (if needed)

Conclusion + exiit


Crafting questions


Binary QuestionsUse yes/no questions when:• It is an inherently binary situation• You want a very simple measure

Have you completed a training course on this subject before?

Did you file your papers with the court? Text Y or N to let us know.


Multiple select questionsUse when you want respondents to select all that apply (not mutually exclusive)

What kind of legal help did you receive? Check all that apply.

1. I was referred to legal information online2. I got assistance filling out a legal documents or forms3. I got legal advice4. A legal professional helped me negotiate with the

other person [...]5. I was represented by a legal professional in court


Multiple choice (unordered)Use when you want a response in only one of a mutually exclusive set of categories

Which of these statements best describes the status of your legal issue?

1. The issue is ongoing, and I’m still dealing with it

2. The issue is not resolved, but I haven’t been dealing with it lately

3. The issue is resolved


Multiple choice (ordered)Use these when you want a response that falls on a scale or range (unipolar or bipolar)

How much did this legal issue personally affect you?

1. Not at all2. Slightly 3. Moderately4. Very Much5. Severely

How strong is your peer support network?

1. Very weak2. Weak3. Fair4. Strong5. Very strong


Likert scalesUse these when you want to gauge agreement with a number of statements

“The attorney listed carefully to what I was saying.”

1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree

“The attorney communicated with me regularly.”

1. Strongly disagree2. Disagree3. Neither agree nor disagree4. Agree5. Strongly agree


Open questionsUse these when you do not want to narrow the possible range of response

What was most helpful about the service you received from Legal Aid?

In your opinion, how could the service be improved?


User testingUse qualitative interviews to test the survey with respondents (esp. if complex)


Fieldwork• Internal and external communications

• Changes to organizational processes

• Training and managing staff / volunteers

• During fieldwork, monitoring

• Response rates

• Non-response to specific questions

• Interviewer consistency

• Data management in spreadsheets

• Managing follow-up


Preparing survey data for analysis

Resp. ID Age Office Legal Area

Q1. Listening

Q1 Numeric

Q3. Filed

1 20 A Family S. Agree 5 Filed

2 30 A Family Agree 4 Filed

3 40 B Family Neither 3 Not Filed

4 50 A Family Disagree 2 Refused

5 40 B Family S. Disagree 1 Filed


Analysis of survey dataFrequencies + proportions

Yes filed Not filed Missing Total

Frequency 35 13 2 50

Percentage 70% 26% 4% 100%

Valid % 73% 27% -- 100%

Cross-tabulationsYes filed Not filed Missing Total

Office A 23 (77%) 6 (20%) 1 (3%) 30 (100%)

Office B 12 (60%) 7 (35%) 1 (5%) 20 (100%)

Total 35 (70%) 13 (26%) 2 (4%) 50 (100%)


Analysis of survey data3. Grouping ordered responses

Agree/strongly agree

Neither Disagree/Strongly disagree

Total

Frequency 30 5 15 50

Percentage 60% 10% 30% 100%

4. Ordered responses as numericStatistic Listening

Mean 4.2

SD 1.2


Survey Design Workshop


Thank you! Wrapping upReflections from today

Date post:	26-May-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	1 times

Qualitative & Survey Methods for Research & Evaluation ...€¦ · Overview of qualitative methods ...

Documents