Quantitative

Methods in

Socio-Legal

Studies: A

Methodology Clinic

Dr. Michelle Cowley

CSLS, University of Oxford

Overview

– What do we mean by a quantitative research process?

– Recent trends in socio-legal research… (e.g., Nuffield Foundation)

– What to be mindful of: variables and measures

– Experimental designs

– Example

– Survey research

– Example

– Secondary datasets

– Example

The Quantitative Process:

A deductive paradigm

1. Theory

2. Hypothesis

3. Research design- testing hypotheses

4. Devise measures of concepts

5. Select research site(s)

6. Select research participants

7. Collect data

8. Process data

9. Analyse data

10. Findings

11. Write-up

Let’s Brainstorm – What are your

research questions? Have you

thought about your methodology?

See Whiteboard & Blank Pages for Working

Illustrative Example: Evaluating PC

admissibility in Child Abuse

Proceedings

– Recent changes to evidence law regarding prior convictions (CJA, 2003)

– … In assessing the probative value of prior conviction evidence for the purposes of subsection (1)(b) the court must have regard to the following factors (and to any others it considers relevant)—

– (a) the nature and number of the events, or other things, to which the evidence relates;

– (b) when those events or things are alleged to have happened or existed;

– (c) where—

– (i) the evidence is evidence of a person’s misconduct, and

– (ii) it is suggested that the evidence has probative value by reason of similarity between that misconduct and other alleged misconduct…

How could we evaluate whether this law

positively or negatively affects the

balance of justice?

– Are there any social theories which predicts how this law change may affect the decision making process in the courtroom?

– Given these theories what would we hypothesise…

– How would we test these hypotheses?

– Let us think of some ways to obtain data to address this question… research designs…

Let’s Brainstorm –

How could we test this

research question?

See Whiteboard & Blank Pages for Working

Measures and data

– Nominal/categorical

– Scales (Interval and Likert scales)

– Reasons for choice…

– Non-parametric and parametric data

Let’s Brainstorm – Can you

think of any examples of

these measures being used in

your research literature?

See Whiteboard & Blank Pages for Working

Research design

– A framework for the collection of data…

– Experimental design- manipulation

– Quasi-experimental design

– Cross-sectional design

– Longitudinal design

– Case study design

Let’s Brainstorm – Can you

describe any research you

have read drawing on any of

these methodologies?

See Whiteboard & Blank Pages for Working

Sorts of quantitative analyses

– Univariate analyses (e.g., means)

– Bivariate analyses (e.g., correlations)

– Multivariate analyses (regression)

– Statistical significance (p <. 05)

– In real-life we need to plan the sorts of analyses at the same time as our design (a-priori vs post-hoc)

Let’s Brainstorm – Are there

examples in your thesis

literature of any of these

analyses – how might they help

us to evaluate theories?

See Whiteboard & Blank Pages for Working

Reliability and Validity

– Reliability- repeatable…

– Validity- integrity of the conclusions…

– Measurement validity

– Internal validity (e.g., cause and directionality)

– External validity (e.g., maps to real world contexts)

– Replication

– Sample representativeness is often the key

– Power of the study

Let’s Brainstorm – Can you

think of why reliability and

validity need to be embedded

in your research design?

See Whiteboard & Blank Pages for Working

An experiment: Previous convictions and

underlying ratings of guilt

Logical v Probabilistic Theoretical Frameworks… (Cowley & Colyer, 2010)

Logic Prediction 1: Participants will conclude that the defendant is ‘not guilty’ regardless of the number of previous convictions.

Probability Prediction 1: Participants will conclude that the defendant is ‘guilty’ or more guilty than chance (.5) when a previous conviction is present.

Logic Prediction 2: Participants’ underlying ratings of guilt will not be increased by the presence of previous convictions between the absolute values of not guilty (zero) and guilt (one).

Probability Prediction 2: Participants’ underlying ratings of guilt will be increased by the presence of previous convictions, and this increase will occur between the absolute values of not guilty (zero) and guilt (one).

Experiment 1: Method

Fifty one participants: 8 men and 43 women

Mean age 20.64years, range from 19 to 33 years

Design: 1 x 3 (control, one previous, two previous)

Materials: Reasoning about a scenario created from a real life case of a child who was killed by a man with two previous convictions for similar offences.

On January 2, 2006, David Baxter had been arrested. He had been accused of killing 18-month-old Joanna Connolly. Joanna’s skull had been fractured when she received a physical blow to the head. She was the daughter of Susan Connolly, the woman who David Baxter had been seeing.

Materials Template

Knowledge of previous convictions (one; two; none):David Baxter had previously served a three year sentence for being physically abusive towards an ex-girlfriend’s three year old girl in 2003.

Please answer the following questions: Q.1 Please tick whether you think:

David Baxter is guilty __David Baxter is not guilty __You cannot decide __

Q.2 On a scale of 1 to 10, circle the number that you think best reflects how guilty you think David Baxter is:

(0 represents Not Guilty, 10 represents Guilty)

Not | | | | | | | | | | | Guilty

Guilty

Experiment 1

0

10

20

30

40

50

60

70

80

90

100

no previous one previous two previous

Number of previous convictions

Perc

en

tag

e o

f verd

ict

typ

e

guilty

not guilty

cannot decide

Fig 1. Percentage of verdict type per condition. Regardless of the number of

previous convictions people could not decide if the defendant was guilty.

Chi2= 75.444 (2), p <.0005

Experiment 1

0

1

2

3

4

5

6

7

8

9

10

no previous one previous two previous

Number of previous convictions

Un

derl

yin

g r

ati

ng

of

gu

ilt

Fig 2. Underlying ratings of guilt were higher when a previous conviction was

present than when a previous conviction was not present regardless of the

number of previous convictions.

Kruskal-Wallis chi2= 16.162 (2), p <.0005

Experiment 2: Method

Seventy-two participants, 24 men and 48 women. Age range 18- 53years, mean 22.4years

Design 3 x 2 between subjects (left-handedness, right-handedness, no handedness) x (previous conviction, no previous conviction) [6 conditions]

Materials: The same scenario and measures either with or without a previous conviction and sort of handedness:

Forensic evidence showed that the blow was delivered by a left-handed person. David Baxter is left-handed

or

Forensic evidence showed that the blow was delivered by a right-handed person. David Baxter is right-handed

Experiment 2

0

2

4

6

8

10

12

Control RH LH PC PC and RH PC and LH

Presence of previous conviction (PC) and left (LH) or

right handedness (RH)

Nu

mb

er

of

pe

op

le p

er

ve

rdic

tguilty

notguilty

cannotdecide

Fig 3. The number of people who came to a ‘guilty’ ‘not guilty’ or ‘cannot decide’ verdict

in the presence of evidence of handedness (RH or LH) and/or previous convictions (PC).

(p < . 564)

People tend to significantly choose ‘cannot decide’ more often except when a previous

conviction is accompanied by evidence of a left handedness match (PC and LH).

Experiment 2

0123456789

10

Control RH LH PC PC and

RH

PC and

LHPresence of previous convictions (PC) and

left (LH) or right handedness (RH)

Un

derl

yin

g r

ati

ng

of

gu

ilt

Control

RH

LH

PC

PC and RH

PC and LH

Fig 4. Underlying ratings of guilt were higher when a previous conviction was

present than when a previous conviction was not present regardless of the number

of previous convictions (Kruskal Wallis, chi2= 12.282 (5), p <.031).

Interpreting the findings…

Even though on the surface it appears that people cannot decide, their underlying ratings of guilt are biased by knowledge of previous convictions. Previous convictions tended to bias verdicts in the presence of small amounts of confirming forensic evidence.

Perhaps indicating that people reason probabilistically towards logical conclusions.

Possibly due to similarity (e.g., PC) and not necessarily the result of a lower probability

Indicating that we next need to understand how the transition from coincidence to evidence takes place (Griffiths & Tenenbaum, 2006).

Survey/Questionnaires

– What is survey research? Data collected primarily by questionnaire or structured interview methods (big samples usually)

– Aim: patterns of association (spina-bifida and eating habits during pregnancy)

– Be aware- response rate! Power calculations are recommended.

– Often when we design surveys we wish to ask specific questions and also give room to participants to qualitatively reflect on their responses. We will focus on the first sort of question.

Let’s Brainstorm – How

might surveys help us to

advance legal empirical

research?

See Whiteboard & Blank Pages for Working

Survey

Structured interview Self completion questionnaire

Face to face Telephone Supervised Postal Internet

Paper

& pencilCAPI Paper

& pencil

…CATI

Constructing survey questions:

Likert scales

– Items (statements)

– Items must relate to the same object of study

– Inter-related items on a scale

– Equal number of positive and negative items

– Usually five point scale

– Example:

Strongly

agree

agree Neither agree

/disagree

disagree Strongly

disagree

Let’s construct some

examples of survey questions

(10mins)… Do they measure

what we expect them to?

See Whiteboard & Blank Pages for Working

Secondary datasets

– Secondary analysis of data collected by other researchers

– Secondary analysis of official statistics (i.e., stats collected by government departments)

– E.g., Ministry of Justice

– Home Office UK

– ESRC

Let’s Brainstorm – Would

secondary data sets be useful to

test your research question? Would

there be advantages or

disadvantages to using them?

See Whiteboard & Blank Pages for Working

Advantages of secondary analysis

– Cost and time

– High quality data (e.g., representative samples)

– Possibility of longitudinal analyses

– Possibility of subgroup analyses

– Possibility of new interpretations

– Unobtrusive method

Disadvantages of secondary

analysis

– Lack of familiarity with the data

– Complexity of the data

– Data quality control (knowledge of refined parameters for data mining)

– Useful- UK data archive www.data-archive.ac.uk

Example of Critical Appraisal - Crime in England and Wales

Home Office Statistical Bulletin January 2006

The risk of being a victim of crime (23%); lowest since the survey (BCS) began in 1981

Violent crime is stable compared with the previous year (BCS) Domestic burglary recorded by the police decreased by 7% (remember

this percentage)

The ‘Dark Figure’ (unreported crime which we will show still exists even in the 2006 statistics…)

Percentage risk of being a victim based on British Crime Survey

interviews in the 12 months to September 2005 compared with the

previous 12 months [adapted from the Home Office Statistical

Bulletin, (Kara & Upson, Jan 2006)]. See infographic below:

0

5

10

15

20

25

30

Interviews in Oct

2003 to Sept 2004

Interviews in Oct

2004 to Sept 2005

All household crime

Domestic burglary

All vehicle thefts

All personal crime

With injury

With no injury

All BCS crime

‘All personal crime’ dropped significantly at the p = .05 level, two tail tests

Domestic burglary didNot drop 7% according To the BCS! Multiple sources of official statistics…

Does some crime ‘disappear’ from official

statistics (i.e. police reports) or ‘remain pending’

from the event to conviction? Simmons Report

(2000)

Step 1. Does the victim notice the crime? Victimless crime Unnoticed crime

Step 2. Will the victim report the crime? Fear (humiliation, being blamed) Beliefs about police competency

Step 3. Will police record the crime? Notifiability (e.g., Ford vs Ferrari wing mirror) Change in policy over time (e.g., Steven Lawrence Inquiry) Political issues (e.g., Gudjonsson et al., 2004) Missing persons are in the inquiry stage Policing decision processes (e.g., malicious accusations)

Step 4. Will the offender be caught? Not all reported crimes are solved

Decision process (e.g., 15-16yr old shoplifting)

Step 5. Will the offender be prosecuted? Seriousness of offence

Strength of the evidence (e.g., Billy Jo Jenkins)

Enough evidence to go to trial (i.e., Crown Prosecution Service decisions)

Step 6. Will the offender be found guilty if he/she is in fact guilty?

Justice is a balancing of many complex considerations

Manslaughter versus Intent

Time to discuss! What are the

main issues and debates

likely to be in legal empirical

research?

See Whiteboard & Blank Pages for Working

General discussion…

– Are there differences between the social and natural worlds?

– … Precision at a price?

– Are there generalisations at the expense of the individual?

– … Individual or contextual factors?