Making Public Spaces Safer

Dr B. Nasa, Prof J. BinnerDr M. A. Ferrario, Dr W. Simm, Prof J. Whittle, Dr B. Lam;

Sheffield Management SchoolLancaster University/ Brunel University

Towards Policing Perception Maps automatically extracting quantitative analysis of

qualitative survey data

Background: Derry/NI

HOW TO build trust in the police after 30+ years of Conflict (The Troubles)?

SELF POLICING & FREE DERRY: 1960s on going clashes between Protestants and Catholics. By the end of 1971, barricades prevented access to Free Derry (Bog Side Area) . No police forces allowed.

BLOODY SUNDAY: 30th January 1972, 27 civilians were shot by the British Army Parachute Regiment during a Civil Rights march. 13 died on the scene.

Derry District Policing Partnership

Derry DPP is one of the 26 DPPs in NI

• Responsibilities:• Monitoring local police performance• Giving a voice to community views on policing• Gaining the public's cooperation to prevent crime.

DDPP Household survey 2009 Confidence and Satisfaction

DDPP Household Satisfaction & Confidence Survey’09

• All-household survey to inform next year local policing plan (2010/11)

• First-ever DPP blanket consultation by Neighbourhood

• 46,000 households: 465 responses received to the survey (1% response rate)

Verbatim vs Tick-Box

• The survey contained questions with multiple choice answers and the opportunity for the respondent to expand upon their selection in free text form.

• Traditionally, this free text is difficult and time consuming to analyse by theme and sentiment.

• Automatically extracting ‘quantitative summaries’ could help identify themes which are not part of the multiple choice options and would allow comparison across results from different districts.

Making Sense of Qualitative Data

Factual vs Perception Mapping

Map of actual crime statistics

Perceptions maps

Challenge: How do people feel...

• 36% of respondents** answered ‘don’t know’ to the satisfaction question => How can the police ‘win’ the undecided?

• Understanding ‘Why’people are undecidedmay help…

SATISFACTION (Derry City Council Area)

Yes35%

No27%

Don't Know36%

No Answer2%

Yes

No

Don't Know

No Answer

** diagram and figures from DDPP survey

Data Set: Original Source

1

2

3

4

56

DDPP IDLOCALECONFIDENCE LEVELCONFIDENCE VERBATIMSATISFACTION LEVELSATISFACTION VERBATIM

123456

Extracted – Full Data Set

First Step: 449 Records in totalSecond Step: Sentence Level Split + Clean Data+ Remove Blank Verbatim = 379 Sentence level comments (Satisfaction)= 485 sentence level comment (Confidence)

Sample Data Set – 95% CL; CI 5%

• 200 comments (Satisfaction)• 221 comments (Confidence)• Manual analysis (three researchers)

• Theme, Actionability, Sentiment (TAS)

• Automated Theme Extraction – Accuracy Testing• VYV• Bayes Classifier

TOP THEMES (Satisfaction & Confidence)Approach:1. Initial Identification of themes (Cross-checked with DDPP)2. Independent Theme Review (by three researchers)3. Up to 4 themes per comments4. Calibration and Compilation of final theme List

Emerging Results:

Satisfaction Frequency % Confidence Frequency %General ImpressionGeneral Impression 6161 17.617.6 Response Time and QualityResponse Time and Quality 7070 18.418.4

Presence and Visibility 48 13.9 Police Behaviour/Political Policing 64 16.8

Police Behaviour/Political Policing 46 13.3 Presence and Visibility 61 16.0

Offence and Crime Level 40 11.6 General Impression 38 10.0

Response Time and Quality 32 9.2 Resources incl. Admin 31 8.1

Other 29 8.4 Offence and Crime Level 23 6.0

Resources incl. Admin 21 6.1 People behaviour and Attitude 21 5.5

People behaviour and Attitude 17 4.9 Other 17 4.5

Community Policing 14 4.0 Community Policing 15 3.9

Specific Events/Location 14 4.0 Specific Events/Location 15 3.9

Complexity of Situation 10 2.9 Complexity of Situation 13 3.4

Authority/Law enforcement 10 2.9 Perceived Safety 7 1.8

Perceived Safety 4 1.2 Authority/Law enforcement 6 1.6

Total – themes 346 100 Total 381 100%

THEMES BY LOCATIONSecond Step: Match Themes to NeighbourhoodsNeighbourhood analysis of Satisfaction with Policing in Derry Themes (aggregated) Foyle City

CentreCity North Foyle City

WestWaterside

RuralWaterside

UrbanTotal by ThemesFreq. %

General Impression 8 17 8 17 11 61 17.6

Presence and Visibility 2 13 9 16 8 48 13.9

Police Behaviour/political policing 4 13 7 13 9 46 13.3

Offences and Crime Level 3 14 8 11 4 40 11.6

Response Time and Quality 1 2 6 11 12 32 9.2

Other 5 3 4 10 7 29 8.4

Resources Incl. Admin 2 3 5 4 7 21 6.1

People Behaviour 3 7 2 2 3 17 4.9

Community Policing - 5 3 3 3 14 4.0

Specific Events/Location - 4 2 5 3 14 4.0

Complexity of Situation 3 2 - 4 1 10 2.9

Authority/Law Enforcement - 5 1 1 3 10 2.9

Perceived Safety - - - 4 - 4 1.2

NeighbourhoodFreq. 31 88 55 101 71 346 100.0

% 9.0 25.4 15.9 29.2 20.5 100.0

3-WAY ANALYSIS: themes v. sentiments v. location

SATISFACTION v. SENTIMENTS

Is stated level of satisfaction reflected in verbatim

Satisfaction Level

Sentiment derived from verbatim

Negative Neutral Positive Total

No 109 14 0 123

32.15% 4.13% 0% 36.28%

Don’t Know 46 40 6 9213.57% 11.8% 1.77% 27.14%

Yes 14 32 78 1244.13% 9.44% 23.01% 36.58%

Total 169 86 84 33949.85% 25.37% 24.78% 100%

Null: SL is independent of VSChi square statistics : χ2 = 211.04 (0.000 )Reject null, so SL is associated with VS

ExampleVERBATIM ANSWERS**

Do you think the PSNI are doing a good job in the Foyle area? Don’t know – PSNI take too long to respond to a reported incident.

Don’t Know - They certainly don’t appear to be able to curb the problems in my area of Culmore.

Don’t Know -Because they let the young joy riders straight out again also they know all the drug dealers in the one area still they turn a blind eye.

Automated Theme Extraction

VYV-Bayes Classifier

Accuracy of Automated VYV Theme Extraction Approach:

1. VYV automatically assigned (two) themes

2. Accuracy assessed on a 3-point Likert scale

3. Independent assessment by 2 reviewers, followed by a ‘calibration exercise’ to address differences

4. 69-72% of the themes were acceptable or accurate for both sets

5. Reviewers agreed on both themes 93-94% of the time

• Noun picked from POS = “Staff” sem tag I3.1

• Secondary sem tag S2

<w pos="NN" lemma="staff" sem="I3.1/S2">staff</w>

“Very helpful and friendly staff”

VYV tags vs Manual (Satisfaction)

VYV identified a total of 90 unique themesManual tagging identified 17 themesSome Interesting Parallels and Challenges

Unique Themes freq % rank (top 17)Places 34 9.7% 1Work and employment: Generally 14 4.0% 2Evaluation: Good 12 3.4% 3Speech acts 11 3.1% 4Belonging to a group 11 3.1% 5Difficult 11 3.1% 6People 10 2.8% 7Time: Period 10 2.8% 8Grammatical bin 9 2.6% 9Time: New and young 9 2.6% 10Location and direction 9 2.6% 11Helping 9 2.6% 12General actions / making 9 2.6% 13Existing 9 2.6% 14Vehicles and transport on land 9 2.6% 15Crime 8 2.3% 16Getting and possession 7 2.0% 17

Theme Assignment Alternative

Classifier methods can be used to classify texts:

1. The classifier is trained using manual data2. A number of exclusive classifications are defined 3. As subset of data is selected and manually classified4. This subset is used to train the classifier on the

features (in this case the words) that make up each classification

5. New comment are classified by the classifier based on features

Bayes Classifier applied to DDPP Data

• Naive Bayes Classifier was applied to “Confidence” data:• Training subset used previous 83 manually tagged

comments• classifier accuracy increases with number of training data• 20 further comments were extracted at random and

classified by the Bayes classifier to test accuracy.

• Preliminary Results:• In a manual review by 2 researchers (method described

previously) the classifier returned an “Acceptable” or “Accurate” classification for 60% of the 20 comments.

Towards Perception Maps

Interactive Map Display

• Plot the information derived from automatic analysis of survey verbatim on an interactive map

• Allows interaction with, and evaluation of the data by stakeholders and possibly public

Safer Streets

Implementation

• Design Storyboard by Information Architecture student Zoe Zhao (Brunel University)

• Development Progress using web technologies• Google maps• Web 2.0 technology including Ajax and JQuery

1. Frontpage, which contains:

i. Map of the whole area

ii. Overall impression – e.g. Themes, Number of People Reporting, etc

StoryBoard A

StoryBoard B

2. Click on the map to select the area that you want more information

3. The selected area will be highlighted

4. The information of the specific area replaces the overall impression

StoryBoard C

5. Select the information that you want to explore further

6. Place the cursor on the theme for actual figures

7. You also can rate your satisfaction level in this category

8. The sentiment charts will be updated automatically

StoryBoard D

9. Click on the themes for actual comments

10. You will see both positive and negative comments under this theme

11. You can enter your comments or respond to other people’s comments

12. You will see your comment appears on the list straightaway

DDPP Survey Data 2009

Challenges:

• Database design• Google map plots• Ajax for realtime updates• Jquery interface animation

Conclusion and Next Step

• Empirical Analysis• Test specific hypothesis such as

• whether there is more crime/offences where police are sectarian

• Is response related with police behaviour

• Automatic TAS• Train bayes classifier using larger dataset with fewer

categories

• Interactive Map• Put the plans into action

Partners & Collaborators

www.voiceyourview.com