Using Clojure for Sentiment Analysis of the Twittersphere

transcript

Sentiment Analysis of the Twittersphere

@APassionForCode

Leiningen Versus the Ants!

Clojure Versus Java!

FP Versus OOP

Leiningen Versus the Ants!

FP Versus OOP

Apache Ant

Leiningen Versus the Ants

FP Versus OOP

Apache Ant

Leiningen Versus the Ants

Clojure Versus Java

Functional Versus Object Oriented

Apache Ant

“You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.”

Joe Armstrong - inventor of Erlang

“Sometimes, the elegant implementation is just a function. Not a method. Not a class. Not a

framework. Just a function.”John Carmack - Lead on Doom, Quake, Rage

Blood aches pain sick shivers spasm vomit dizzy fainting colic and in a coma!

1 FAVORITE

Scores HIGH for flu symptoms

1 FAVORITE

Who is Catalina Rubottom?

1 FAVORITE

HDFS/Hadoop

Mongo/Aggregation

Mongo/MapReduce

Postgres

30 million geo-tagged tweets sent from UK

How can we do fast, real time analytics of

social media?

Store this!

Not this!

WARNING!You’re about to see real Clojure code!

increment 1)(

Congratulations - you’re now fully qualified!

increment 1)(

ptaoussanis/carmine

(:require [taoensso.carmine :as car]) !!(defn create-tweet-id [] (first (wcar* (car/incr "global:tweet.id")))) !!(defn set-values [sentiment location created-at] (let [tweed-id (create-tweet-id)] (wcar* (car/setbit sentiment tweet-id 1) (car/setbit location tweet-id 1) (car/setbit created-at tweet-id 1))))

find value for keyfind bit of index

set bit to 1

(wcar* (car/bitcount "SCOTLAND") (car/bitcount "JOVIALITY") (car/bitcount "26062014114532"))

valuekey

(car/setbit “ENG” 0 1)

tweet-idkey

valuekey

ENG 0 1

SCT 1 0

WAL 0 0

NIR 0 0

(car/setbit “SCT” 1 1)

tweet-idkey

valuekey

ENG 1 0 1

SCT 0 1 0

WAL 0 0 0

NIR 0 0 0

(car/setbit “ENG” 2 1)

tweet-idkey

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/setbit “SCT” 15 1)

tweet-idkey

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "ENG")

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "SCT")

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "WAL")

valuekey

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

WAL 0 1 0 1 0 0 0 0 0 1 0 0 0 0 0 0

NIR 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

(car/bitcount "NIR")

valuekey

JOVIAL

HOSTILE

FATIGUE

valuekey

JOVIAL 1

HOSTILE 0

FATIGUE 0

(car/setbit “JOVIAL” 0 1)

tweet-idkey

valuekey

JOVIAL 0 1

SHY 0 0

HOSTILE 1 0

FATIGUE 0 0

(car/setbit “HOSTILE” 1 1)

tweet-idkey

valuekey

JOVIAL 0 0 1

SHY 1 0 0

HOSTILE 0 1 0

FATIGUE 0 0 0

(car/setbit “SHY” 2 1)

tweet-idkey

valuekey

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0

FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

(car/setbit “JOVIAL” 15 1)

tweet-idkey

valuekey

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

SHY 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0

FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

(car/bitcount [*])

valuekey

(wcar*

(car/bitop "AND" “ENGLAND&JOVIALITY" "ENGLAND" "JOVIALITY")

(car/expire “ENGLAND&JOVIALITY" 10)

(car/bitcount “ENGLAND&JOVIALITY"))

How many people in England are Happy about the referendum result?

(wcar*

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1

ENG 0 0 1 0 1 1 0 1 0 0 1 0 1 1 0 1

JOVIAL 1 0 1 0 1 1 1 0 0 1 0 1 0 0 0 1

0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 1

(car/bitcount “ENGLAND&JOVIALITY")

(wcar*

(car/bitop "OR" "FATIGUE|HOSTILITY" "FATIGUE" “HOSTILITY")

(car/expire "FATIGUE|HOSTILITY" 10)

(car/bitop "AND" "SCOTLAND&(FATIGUE|HOSTILITY)"

"SCOTLAND" "FATIGUE|HOSTILITY")

(car/expire "SCOTLAND&(FATIGUE|HOSTILITY)" 10)

(car/bitcount "SCOTLAND&(FATIGUE|HOSTILITY)"))

How many people in Scotland are Tired and Grumpy after the referendum?

(wcar*

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

SCT 1 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 00 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0

HOSTILE 0 0 0 0 0 0 0 1 0 0 1 0 0 0 1 0FATIGUE 0 1 0 1 0 0 0 0 1 0 0 0 1 0 0 0

BOTH 0 1 0 1 0 0 0 1 1 0 1 0 1 0 1 0

RINGSyncing

Journaling

Business logic

Incoming Data

RINGSyncing

Journaling

Business logic

Incoming Data

adamwynne/twitter-api

RINGSyncing

Journaling

Business logic

Incoming Data

clojurewerkz/meltdown

LMAX-Exchange/disruptor

RINGSyncing

Journaling

Business logic

Incoming Data

dakrone/clojure-opennlp

“I’m loving #XConfMCR! ”

Positive Affect: enthusiastic, active, alert !

Negative Affect: subjective distress !

Emerged as distinctive, orthogonal dimensions

2335345221

1 3 5 9 10 12 14 16 17 19

2 4 6 7 8 11 13 15 18 20

233534

5 22 1

1 3 5 9 10 12 14 16 17 19

2 4 6 7 8 11 13 15 18 20

positive affect! negative affect

PANAS-xGeneral Dimension Scales!Negative Affect (10) Positive Affect (10)

Basic Positive Emotions Scales!Joviality (8) Self-assurance (6) Attentiveness (4)

Basic Negative Emotions Scales!Fear (6) Hostility (6) Guilt (6) Sadness (5)

Other Affective States!Shyness (4) Fatigue (4) Serenity (3) Surprise (3)

PANAS-t

Accounts for bias on social media!

Outlines sanitisation

Validate against 10 real events

Sanitisation

• Exclude media/advertising/spam

• Account for text speak

• Account for smileys & emoj’s

• Word stemming (or lemmatisation)

• Part of Speech Tagging (POS)

GetRichQuick.com

dakrone/clojure-opennlp

SHYNESS FATIGUE

SERENITY SURPRISE

FEAR HOSTILITY

GUILT SADNESS

JOVIALITY SELF ASSURANCE ATTENTIVENESS

We have sentiment key

RINGSyncing

Journaling

Business logic

Incoming Data

mbostock/d3

[-75.14310264, 40.05701649]

Reverse Geocoding Issues• Don’t want external services

• Don’t want heavy IO

• Don’t want round trips to the database

• Accuracy not too much of a concern

Equirectangular Projection

Courtesy of Eric Gaba (Wikimedia Commons User: Sting)

x = λcosφ1 y = φ

Tissot's indicatrix of deformation

Courtesy of Eric Gaba (Wikimedia Commons User: Sting)

x (px)

y (px)

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

[x, y] = = Scotland!

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

x (px)

y (px)

[lon, lat] = [55.86140, -4.26477]

[x, y] = [329, 601]

Server side! (JavaFX)

Mike Bostock & Nick Rabinowitz

(:require [hiccup.page :as hiccup]) (:import (javafx.scene.web WebView)) !(def scripts ["uk-data.js" "topojson.v1.min.js" "d3.v3.min.js" "geocoder.js"]) !(defn create-browser [html] (-> WebView. .getEngine (.loadContent html))) !(defn build-page [script-paths] (hiccup/html5 [:head (for [path script-paths] [:script (slurp (io/resource path))])])) !(defn coordinates->name [longitude latitude] (let [browser (-> scripts build-page create-browser)] (.executeScript browser (str "geo.decode([" longitude "," latitude "])"))))

JavaFX

Stuart Sierras

Component lib!!

JavaFX

RINGSyncing

Journaling

Business logic

Incoming Data

KirinDave/clj-time

“Thu Jun 26 14:35:57 +0000 2014”

over 86 thousand seconds in a dayover 31 million seconds in a year

mm:260620141435

HH:260620141400

Monday Tuesday Wednesday Thursday Friday Saturday Sunday

2 3 4 5 6 7 8

9 10 11 12 13 14 15

16 17 18 19 20 21 22

23 24 25 27 28 29

dd:260620140000

JAN FEB MAR APR

MAY JUL AUG

SEP OCT NOV DEC

MM:010620140000

yy:010120140000

(:require [clj-time.format :as time]) !

(def input-formatter "EE MMM d HH:mm:ss Z yyyy") (def formats { "mm:" "ddMMyyHHmm" "HH:" "ddMMyyHH00" ... }) !

(defn create-key [date [prefix formatter]] (let [out-date (time/unparse formatter date)] (str prefix out-date))) !

(defn date->keys [created_at] (let [date (time/parse input-formatter created_at)] (map #(create-key date %) formats)))

RINGSyncing

Journaling

Business logic

Incoming Data

ptaoussanis/carmine

SENTIMENT !

LOCATION !

mm:date-time HH:date-time dd:date-time MM:date-time yy:date-time

We have 11 keys in total!

(:require [taoensso.carmine :as redis]) !

(defn create-tweet-id [] (redis/incr "global:tweet.id")) !

(defn set-bits [data-keys tweet-id] (doseq [data-key data-keys] (redis/setbit data-key tweet-id 1))) !

(defn data->analytics [data-keys] (let [tweet-id (create-tweet-id)] (set-bits data-keys tweet-id)))

{ 'query-type': 'count', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }

weavejester/compojure

JSON query

/api?_query={json}

(defmulti execute :query-type) !!(defmethod execute “count" [] … logic here) !!(defmethod execute “p-affect“ [] … logic here) !!(defmethod execute “n-affect“ [] … logic here)

JSON query

/api?_query={json}

JSON query

/api?_query={json}

JSON query

/api?_query={json}

JSON query

/api?_query={json}

{ 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }

JSON query

/api?_query={json}

{ 'query-type': ‘p-affect', 'params': { 'sentiments': [ 'FATIGUE', 'HOSTILITY' ], 'locations': [ 'SCT' ] } }

Using Clojure for Sentiment Analysis of the Twittersphere

Technology