Integrating ordinal, multitask deep learning with faceted ... · E.g. Likert scales: strongly...

transcript

Integrating ordinal, multitask deep learning with faceted Rasch measurement theory

Debiased, explainable, interval measurement of hate speech

Mar. 2020

Research Team● Chris Kennedy (Lead) – Biostatistics PhD student

● Claudia von Vacano (PI) – Policy, Organizations, Measurement & Evaluation PhD

● Geoff Bacon – Linguistics PhD candidate

● Alexander Sahn – Political Science PhD candidate

● Nora Broege – Sociology PhD and post-doc at Rutgers University

And with special thanks to:

● Professors Mark Wilson & Karen Draney

○ Graduate School of Education

○ Berkeley Evaluation & Assessment Research Center

Previous BEAR Seminar talk● October 2017 - Phase 1 of this project

Scientific goals of our method1. Create an outcome variable that is precise (interval) and with

minimal bias from the humans that labeled the data

- item response theory

2. Use machine learning to predict that outcome measure in a

scalable way, also with minimal human bias, and with a clear

explanation of what led to the predicted score

- deep learning

Categorical, ordinal, and interval variables● Categorical / nominal variables

○ Variable value is a code for different qualitative labels

○ Can be seen as a way of encoding multiple mutually exclusive binary variables

○ E.g. color: red (1), blue (2), or green (3)

○ Alive: yes (1), no (0)

● Ordinal variables

○ Values have an ordering from lower to higher on some variable

○ We cannot take differences between the exact distance between values is unknown

○ E.g. Likert scales: strongly disagree (0), disagree (1), neutral (2), agree (3), strongly agree (4)

○ Or disease severity: mild (1), moderate (2), severe (3)

● Interval variables

○ Continuous variable in which differences between values are meaningful

○ I.e. magnitude or scale of units is constant across the range of the variable

○ A "ruler" that measures the location on an abstract continuum of a variable

Applicable to two types of supervised outcomes to measure1. Complex outcome variable currently measured as a

human-reviewed binary or ordinal variable for convenience,

but that could be decomposed into multiple constituent

components

2. Existing outcome variables measured as an index of multiple

components rated by human reviewers, but not yet using

item response theory

Our method applies to any human-rated data used forsupervised classification or regression

Examples: Text Examples: Images

Hate speech Radiological image review (e.g. CT severity index for acute pancreatitis)

Toxic language / bullying Grading of agricultural produce

Sentiment Satellite image rating for development

Essay grading Pornography detection

Conference abstract or article review Artist identification of paintings

Treason analysis of audio transcripts Microscopy analysis of liver biopsy

Also: time-series, like ECG classification. Other ideas from you?

How does our method work? Details to be described● The core task is to decompose a single-question outcome (e.g. "Is this comment hate

speech?") into a series (say 5) of ordinal components (respect, dehumanization, insult,

● Recruit human labelers to review observations on those components (online survey)

○ Batches of comments should be created in an overlapping fashion so that the labelers are densely linked in

a single network

● Apply item response theory to aggregate those components into a continuous scale

○ Simultaneously estimate the bias of each labeler and eliminate its influence from the scale

○ Estimate the randomness in each labeler and remove labelers with inconsistent labels

● Use deep learning in a multitask architecture to predict each component (ordinal

classification) using the human labeled data, also incorporating the bias of labelers

○ The deep learning component predictions are then transformed to a continuous scale through IRT

● The result is a debiased, explainable, efficient prediction machine for measuring the

construct of interest on a continuous, interval scale (with std. errors)

Standard machine learning approach● binary definition of hate speech (yes or no) - qualitative● probability prediction: Pr(Y = 1 | X)● no sense of magnitude: how extreme is the hate speech?● biased by the interpretation of the humans that labeled data● no explanation● not generalizable to future time periods when our sensitivity to

hate speech may change

Machine learning model

[Comment someone makes on Twitter]

Hey AI, is that comment hate speech?

Research team, social media platform, or judge/jury

I estimate 37% probability of being hate

speech.

Standard approach is limited, and not considered measurement

New approach:● continuous hate speech scale (roughly -4.0 to +4.0)● magnitude is incorporated - true quantitative measurement● regression prediction: E[Y | X]● prediction can be explained by intermediate components● debiased from how humans labeled the data● generalizes beyond the specific components measured, comments

analyzed, and raters who labeled data

Our machine learning model

[Comment someone makes on Twitter]

Hey AI, where do you place this comment on your hate speech scale?

Research team, social media platform, or judge/jury

I estimate the comment at 2.5 (+/- 0.3) on the hate speech scale - an extremely hateful comment. My reasoning is that this comment appears to have strongly negative sentiment (75% certainty), likely threatens violence (85% certainty), includes an identity group target (99%), and is likely humiliating to the target group (92%).

Our method measures hate speech as an interval variable, and explains why

Review our scientific contribution● We develop a method for integrating the measurement benefits of item response

theory with the scalable, accurate prediction provided by deep learning

● Our method makes five contributions:

○ Realistic granularity: Outcomes can be measured as interval variables on a continuous scale, rather

than simplistic yes/no binaries

○ Labeler debiasing: we estimate the first-order bias of individual labelers (a "fixed effect") and

eliminate that bias from the estimation of the continuous outcome

○ Sample efficiency: we can achieve greater predictive accuracy for a given sample size because our

ordinal components become supervised latent variables in a multitask neural architecture

○ Explainability: we can explain the predicted continuous score of any observation by examining the

predictions of the individual components

○ Labeler quality: we show how item response theory can estimate the quality of labelers' responses,

allowing low-quality labelers to be removed or down-weighted

● In sum, our method stands to drastically change how we measure outcomes and

conduct machine learning in big data

Agenda for Talk1. Theorize construct (reference set, components)

2. Collect comments (web APIs)

3. Label components (crowdsourcing)

4. Scale (faceted Rasch IRT)

5. Predict (deep learning for NLP)

Theory development

● EDUC 274A (K. Draney) - Fundamentals of Measurement

● EDUC 274B (M. Wilson) - Statistics of IRT

Construct Map: theoretical levels of hate speech

Qualitative ordered value, does not reflect an

interval value on the final hate speech scale

Reference set: empirical grounding of theory● 10+ comments for each of our theorized levels

● Forms an empirical lattice that constrains the theory

● Prompts introspection and debate, leading to improved

understanding of how we truly theory our construct and its

associated levels

● Leads to confirmatory analysis, not exploratory

Components of hate speech

Survey details● Initial screen on identity group targets

○ Major identity groups: race/ethnicity, gender, religion, sexual orientation, disability, age

○ One follow-up question for sub-identity group for each major group

● Hate speech scale questions (~10)

● Participant demographics

○ Gender, education, race, age, income, religion, sexual orientation, political ideology

● Free response feedback (optional)

Comment Collection

Stream comments

Reddit: Most recently published comments on any post in /r/all.

Twitter: Most recent tweets from their streaming API.

YouTube: Search for videos around major US cities, take all

comments on them.

Class imbalance, statistical power, & budget limits

● Binarized hate speech is < 1% of general internet content

● If we had a yes/no outcome for hate speech, what hate speech

proportions would we prefer the training data?

● For a 8-level hate speech construct, we want a mostly even

distribution over each level (~12.5% each)

● Our labeling budget is finite, so we want to avoid spending a

ton of money on imbalanced training data

Sample commentsWe’ve collected over 75 million comments, but we only want to annotate

Over-sample comments with identity groups, and stratify on estimated

hatefulness.

20k 20k 10k

Comment batch creation

Augment comments

Perspective API: Trained NLP models from Jigsaw for detecting

various kinds of abusive language. We use their identity attack and

threat models.

Word embeddings help us answer “How relevant is this comment to

the identity groups we’re looking for?”

Bin commentsWe use the metadata added from step 2 to bin the comments into

5 bins:

- Not relevant (does not appear to target identity groups)

- Relevant and low on hate scale

- Relevant and neutral on hate scale

- Relevant and high

- Relevant and very high

Stratification: maximize power without eliminating any cells

Positive Neutral Low Hate High Hate

Identity groups 7,500 5,000 18,300 14,200

No identity groups 5,000

Hypothesis dimension: E[ hate score | X ]

Relevance dimension:Pr[ identity groups = 1 | X ]

Total labeling budget: 50,000 comments

Comments downloaded: 75 million

Sampling design for human review of comments

Naive annotation plan can lead to distinct networks with disjoint subsets● Batches of 5 distinct comments● Each batch rated by 3 labelers● Each labeler rates only one batch● We cannot differentiate if a batch is more hateful or a set of raters is

more lenient in their rating - we can't calibrate across batches

Batch 1 Batch 2 Batch 3

R 1 R 2 R 3 R 4 R 5 R 6 R 7 R 8 R 9

● Allowing workers to label comments randomly, like on Figure 8's system, would likely also lead to disjoint subsets○ But maybe one could get lucky and not have any disjoint subsets?

Overlapping reviews lead to a single linked network of raters + comments

Rater A

Rater B

Rater C

Rater D

Rater E

2 3 4 5 6 7Comments

Labelers / annotators

● Here is an example with 7 comments reviewed by 5 raters. Every rater reviews 3 comments● Each review creates a link (or connection, edge) between the rater and the comment.

Unfolded version of the same network

Densely linked network for human labeler debiasing

Scaling

Overview of item response theory scaling● Item response theory analyzes the patterns in the ordinal survey responses

(components of hate speech) to create a continuous latent variable (hate speech scale)

● That continuous hate speech score best explains the combined ratings on the survey

instrument for each comment, after correcting for reviewer bias.

● While doing that, IRT simultaneously estimates:

○ Where each survey item falls on the hate speech scale (where it is most informative)

○ Where each response option for each item falls on the hate speech scale

○ The bias (or "severity") of each annotator

● This estimation is through maximum likelihood

○ We use joint maximum likelihood, but marginal or conditional maximum likelihood are options

● It provides statistical diagnostics to evaluate the results

○ Reliability is the primary metric, ranging from 0 to 1. Our scale has a reliability of 0.94.

■ Interpretation: similar to R

, it is proportion of variance accounted for by the model

○ It also generates fit statistics for each reviewer, which can identify reviewers who are selecting randomly

○ Fit statistics for each survey item tell us how well the item fits into the scale

● Readings: Wilson (2004) Constructing Measures (Ch. 5 - 7), Wright & Masters (1982) Rating Scale Analysis

Item response theory estimation goal (slightly simplified)

Predict probability of response option R on item I for comment C by annotator A

Based on the subtraction formula:

hate score for comment C

- hate score for item I

- annotator A's bias (aka severity)

- hate score for response option R

See formula 1 in manuscript

for the more technical version

Fixed effect terms

Latent variable of interest (random effect)

Estimation methods for IRT(Add in highlights on JML, MML, CML, non-parametric)

Scaling results from item response theory

Most hateful

Somewhat hateful

Neutral

Counterspeech

Supportive

Very hateful

Reliability: 0.94!

Item fit statistics

Example scaling results (trigger warning)

Crowdsource worker quality analysis

Crowdsource worker quality: identity rate

Worker quality: mean-squared statistic vs. identity rate

Scaled reference set - initial

Scaled reference set - revised

Estimating thresholds for theorized levels

Distribution across social media platforms

Comparison to single binary hate item

We have created a measure of our construct.

Can we predict it with machine learning on raw text?

With robots, if possible.

Short Circuit (1986)

Fully connected

layers

Raw comment text

Binary hate speech status

Deep NLP(BERT,

ULMFiT, GPT)

Language representation

Latent variables related to hate speech

Current best practice in supervised NLP

Comparison to related work

Fully connected

hidden layers

Raw comment text

Intermediate ordinal outcomes(ratings on hate scale items)

1. Sentiment

2. Respect

8. Genocide

7. Violence

9. Attack-Defend

Continuous hate score

Item Response

Theory

Estimated labeler bias (“fixed effect”)

Deep NLP(USE,

XLNet, RoBERTa, ULMFiT)

Learning to rate

Neural architecture for predicting a continuous score with multiple intermediate outcomes, labeler bias adjustment, and IRT activation

Loss: quadratic weighted kappa

Loss: squared-error

3. Insult

4. Humiliate

5. Status

6. Dehumanize

Final outcome

Non-linear activation function

Correlation of items suggests benefit from multitask approach

Ordinal classification with labeler bias adjustment

Final hidden layer

Output: Violence Item

Wording: "This comment calls for using violence against the group(s) you previously identified. "

1. Strongly disagree

2. Disagree

3. Neutral

4. Agree

5. Strongly agree

Estimated labeler bias (“fixed effect”) - concatenated onto the final hidden layer

Predicted probabilities using only text (no bias adjustment)

Predicted probabilities with bias adjustment

Proportional Odds Latent

Variable

(See Vargas et al. 2019 Deep ordinal classification)

Quadratic weighted kappa loss: cost matrixPredicted

Actual

Quadratic weighted kappa example:

Predicted Prob 12% 18% 35% 20% 15%

Distance 1 0 1 2 3

Weight 0.0625 0.0625 0.25 0.5625

Loss contribution 0.0075 0 0.02187 0.05 0.08438= 0.16375

Compare to NLL:

-log(0.18) = 1.715

Labeler bias as an auxiliary input● During deep learning, each observation (comment text plus the set of ratings for a

given comment) will have the estimated labeler bias (severity) as an auxiliary input

● Labeler bias is a value on the hate speech scale: centered around 0 and within (-3, +3)

● We include this scalar value as another latent variable in the final hidden layer

● Those values are then inputs into the latent hidden value for each item's ordinal

prediction

○ (Which is evaluated with quadratic weighted kappa loss)

● The effect of the bias input is that the neural network can adjust its probability

predictions for each item based on whether the rater for that observation was more or

less severe.

○ Ex.: based on the text of a comment, the network might predict "strongly agree" for the genocide item

○ But if it knows the rater is severe, it should shift its prediction down, e.g. to "agree" or even "disagree"

Categorical classification with labeler bias adjustment

Final hidden layer

Loss: categorical cross-entropy

2. Disagree

3. Neutral

4. Agree

5. Strongly agree

ax activation

Predicted probabilities using only text (no bias adjustment)

Predicted probabilities with bias adjustment

Statistics of ordinal classification

(Add in some here)

Current results (work in progress)

Future work● Partnerships - interest from Facebook, Pinterest, Blizzard, et al.

● Causal inference (interrupted time series, randomized interventions, user accounts)

● Listening to victims: collect stories and experiences of hate speech

● Focus on genocide in developing countries (Sri Lanka, Myanmar, India, Brazil)

● Improved labeling: incorporate message context

● New platforms: Facebook, Instagram, Wikipedia, game chats (Blizzard)

● New languages

● New constructs: toxicity

● New data types: images, audio, video

● Other applications: automated essay grading, etc.

● Exploring a possible patent application

Concluding inspirational quotation

Comments, questions, feedback?

hatespeech.berkeley.edu

ck37@berkeley.educvacano@berkeley.edu

Appendix

Implementation diagram

Technical implementation: Google serverless functions

Labeling instrument (Qualtrics)

Rater recruitment (Amazon Mechanical Turk)

Google Cloud

SQL Database

Comment Batches

Reserve comment batch

Ratings

Complete comment batch

Serverless functions pool

Fully connected

hidden layers

Raw comment text

Deep NLP(USE,

Learning to rate

Labeler bias as auxiliary input (violence item)

2. Disagree

3. Neutral

4. Agree

5. Strongly agree

Final hidden layer

Proportional Odds Latent

Variable

Fully connected

hidden layers

Raw comment text

Deep NLP(USE,

Learning to rate

Labeler bias as auxiliary input (violence item)

Loss: categorical cross-entropy

2. Disagree

3. Neutral

4. Agree

5. Strongly agree

Final hidden layer

ax activation

Integrating ordinal, multitask deep learning with faceted ... · E.g. Likert scales: strongly...

Documents