Text as Data
Brandon M. Stewart1
Department of Government, Harvard University
Camp Resources XX, August 5, 2013
1Thanks to Gary King, Justin Grimmer, Rich Nielsen and Molly Roberts for permission to include figures here.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 1 / 38
Overview
Two points and an application.
1 Large bodies of text can provide a new source of data for socialscience research.
2 Big data isn’t about the data (it’s about the methods).
3 Application to censorship and media control in China.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 2 / 38
Overview
Two points and an application.
1 Large bodies of text can provide a new source of data for socialscience research.
2 Big data isn’t about the data (it’s about the methods).
3 Application to censorship and media control in China.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 2 / 38
Overview
Two points and an application.
1 Large bodies of text can provide a new source of data for socialscience research.
2 Big data isn’t about the data (it’s about the methods).
3 Application to censorship and media control in China.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 2 / 38
Overview
Two points and an application.
1 Large bodies of text can provide a new source of data for socialscience research.
2 Big data isn’t about the data (it’s about the methods).
3 Application to censorship and media control in China.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 2 / 38
Overview
Two points and an application.
1 Large bodies of text can provide a new source of data for socialscience research.
2 Big data isn’t about the data (it’s about the methods).
3 Application to censorship and media control in China.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 2 / 38
References
Grimmer Justin and Brandon M. Stewart. (2013) “Text as Data: ThePromise and Pitfalls of Automatic Content Analysis Methods for PoliticalTexts.” Political Analysis.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 3 / 38
Overview
1 Text as Data
2 Big Data isn’t about Data
3 Information Control in China
4 Conclusion
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 4 / 38
Text as Data
Massive quantities of unstructured text are increasingly available and openup new possibilities.
1 More Systematic and Replicable Data collection
2 Cheaper to collect!
3 New Quantities of Interest
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 5 / 38
Text as Data
Massive quantities of unstructured text are increasingly available and openup new possibilities.
1 More Systematic and Replicable Data collection
2 Cheaper to collect!
3 New Quantities of Interest
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 5 / 38
Text as Data
Massive quantities of unstructured text are increasingly available and openup new possibilities.
1 More Systematic and Replicable Data collection
2 Cheaper to collect!
3 New Quantities of Interest
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 5 / 38
Text as Data
Massive quantities of unstructured text are increasingly available and openup new possibilities.
1 More Systematic and Replicable Data collection
2 Cheaper to collect!
3 New Quantities of Interest
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 5 / 38
What Changed?
Massive increase in availability of unstructured text (10 minutes ofworldwide email = 1 LOC )
Cheap storage: 1956: $10,000 megabyte. 2011: <<< $0.0001 permegabyte (Unless you’re sending an SMS)
Explosion in methods and programs to analyze texts
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 6 / 38
What Changed?
Massive increase in availability of unstructured text (10 minutes ofworldwide email = 1 LOC )
Cheap storage: 1956: $10,000 megabyte. 2011: <<< $0.0001 permegabyte (Unless you’re sending an SMS)
Explosion in methods and programs to analyze texts
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 6 / 38
What Changed?
Massive increase in availability of unstructured text (10 minutes ofworldwide email = 1 LOC )
Cheap storage: 1956: $10,000 megabyte. 2011: <<< $0.0001 permegabyte (Unless you’re sending an SMS)
Explosion in methods and programs to analyze texts
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 6 / 38
What Can Text Methods Do?
Haystack metaphor:
Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
What Can Text Methods Do?
Haystack metaphor:Improve Reading
- Interpreting the meaning of a sentence or phrase Analyzing a strawof hay
- Humans: amazing (Straussian political theory, analysis of Englishpoetry)
- Computers: struggle
- Comparing, Organizing, and Classifying Texts Organizing hay stack
- Humans: terrible. Tiny active memories- Computers: amazing and getting better all the time!
What They Don’t Do:
- Develop a comprehensive statistical model of language
- Replace the need to read
- Develop a single tool + evaluation for all tasks
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 7 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
How to Use Text as Data
A simple recipe for automated content analysis:
1 Define the Problem
2 Find a Text Corpus
3 Analyze with an Appropriate Model
4 Validate and Visualize
But seeing is believing...
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 8 / 38
Islamic Clerics and Jihad (Nielsen)Why do some Islamic Clerics support militant Jihad?
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●●
● ●
●
●●
●
●●
●
●
●●
●
●
●
●●
●●
●
●
●
●●●
● ●
●●
●
●
●
● ●
●
● ●●●
●
●●
●
●
●●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●
● ●
●
●●● ●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
● ●
●●
● ●
●
●
●●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●●
●●
●
● ●●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●
●●
●●
●●●
●
●
●
●
●
●●
●
●
●
●●
● ●●
● ●
●
●●
●●
●
●●
●●
●
●
●
●
●●
●
●
●
● ●●
●
● ●
●
● ●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●● ●
●
●
●●
●●
●
●●
●
●
●●
●● ●
●●
●
●
●●
●
●
●
●
● ● ●
●●
●●●
●
●
●
●
●
●
●●
●
● ●
● ●●
●●
● ●
●
●
● ●
●
●
●
● ●● ●
●●
●
●●
●
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 9 / 38
Islamic Clerics and Jihad (Nielsen)
−0.15 −0.1 −0.05 0 0.05Jihad Score
Histogram of cleric Jihad scores
Ruling on Fighting Now in Palestine and Afghanistan. The foregoing has clarified that if an inch of Muslim lands are attacked, then Jihad is obligatory for the people of that area,and those near by. If they do not succeed or are incapable or lazy, the individual obligation widens to those behind them and then gradually the individual obligation expands until it is general for the whole land, from East to West. (Abdallah Azzam)
If a person arrives while the Imam is preaching at Friday prayers, he should pray two brief prostrationsand sit without greeting anyone asgreeting people in this circumstanceis forbidden because the Prophet, peace be upon him, says, "If your friend speaks to you during the Friday prayers, silence him while the Imam preaches because it is idle talk." (Ibn Uthaymeen)
There is a fundamental fact about the nature of this religion and the way it works in people's lives. A fundamental, simple fact, but although it is simple, it is often forgotten or not realized at all. Forgetting this fact, or failing to recognize it arisesfrom a serious omission from views of this religion: its truthfulness and historical, present, and future reality. (Sayyid Qutb)
Abdallah Azzam
Sayyid Qutb
Usama bin Laden
Ibn UthaymeenIbn Baz
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 10 / 38
Islamic Clerics and Jihad (Nielsen)
Afraid of
Narrow
Is believed
Fear
Secondly
Know
Remains
We ask
And here
Thirdly
Effort
Displays
Reported
Other
Summary
Accept
Does it
Spirit
Fidelity
Anger
Pan
Creature
Extent
Several
Updated
Completely
Under no
Street
Complex
Indeed
Differentiate
Moses
Send
For some
I got
After them
Ideas
Be taken
Get
Without
Descent
Likely
Out
One
Position
Evil
Result
Text
Signed
The Creator
Penalty
Deny
Shows
Underneath
Body
Aware of
Blessed
Aslam
I learned
Particularly
Him
Literature
Responsible
And even
Harm
Approved
Cattle
Benefit
And more
Alice
Survival
Sir
A lot
Improves
The seventh
Packages
End
Visualization
Prove
Was
The body
Use
Branches
Defeated
Take
Opportunity
Says
Money
Great
Measurement
Task
Make
Supreme
Explanation
Transferred
Amr
Toxic
Quasi−
Risk
Trees
Patience
Proof
Heart
Before
I bear witness
Writes
Excuse
Poverty
Virgin
Actually
Punk
Story
Torment
Left
Determination
Speak
Follow
Be
Long
So−called
For himself
Conditions
On
Prophet
Together
Display
Are not
Beliefs
Never
Guidance
Still
Witness
Check
Blocker
We know
Called
Word
In
First
King
Murderer
Detail
Lord
I said
Abe
Explain
Option
Work
Said
High
Death
A large
Believe in
Comprehensive
One of them
At least
Followed by
Orientation
We have
Jurisprudential
Understand
Necessity
Since
Example
Roads
Section
Violates the
Issuance of
Far
The eyes
Fourth
Social
Ray
Originally
You want
Certainty
Solomon
Neighborhoods
Forgive
See
Never
Ordered them
Contained
All
Re−
News
And you
For these
Friend
Love
Labbe
Call
State
Add
Walking
Hurry
Degree
Qadir
Insults
Hundred
Denied
A particular
Evil
Not
Becomes
Bone
Received
Long as
Senior
Aversion
Staff
Warning
Adults
You
Inn
It
Follow
Sin
Some
As the
Is
Lord
Provides
As well as
Wine
Both
Has become
Appearance
Demonstration
A cause
Huda
Minds
Nuclear
Means
Whenever
Qadeer
Beginning
Reality
Your
Attempt
Companions
We
Corresponding to
According to
The lowest
Order the
Address
Stand
Allah will
By virtue of
Show
Supreme
Level
Gathering
The effects of
Intensity
Change
Side
The hands
So
You
Most of them
The
See
Acknowledges
Israel
Stands
Application
God
Edition
Know
Perhaps
Show
Needs
We
What
Report
Conflict
Field
His statement
The Messenger
Capital
Us
Years
Something
According to
Exposure
Just
Hussein
New
Faces
Show
To be
Versus
Obedience
Claim
A way
Reality
See
Fancy
Significant
Few
Sub−
Discourse
Muhammad
Recognition
Safe
Their hands
Badr
On behalf of
Yourselves
Sea
Other
Phrase
The absolute
Secretariat
Agreement
Away
A people
Revelation
Cities
His brothers
Raina
To differentiate
Are
Which
Denial
Within
The soul
Face
Check
The facts
Control
Partner
Alpha
Mansoura
Were it not for
Nullity
Imams
Certificate
They said
Group
According to
Concept
Balance
Will not
Some
Pharaoh
Issued
Consideration
We want
Economic
While
Wi
This
And all
Ourselves
Is
Other than
Student
Deliverance
With
Aspects of
Line
Confrontation
List
Condition
Judge
Do
They wanted
Wonder
BooksPlaces
Damage
And they
Issues
Shown
Owner
Scourge
Find
Serious
Evil
Call
Corrupt
Very
Worship
Malignant
He
Lying
Ethics
Iron
Think
Tongue
Fancies
One
IgnoranceSaid
You
Distinguished
Opposition
Denial
The apparent
Side
Powers
How
Important
Party
Hell
The doors
Advanced
Examples
Ill
Us
Hopes
Presence
His faith
At least
Our
Put
Still
Hearts
How
Post
Kinds of
Target
View
Can
Worship
Live
Sheikhs
IncludingMeanings
Foot
Stray
Was
Wright
All
Mr.
Spoiler
Court
Significance
Issued
The following
Minute
OneAcceptance
Spring
Sadiq
Shadow
Else
Owners
Article
Champions
Claimed
Including
Their hearts
Symptoms
Religion
Saying
Board
Time
Sons
Make
Interest
Calling
Like
Display
Claim
Their money
Probably
You
Ibrahim
Him
Quraish
Impeller
Martyrs
YearPrevious
We
Team
Spread
Predecessor
Open
Trial
Many
Minimum
I say
Intent
Announcement
Community
Disclosure
Legislation
Hit
Some of them
Prophets
Dear
Justice
Life
Rules
We
Humiliation
We find
ZoneSend
Hypocrisy
That
Wahab
Events
Seemed
The above
Protection
Scientists
Senate
Prince
Like
Positions
Look
You
IssueReason
The most
History
Contemporary
Regardless
Obaid
Difference
United
To order
Sham
ImmigrationShan
Hostility
We say
We
Really
Thought
Sword
Appeal
Taymiyah
All
Stage
Satisfaction
Unbelievers
Delusion
Position
All
Border
Method
A
Kharijites
Head
The most important
Shirk
Parties
How
Pain
Aba
Leadership
Infidel
Violation
Rightly
Say
Parents
He wants
Movement
Explicitly
The belief
Words
Muslim
Invasion
For
Description
Supports the
Defense
Corruption
WhateverRights
Soldiers
They
Frank
Consensus
Transfer
We
Numbers
Without
But
Progress
Yard
Roman
Them
Evidence
Man
Exegesis
Alert
Article
While
Blood
Wise
Investigation
Issues
Page
Loyalty
Joseph
Sect
Martyr
Disbelief
Rahim
Mustafa
Systems
Must
The battle
RahmanSoldiers
Issue
Time
Which
Information
The provisions of
HypocritesLaw
And others
Believe
Where
Doctrine
Christianity
Military
In the name of
Seer
Crusade
The island
Will
Our brothers
Disbelief
All
Was
Era
Pregnancy
Their religion
Apostles
They said
Arms
Interests
Those
Message
West
Assets
Young people
Fighting
Texts
Great
House
Al Qaeda
Fight
I
Uh
Sharia
Themselves
And between
Alone
Loyalty
Known
Who
People
Supporters
The greatest
Were
Fact
Sultan
Army
Governing
Preachers
Criminals
Word
Kill
Milli
I say
Mr.
Order
Response
People
I
More
Party
Arabs
Unitarian
Freedom
Laws
Security
Brother
Platform
Political
Faith
Palestine
Brotherhood
Falsehood
Governor
Legislation
All
Apostatize
Ignorance
Injustice
Worship
Iraq
The infidels
AfghanistanTerrorism
People
Government
Statement
Down
The enemies
The faithfulAgainst
And Enmity
Judgements
Apostates
Maqdisi
Idolaters
Prison
Fighting
Without
Oppressors
Country
GodAmerica
JewsForce
War
LandWay
Victory
Enemy
American
Excommunication
Killing
The worldIdolatry
The nationState
Idolators
Infidel
Monotheism
Mujahideen
Apostasy
Jihad The
Are
Zaid
Like
Newly
Directly
My nation
The
You
Legalization
Wants to
The
The
News
Prove
Opinion
Search
The nearest
To the son of
Human
By saying
Accept
I like
Consequent
We know
Rest
Indicates
Spoke
Strange
And how much
Not
Present
Bammer
Valley
Prove
Weakness
Dies
The old
Ray
Log in
ValuesIsaac
Estimated
Staff
Stronger
Century
Revolution
Requires
Known
Lord
Contrary to
Decision
Delivery
Assesses
After
Down
Eye
And possibly
Custom
Individual
Near
Series
Is
With
Fired
Tracking
Either
By himself
Obstacle
To the people
Line
Doctrines
Rejection
Reform
Tender
Detection
More
A way
Was
Once the
Stone
Should be
Deficit
Center
The sixth
Companions
And it
Type
Acts
PenaltyChildren
Skies
Bear
His hand
To reconcile
One thousandth
Follow the
Across
Jurisprudence
Fit
NotA
Hamid
Comes
Puts
Guides
Wise
Difference
Intervention
Easy
The fourth
Nine
Yemen
Raises
Sins
Remember
Rectum
As
Maintain
Del
EBay
Religiously
And other
Ruled that
Impose
Is
Before
The sky
Charity
Permanent
I see
Aas
Promise
Resurrection
Enlarge
Number
Evidence
Enough
LaSalle
Always
Serves as
Gives
Nature
Have
Found
And
See
One day
The basis of
Different
Being
Can
Mahmoud
Past
Ezz
Back
Sex
Pursuant to
Thousands of
Advice
Massoud
Confidence
Has become
Conditions
Was
Estimate
And piety
A little
I took
Two
Prevent
Strong
Number
Arrived
The large number of
V
Hatred
Circle
Differ
Forms
Caution
Necessary
Money
Increase
Author
Twice
Role
So and so
Do
Was
Desire
More
Roy
Except
Selection
The
Has become
The contrary
Remain
Ashraf
Provide
Between them
Provided
Intended
The famous
Doubt
As
The pretext of
Salah
Latecomers
Place
Hanifa
Presence
Hearing
Want to
Specific
Concern
Yourself
Families
At
Dhar
Can
Knowing
Drink
Asked
Palm
Show
There are
Wide
As
Willing
Never
Fear
Keeper
Names
Different
Study
Members
The best
Gel
Such as
Did
So
LeavesLearning
Reasons
Shed
Whole
Sayings
Adam
This
Hear
In other words
For
Lack
Right
Diligence
Sharp
Table
Light
Came
Optical
Man
I
Sharif
Asked
Terms
Then
If
Bad
Useful
His life
Mark
Seen
Halt
OneNo
About us
Exchange
Separation
Hanbal
And followers
Take
I heard
Kara
Comment
Find
Age
Because
Dispute
To be
The following
Ring
Cut
Livelihood
Agreed
Hidden
Delivers
Education
Perfection
Possible
Obtained
Certain
Paper
Angels
Foreign
Fire
For non−
Need
Perhaps the
Required
Interest
Saad
Years
Comes
Found
Contract
To imitate
Get
Science
Income
Quest
The establishment of
Other
Pay
A means
Has
Pay
Land
Leave
Satisfaction
Very
Desires
First
Often
Appropriate
The unseen
Knowledge
Treatment
Sin
Not
Request
The said
Account
Uncle
Education
Occurs
Combines
The
City
Great
Yam
Scholars
Jazz
Works
Service
Results
Tell
Love
Sentence
Kaaba
Lead
Close
Must
Foot
Back
Disposal
Wife
Egypt
Hit
When
And messengers
Must
Osman
Required
Grandfather
Said
Deal
Fixed
Language
Questions
Almighty
Suspicion
Out
Business
The day
Meaning
Full
The public
Virtue
Is called
Kitten
Will
See
A
Is located
Read
I
Roy
Up
Shows
Entry
Nuha
Sometimes
Status
Makes
Duty
The mind
I know
Six
Something
Feast
Preferred
HappyMeans
Character
Matt
Sins
Seven
Event
Should be
Question
Shafei
Recent
Means
In
And so on
Out
Trade
Hand
Graduation
Proven
Back
Whether
When
Relationship
Solution
Behind
Project
Problem
CameGo
Neighborhood
Provided that
Around him
Take
Re−
Period
Solve the
Blood
Abbas
Part
Adultery
Nasser
Affect
Prayers
Four
The time
Months
Return
End
Things
He wanted
And only
Calls for
Sin
Country
Not
Aisha
Guidance
Good
Parental
Prophecy
Is
Well
Clear
Word
Prevents
And then
Of religion
Out
The third
Things
Do
Of
Pronunciation
Link
The intended
Good
Five
Born
The original
Name
Wage
Passport
Increase
Likely
Men
Time
Is
Small
Travel
I hope
Innovation
And Prevention
Doctrine
The answer
Majah
Non−
Women
Children
Permissible
Create
Khaled
Satan
Full
True
Sunan
Ban
Old
Enters
DoorMorocco
Existing
Directed by
And because
To be
And blessings
Ask
WillBack
Thread
Raise
First
Especially
What
Cow
Messages
Episode
During the
Secrecy
About
Speech
Curse
Tirmidhi
True
Almighty
Note
Voice
UsurySecond
Intention
Save
Sun
Truth
Paradise
Period
Wear
Ras
I want
Half
Building
Young
Answer
Amount
But
GoldTotal
Owner
For
Mosques
Disease
Necessary
Repentance
For example
Food
Be
Days
Place
Weak
Back
You
Eating
Daybreak
DeprivedRoad
Directed by
Links
Valid
Dead
Do
And companions
Eg
Chatter
Al
David
Instinct
Ahmed
Hand
Good
Night
Elimination
For saying
Image
Month
Sale
Honorable
Noble
Take
Hair
Place
Reading
Women
Be
Collection
Water
Three
Critical
Woman
The best
Person
Are
Sacred
Need
Fasting
Up
Quoting
Bukhari
House
Means
Yes
Forbidden
Pray
Ramadan
Bass
Ten
ManTime
MaySource
Narrated byMustAge
HumanImam
Money
Mosque
QuranHajj
Zakat
Her husbandWomen
Afraid of
Narrow
Is believed
Fear
Secondly
Know
Remains
We ask
And here
Thirdly
Effort
Displays
Reported
Other
Summary
Accept
Does it
Spirit
Fidelity
Anger
Pan
Creature
Extent
Several
Updated
Completely
Under no
Street
Complex
Indeed
Differentiate
Moses
Send
For some
I got
After them
Ideas
Be taken
Get
Without
Descent
Likely
Out
One
Position
Evil
Result
Text
Signed
The Creator
Penalty
Deny
Shows
Underneath
Body
Aware of
Blessed
Aslam
I learned
Particularly
Him
Literature
Responsible
And even
Harm
Approved
Cattle
Benefit
And more
Alice
Survival
Sir
A lot
Improves
The seventh
Packages
End
Visualization
Prove
Was
The body
Use
Branches
Defeated
Take
Opportunity
Says
Money
Great
Measurement
Task
Make
Supreme
Explanation
Transferred
Amr
Toxic
Quasi−
Risk
Trees
Patience
Proof
Heart
Before
I bear witness
Writes
Excuse
Poverty
Virgin
Actually
Punk
Story
Torment
Left
Determination
Speak
Follow
Be
Long
So−called
For himself
Conditions
On
Prophet
Together
Display
Are not
Beliefs
Never
Guidance
Still
Witness
Check
Blocker
We know
Called
Word
In
First
King
Murderer
Detail
Lord
I said
Abe
Explain
Option
Work
Said
High
Death
A large
Believe in
Comprehensive
One of them
At least
Followed by
Orientation
We have
Jurisprudential
Understand
Necessity
Since
Example
Roads
Section
Violates the
Issuance of
Far
The eyes
Fourth
Social
Ray
Originally
You want
Certainty
Solomon
Neighborhoods
Forgive
See
Never
Ordered them
Contained
All
Re−
News
And you
For these
Friend
Love
Labbe
Call
State
Add
Walking
Hurry
Degree
Qadir
Insults
Hundred
Denied
A particular
Evil
Not
Becomes
Bone
Received
Long as
Senior
Aversion
Staff
Warning
Adults
You
Inn
It
Follow
Sin
Some
As the
Is
Lord
Provides
As well as
Wine
Both
Has become
Appearance
Demonstration
A cause
Huda
Minds
Nuclear
Means
Whenever
Qadeer
Beginning
Reality
Your
Attempt
Companions
We
Corresponding to
According to
The lowest
Order the
Address
Stand
Allah will
By virtue of
Show
Supreme
Level
Gathering
The effects of
Intensity
Change
Side
The hands
So
You
Most of them
The
See
Acknowledges
Israel
Stands
Application
God
Edition
Know
Perhaps
Show
Needs
We
What
Report
Conflict
Field
His statement
The Messenger
Capital
Us
Years
Something
According to
Exposure
Just
Hussein
New
Faces
Show
To be
Versus
Obedience
Claim
A way
Reality
See
Fancy
Significant
Few
Sub−
Discourse
Muhammad
Recognition
Safe
Their hands
Badr
On behalf of
Yourselves
Sea
Other
Phrase
The absolute
Secretariat
Agreement
Away
A people
Revelation
Cities
His brothers
Raina
To differentiate
Are
Which
Denial
Within
The soul
Face
Check
The facts
Control
Partner
Alpha
Mansoura
Were it not for
Nullity
Imams
Certificate
They said
Group
According to
Concept
Balance
Will not
Some
Pharaoh
Issued
Consideration
We want
Economic
While
Wi
This
And all
Ourselves
Is
Other than
Student
Deliverance
With
Aspects of
Line
Confrontation
List
Condition
Judge
Do
They wanted
Wonder
BooksPlaces
Damage
And they
Issues
Shown
Owner
Scourge
Find
Serious
Evil
Call
Corrupt
Very
Worship
Malignant
He
Lying
Ethics
Iron
Think
Tongue
Fancies
One
IgnoranceSaid
You
Distinguished
Opposition
Denial
The apparent
Side
Powers
How
Important
Party
Hell
The doors
Advanced
Examples
Ill
Us
Hopes
Presence
His faith
At least
Our
Put
Still
Hearts
How
Post
Kinds of
Target
View
Can
Worship
Live
Sheikhs
IncludingMeanings
Foot
Stray
Was
Wright
All
Mr.
Spoiler
Court
Significance
Issued
The following
Minute
OneAcceptance
Spring
Sadiq
Shadow
Else
Owners
Article
Champions
Claimed
Including
Their hearts
Symptoms
Religion
Saying
Board
Time
Sons
Make
Interest
Calling
Like
Display
Claim
Their money
Probably
You
Ibrahim
Him
Quraish
Impeller
Martyrs
YearPrevious
We
Team
Spread
Predecessor
Open
Trial
Many
Minimum
I say
Intent
Announcement
Community
Disclosure
Legislation
Hit
Some of them
Prophets
Dear
Justice
Life
Rules
We
Humiliation
We find
ZoneSend
Hypocrisy
That
Wahab
Events
Seemed
The above
Protection
Scientists
Senate
Prince
Like
Positions
Look
You
IssueReason
The most
History
Contemporary
Regardless
Obaid
Difference
United
To order
Sham
ImmigrationShan
Hostility
We say
We
Really
Thought
Sword
Appeal
Taymiyah
All
Stage
Satisfaction
Unbelievers
Delusion
Position
All
Border
Method
A
Kharijites
Head
The most important
Shirk
Parties
How
Pain
Aba
Leadership
Infidel
Violation
Rightly
Say
Parents
He wants
Movement
Explicitly
The belief
Words
Muslim
Invasion
For
Description
Supports the
Defense
Corruption
WhateverRights
Soldiers
They
Frank
Consensus
Transfer
We
Numbers
Without
But
Progress
Yard
Roman
Them
Evidence
Man
Exegesis
Alert
Article
While
Blood
Wise
Investigation
Issues
Page
Loyalty
Joseph
Sect
Martyr
Disbelief
Rahim
Mustafa
Systems
Must
The battle
RahmanSoldiers
Issue
Time
Which
Information
The provisions of
HypocritesLaw
And others
Believe
Where
Doctrine
Christianity
Military
In the name of
Seer
Crusade
The island
Will
Our brothers
Disbelief
All
Was
Era
Pregnancy
Their religion
Apostles
They said
Arms
Interests
Those
Message
West
Assets
Young people
Fighting
Texts
Great
House
Al Qaeda
Fight
I
Uh
Sharia
Themselves
And between
Alone
Loyalty
Known
Who
People
Supporters
The greatest
Were
Fact
Sultan
Army
Governing
Preachers
Criminals
Word
Kill
Milli
I say
Mr.
Order
Response
People
I
More
Party
Arabs
Unitarian
Freedom
Laws
Security
Brother
Platform
Political
Faith
Palestine
Brotherhood
Falsehood
Governor
Legislation
All
Apostatize
Ignorance
Injustice
Worship
Iraq
The infidels
AfghanistanTerrorism
People
Government
Statement
Down
The enemies
The faithfulAgainst
And Enmity
Judgements
Apostates
Maqdisi
Idolaters
Prison
Fighting
Without
Oppressors
Country
GodAmerica
JewsForce
War
LandWay
Victory
Enemy
American
Excommunication
Killing
The worldIdolatry
The nationState
Idolators
Infidel
Monotheism
Mujahideen
Apostasy
Jihad The
Are
Zaid
Like
Newly
Directly
My nation
The
You
Legalization
Wants to
The
The
News
Prove
Opinion
Search
The nearest
To the son of
Human
By saying
Accept
I like
Consequent
We know
Rest
Indicates
Spoke
Strange
And how much
Not
Present
Bammer
Valley
Prove
Weakness
Dies
The old
Ray
Log in
ValuesIsaac
Estimated
Staff
Stronger
Century
Revolution
Requires
Known
Lord
Contrary to
Decision
Delivery
Assesses
After
Down
Eye
And possibly
Custom
Individual
Near
Series
Is
With
Fired
Tracking
Either
By himself
Obstacle
To the people
Line
Doctrines
Rejection
Reform
Tender
Detection
More
A way
Was
Once the
Stone
Should be
Deficit
Center
The sixth
Companions
And it
Type
Acts
PenaltyChildren
Skies
Bear
His hand
To reconcile
One thousandth
Follow the
Across
Jurisprudence
Fit
NotA
Hamid
Comes
Puts
Guides
Wise
Difference
Intervention
Easy
The fourth
Nine
Yemen
Raises
Sins
Remember
Rectum
As
Maintain
Del
EBay
Religiously
And other
Ruled that
Impose
Is
Before
The sky
Charity
Permanent
I see
Aas
Promise
Resurrection
Enlarge
Number
Evidence
Enough
LaSalle
Always
Serves as
Gives
Nature
Have
Found
And
See
One day
The basis of
Different
Being
Can
Mahmoud
Past
Ezz
Back
Sex
Pursuant to
Thousands of
Advice
Massoud
Confidence
Has become
Conditions
Was
Estimate
And piety
A little
I took
Two
Prevent
Strong
Number
Arrived
The large number of
V
Hatred
Circle
Differ
Forms
Caution
Necessary
Money
Increase
Author
Twice
Role
So and so
Do
Was
Desire
More
Roy
Except
Selection
The
Has become
The contrary
Remain
Ashraf
Provide
Between them
Provided
Intended
The famous
Doubt
As
The pretext of
Salah
Latecomers
Place
Hanifa
Presence
Hearing
Want to
Specific
Concern
Yourself
Families
At
Dhar
Can
Knowing
Drink
Asked
Palm
Show
There are
Wide
As
Willing
Never
Fear
Keeper
Names
Different
Study
Members
The best
Gel
Such as
Did
So
LeavesLearning
Reasons
Shed
Whole
Sayings
Adam
This
Hear
In other words
For
Lack
Right
Diligence
Sharp
Table
Light
Came
Optical
Man
I
Sharif
Asked
Terms
Then
If
Bad
Useful
His life
Mark
Seen
Halt
OneNo
About us
Exchange
Separation
Hanbal
And followers
Take
I heard
Kara
Comment
Find
Age
Because
Dispute
To be
The following
Ring
Cut
Livelihood
Agreed
Hidden
Delivers
Education
Perfection
Possible
Obtained
Certain
Paper
Angels
Foreign
Fire
For non−
Need
Perhaps the
Required
Interest
Saad
Years
Comes
Found
Contract
To imitate
Get
Science
Income
Quest
The establishment of
Other
Pay
A means
Has
Pay
Land
Leave
Satisfaction
Very
Desires
First
Often
Appropriate
The unseen
Knowledge
Treatment
Sin
Not
Request
The said
Account
Uncle
Education
Occurs
Combines
The
City
Great
Yam
Scholars
Jazz
Works
Service
Results
Tell
Love
Sentence
Kaaba
Lead
Close
Must
Foot
Back
Disposal
Wife
Egypt
Hit
When
And messengers
Must
Osman
Required
Grandfather
Said
Deal
Fixed
Language
Questions
Almighty
Suspicion
Out
Business
The day
Meaning
Full
The public
Virtue
Is called
Kitten
Will
See
A
Is located
Read
I
Roy
Up
Shows
Entry
Nuha
Sometimes
Status
Makes
Duty
The mind
I know
Six
Something
Feast
Preferred
HappyMeans
Character
Matt
Sins
Seven
Event
Should be
Question
Shafei
Recent
Means
In
And so on
Out
Trade
Hand
Graduation
Proven
Back
Whether
When
Relationship
Solution
Behind
Project
Problem
CameGo
Neighborhood
Provided that
Around him
Take
Re−
Period
Solve the
Blood
Abbas
Part
Adultery
Nasser
Affect
Prayers
Four
The time
Months
Return
End
Things
He wanted
And only
Calls for
Sin
Country
Not
Aisha
Guidance
Good
Parental
Prophecy
Is
Well
Clear
Word
Prevents
And then
Of religion
Out
The third
Things
Do
Of
Pronunciation
Link
The intended
Good
Five
Born
The original
Name
Wage
Passport
Increase
Likely
Men
Time
Is
Small
Travel
I hope
Innovation
And Prevention
Doctrine
The answer
Majah
Non−
Women
Children
Permissible
Create
Khaled
Satan
Full
True
Sunan
Ban
Old
Enters
DoorMorocco
Existing
Directed by
And because
To be
And blessings
Ask
WillBack
Thread
Raise
First
Especially
What
Cow
Messages
Episode
During the
Secrecy
About
Speech
Curse
Tirmidhi
True
Almighty
Note
Voice
UsurySecond
Intention
Save
Sun
Truth
Paradise
Period
Wear
Ras
I want
Half
Building
Young
Answer
Amount
But
GoldTotal
Owner
For
Mosques
Disease
Necessary
Repentance
For example
Food
Be
Days
Place
Weak
Back
You
Eating
Daybreak
DeprivedRoad
Directed by
Links
Valid
Dead
Do
And companions
Eg
Chatter
Al
David
Instinct
Ahmed
Hand
Good
Night
Elimination
For saying
Image
Month
Sale
Honorable
Noble
Take
Hair
Place
Reading
Women
Be
Collection
Water
Three
Critical
Woman
The best
Person
Are
Sacred
Need
Fasting
Up
Quoting
Bukhari
House
Means
Yes
Forbidden
Pray
Ramadan
Bass
Ten
ManTime
MaySource
Narrated byMustAge
HumanImam
Money
Mosque
QuranHajj
Zakat
Her husbandWomen
−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−<< >>Jihadi Not Jihadi
Word Frequency
aaaa
= 1/250
= 1/500
= 1/1000= 1/2000
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 11 / 38
Digital Literature Reviews (Nielsen & Stewart)
The International Relations Literature
Year
Num
ber
of A
rtic
les
1980 1985 1990 1995 2000 2005
010
2030
4050
6070 Realism
LiberalismConstructivismNon−paradigmatic
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 12 / 38
Digital Literature Reviews (Nielsen & Stewart)
Realism
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 13 / 38
Digital Literature Reviews (Nielsen & Stewart)
Liberalism
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 14 / 38
Digital Literature Reviews (Nielsen & Stewart)
Constructivism
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 15 / 38
Digital Literature Reviews (Nielsen & Stewart)
Who are the big names?
0
Constructivism
Rationalism
Realism
Liberalism
Wendt
Mearsheimer
FinnemoreNye
Katzenstein
Barnett
Walt
Ikenberry
BuzanSikkink
Snyder
Pape
Ruggie
Keohane
WaeverJervis
SchwellerWendt
Mearsheimer
Fearon
Nye
Katzenstein
Walt
Ikenberry
SnyderHuntington
Pape
Ruggie
KeohaneBueno de Mesquita
WaeverJervis
Deudney
SchwellerWendt
Fearon
FinnemoreNye
Katzenstein
Ikenberry
BuzanSikkink
Snyder
Simmons
Ruggie
KeohaneBueno de Mesquita
WaeverJervis
Gartzke
Wendt
Mearsheimer
FinnemoreNye
Katzenstein
Barnett
Walt
Ikenberry
BuzanSikkink
Snyder
Pape
Ruggie
Keohane
WaeverJervis
SchwellerWendt
Mearsheimer
Fearon
Nye
Katzenstein
Walt
Ikenberry
SnyderHuntington
Pape
Ruggie
KeohaneBueno de Mesquita
WaeverJervis
Deudney
SchwellerWendt
Fearon
FinnemoreNye
Katzenstein
Ikenberry
BuzanSikkink
Snyder
Simmons
Ruggie
KeohaneBueno de Mesquita
WaeverJervis
Gartzke
Wendt
Fearon
FinnemoreNye
Katzenstein
Barnett
Walt
Buzan
SnyderHuntington
Pape
KeohaneBueno de Mesquita
WaeverJervis
Schweller
Gartzke
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 16 / 38
International Events (O’Connor, Stewart & Smith)
Want to code international events of form “someone did something tosomeone else”
Original Approach: Manual Coding
First Automated Approach: 15,000 patterns, 200 event classes
Our Approach: Learn Event Types from the Text with a ProbabilisticModel!
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 17 / 38
International Events (O’Connor, Stewart & Smith)
Want to code international events of form “someone did something tosomeone else”
Original Approach: Manual Coding
First Automated Approach: 15,000 patterns, 200 event classes
Our Approach: Learn Event Types from the Text with a ProbabilisticModel!
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 17 / 38
International Events (O’Connor, Stewart & Smith)
Want to code international events of form “someone did something tosomeone else”
Original Approach: Manual Coding
First Automated Approach: 15,000 patterns, 200 event classes
Our Approach: Learn Event Types from the Text with a ProbabilisticModel!
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 17 / 38
International Events (O’Connor, Stewart & Smith)
Want to code international events of form “someone did something tosomeone else”
Original Approach: Manual Coding
First Automated Approach: 15,000 patterns, 200 event classes
Our Approach: Learn Event Types from the Text with a ProbabilisticModel!
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 17 / 38
International Events (O’Connor, Stewart & Smith)
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 18 / 38
International Events (O’Connor, Stewart & Smith)
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 19 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Many More Possibilities!
Some other interesting questions addressed with Text Data
How do civilians differ from military leaders on foreign policy?(Stewart and Zhukov 2009)
How do legislators relate to their constituents? (Grimmer 2010,2013)
What drives media slant? (Gentkow and Shapiro 2010)
Was there a constitutional moment in the 1860s? (Stewart andYoung 2013)
Where does [insert party/legislator] fall on the ideological spectrum?
Plus: Open-ended survey analysis (Roberts et. al 2013), DigitalHumanities (Jockers 2012), Digital Historiography (Mimno 2012),Public Opinion (Hopkins and King 2010), Congressional Discourse(Quinn et al 2010), Legal Analysis (Gill and Hall 2013) etc.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 20 / 38
Overview
1 Text as Data
2 Big Data isn’t about Data
3 Information Control in China
4 Conclusion
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 21 / 38
Big Data Isn’t About the Data
Data only gets you so far.
Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
Big Data Isn’t About the Data
Data only gets you so far.Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
Big Data Isn’t About the Data
Data only gets you so far.Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
Big Data Isn’t About the Data
Data only gets you so far.Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
Big Data Isn’t About the Data
Data only gets you so far.Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
Big Data Isn’t About the Data
Data only gets you so far.Four principles for automated text analysis:
1 Statistical models of language are wrong but useful.
2 Best methods amplify resources and augment humans.
3 There is no globally best method.
4 Validate, Validate, Validate.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 22 / 38
A unifying view of text methods
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 23 / 38
The Value of Better Methods
Moores Law (doubling speed/power every 18 months) vs. BetterAlgorithm (1000x speed increase in 1 day)
Statistics can see the needle in the haystack
Simple Methods can lead you astray.
Examples of bad analytics (Jobs and Jordan).
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 24 / 38
The Value of Better Methods
Moores Law (doubling speed/power every 18 months) vs. BetterAlgorithm (1000x speed increase in 1 day)
Statistics can see the needle in the haystack
Simple Methods can lead you astray.
Examples of bad analytics (Jobs and Jordan).
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 24 / 38
The Value of Better Methods
Moores Law (doubling speed/power every 18 months) vs. BetterAlgorithm (1000x speed increase in 1 day)
Statistics can see the needle in the haystack
Simple Methods can lead you astray.
Examples of bad analytics (Jobs and Jordan).
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 24 / 38
The Value of Better Methods
Moores Law (doubling speed/power every 18 months) vs. BetterAlgorithm (1000x speed increase in 1 day)
Statistics can see the needle in the haystack
Simple Methods can lead you astray.
Examples of bad analytics (Jobs and Jordan).
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 24 / 38
Example: International Events
200 Million International Events
Image from GDELT data by Kalev Leetaru
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 25 / 38
Example: International Events
Able to track violence over time.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 26 / 38
Example: International Events
Suddenly in the 1990’s Jordan began attacking everyone.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 27 / 38
Example: International Events
Because Michael Jordan was attacking the net.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 28 / 38
Overview
1 Text as Data
2 Big Data isn’t about Data
3 Information Control in China
4 Conclusion
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 29 / 38
Chinese Censorship (King, Roberts and Pan 2013)
Largest selective suppression of human expression in history
Press freedom ranking: 187th out of 197
Scale of Effort:I Total Government Censors: ≈ 200,000I Internet Police: 20, 000 − 50, 000
11 Million Posts analyzed, about 13% censored
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 30 / 38
Chinese Censorship (King, Roberts and Pan 2013)
Largest selective suppression of human expression in history
Press freedom ranking: 187th out of 197
Scale of Effort:I Total Government Censors: ≈ 200,000I Internet Police: 20, 000 − 50, 000
11 Million Posts analyzed, about 13% censored
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 30 / 38
Chinese Censorship (King, Roberts and Pan 2013)
Largest selective suppression of human expression in history
Press freedom ranking: 187th out of 197
Scale of Effort:
I Total Government Censors: ≈ 200,000I Internet Police: 20, 000 − 50, 000
11 Million Posts analyzed, about 13% censored
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 30 / 38
Chinese Censorship (King, Roberts and Pan 2013)
Largest selective suppression of human expression in history
Press freedom ranking: 187th out of 197
Scale of Effort:I Total Government Censors: ≈ 200,000I Internet Police: 20, 000 − 50, 000
11 Million Posts analyzed, about 13% censored
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 30 / 38
Chinese Censorship (King, Roberts and Pan 2013)
Largest selective suppression of human expression in history
Press freedom ranking: 187th out of 197
Scale of Effort:I Total Government Censors: ≈ 200,000I Internet Police: 20, 000 − 50, 000
11 Million Posts analyzed, about 13% censored
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 30 / 38
What Do They Censor?
Previous understanding: they censor criticisms of the government
New Results: they silence collective action
Criticism relatively uncensored.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 31 / 38
What Do They Censor?
Previous understanding: they censor criticisms of the government
New Results: they silence collective action
Criticism relatively uncensored.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 31 / 38
What Do They Censor?
Previous understanding: they censor criticisms of the government
New Results: they silence collective action
Criticism relatively uncensored.
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 31 / 38
What Do They Censor?
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 32 / 38
China and the Media (Roberts, Stewart and Airoldi 2013)
Want to understand how the traditional media covers China’s rise.
Topic Models to estimate and track topics in millions of news reports.
Uses the Structural Topic Model (an extension of Latent DirichletAllocation)
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 33 / 38
China and the Media (Roberts, Stewart and Airoldi 2013)
Want to understand how the traditional media covers China’s rise.
Topic Models to estimate and track topics in millions of news reports.
Uses the Structural Topic Model (an extension of Latent DirichletAllocation)
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 33 / 38
China and the Media (Roberts, Stewart and Airoldi 2013)
Want to understand how the traditional media covers China’s rise.
Topic Models to estimate and track topics in millions of news reports.
Uses the Structural Topic Model (an extension of Latent DirichletAllocation)
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 33 / 38
China and the Media (Roberts, Stewart and Airoldi 2013)
Relations with Australia
Poverty Alleviation
Reform of Financial System
ASEAN
Finance and Currency Exchange
General Human Rights
Official State Visits
Relations with Hong Kong
Relations with South Korea
Nuclear ProliferationBanking
Manufacturing and Growth
16 Party Congress
Technology and Industry
Tourism, Students, and Families
Television and Online Media
Chinese News Media
Exports
Chinese Political Leaders
Iraq
Northeast Asia Relations
Falungong, Activists, and Crime
Tibet
Olympics
SARS
Japanese Visits to Shrine
WTO
North Korea Relations
Planes and Military Weapons
Hong Kong Handover
Relations with Taiwan
India/Pakistan Relations
Six−Party Talks and North Korea
International Trade
Relations with Thailand
Energy and Oil
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 34 / 38
Topics by Time
●
0.00
0.05
0.10
0.15
Year
Mea
n To
pic
Pro
port
ions
1997 1999 2001 2003 2005 2007
●
●
●●
●●
●
●
●
●
●●●
●●
●
●
●
●●
●
●
●
●●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●●
●
●●●
●
●●●
●
●
●
●
●●●
●●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●●
●●●●●
●
●●
●●
●●
●●
●
●
●
2000 Taiwan Presidential Elections
2004 Taiwan Presidential Elections
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 35 / 38
Comparing Coverage by Source
Compare Xinhua to Western News Wires
Same topics covered but in different ways.I Taiwan: ‘one-china’, ‘reunification’ vs. ‘elections’, ‘democratic’I Falungong: ‘crime’, ‘illegal’ vs. ‘protest’, ‘crackdown’
Delayed coverage: SARS
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 36 / 38
Comparing Coverage by Source
Compare Xinhua to Western News Wires
Same topics covered but in different ways.
I Taiwan: ‘one-china’, ‘reunification’ vs. ‘elections’, ‘democratic’I Falungong: ‘crime’, ‘illegal’ vs. ‘protest’, ‘crackdown’
Delayed coverage: SARS
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 36 / 38
Comparing Coverage by Source
Compare Xinhua to Western News Wires
Same topics covered but in different ways.I Taiwan: ‘one-china’, ‘reunification’ vs. ‘elections’, ‘democratic’
I Falungong: ‘crime’, ‘illegal’ vs. ‘protest’, ‘crackdown’
Delayed coverage: SARS
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 36 / 38
Comparing Coverage by Source
Compare Xinhua to Western News Wires
Same topics covered but in different ways.I Taiwan: ‘one-china’, ‘reunification’ vs. ‘elections’, ‘democratic’I Falungong: ‘crime’, ‘illegal’ vs. ‘protest’, ‘crackdown’
Delayed coverage: SARS
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 36 / 38
Comparing Coverage by Source
Compare Xinhua to Western News Wires
Same topics covered but in different ways.I Taiwan: ‘one-china’, ‘reunification’ vs. ‘elections’, ‘democratic’I Falungong: ‘crime’, ‘illegal’ vs. ‘protest’, ‘crackdown’
Delayed coverage: SARS
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 36 / 38
Conclusion
Two Points:
Text as Data
Data + Analysis > Data
More Info: scholar.harvard.edu/bstewart
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 37 / 38
Conclusion
Two Points:
Text as Data
Data + Analysis > Data
More Info: scholar.harvard.edu/bstewart
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 37 / 38
Conclusion
Two Points:
Text as Data
Data + Analysis > Data
More Info: scholar.harvard.edu/bstewart
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 37 / 38
Conclusion
Two Points:
Text as Data
Data + Analysis > Data
More Info: scholar.harvard.edu/bstewart
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 37 / 38
Conclusion
Thank You!
Brandon M. Stewart (Harvard) Text as Data August 5, 2013 38 / 38
