Tools and Methods for Processing and Visualizing
Large Corpora
Gerold Schneider, Unversity of Zurich and University of Konstanz Mennatallah El-Assady, University of Konstanz
Hans Martin Lehmann, University of Zurich
d2e Conference, Helsinki
Page 1
Overview and Contents
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 2
Several approaches and methods which we develop or use to create workflows from data to evidence.
1. Do-it-yourself from words via numbers to trends as we use and develop them at the English Department in Zurich
• Searching specific instances: Dependency Bank
• Data-Driven Overuse: Lightside • Visualize your trends: GoogleViz
2. Interactive Visualizations as we use and develop them at the Data Analysis and Visualization Group at the University of Constance
• Topic Matrix View • Statistical Visualizations: Tableau
• Lexical Episode Plot
1.1 DYI d2e: Dependency Bank
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 3
• Search specific instances & sum by period/genre/etc.: Dependency Bank e.g. ‘education system’ vs. ‘system of education’
1.1 DYI d2e: Dependency Bank
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 4
‘education system’ vs. ‘system of education’. Trend confirmed by Google Books Noun-noun compounds are generally increasing
Relative Frequency of open-form noun-noun compounds
University of Zurich, English Department, Hans Martin Lehmann Page 5
confirms and extends Leech et al. 2009
Syntactic query: stative verbs in the progressive
“I’m loving it” in the BNC
6
1.2 DIY d2e: Data-driven Overuse
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 7
Machine-Learning for the Masses: Lightside http://lightsidelabs.com/what/research/ Allows you to do Machine Learning without programming skills. E.g. can we classify American speeches to republican/democrat? What are their typical linguistic features?
We use CORPS II corpus: 8 mio words, 3618 speeches (Guerini et al. 2013).
To do doc classfication with Lightside you must • Create TAB-separated input.
• Understand the broad idea of Naïve Bayes or regression, just because
• you need to interpret the results • you don’t want to crash your computer
• Be patient during the massive calculations
1.2 DIY d2e: Data-driven Overuse: Lightside
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 8
• Create TAB-separated input.
• Understand the broad idea of Naïve Bayes or regression, just because
• you need to interpret the results • you don’t want to crash your computer
Automated Media content analysis, Gerold Schneider Seite 9
1.2 DIY d2e: Data-driven Overuse: Lightside
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 10
• you need to interpret the results
1.3 DIY d2e: Google Visualization
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 11
• Back to noun-noun compounds: Relative frequency of the Alternation
1.3 DIY d2e: Google Visualization
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 12
Back to noun-noun compounds: Relative frequency of the alternation in COHA You need basic programming skills in R Google Viz is an R library
There are excellent instructions by Martin Hilpert: thanks!
2. Interactive Visualizations We use and develop Interactive Visualizations at the Data Analysis and Visualization Group at the University of Constance.
Visualization for Digital Humanities A main driving force for visualization in linguistics are Digital Humanities projects. Visualization is needed because the massive amounts of information can’t easily be viewed or understood using the traditional method of reading the texts.
Idea: Spot concepts, and zoom in to read the interesting parts.
And how do we spot concepts? By looking at keywords, such as words that are overused in particular documents/sections/paragraphs.
Unfortunately, the mapping from words to concept is not 1:1 • The same word can mean many things • Different words can refer to the same concept • Meanings change over time (Tony McEnery’s plenary) • Proper names are often just actors and witnesses in bigger concepts.
Vita brevis, arma longa Noun-noun compounds suffer less from these, they denote (new) concepts
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 13
2.1 Firthian Hypothesis and Topic Models Unfortunately, the mapping from words to concept is not 1:1
• The same word can mean many things: but contexts disambiguate • Different words can refer to the same concept. But only if they are in similar
contexts • Word senses change over time: need data-driven, context-based methods
“words with similar distributional properties have similar meanings” (Sahlgren, 2006: 21)
• Words in immediate Context ! Collocations (syntagmatic axis) • Words in larger Context ! semantic associations, topics (paradigmatic axis)
Topics are clusters of words that frequently co-occur. Many approaches to topic modelling (e.g. LDA) use a probability distribution model maximizing
We apply a deterministic topic model
approach (IHTM) to of 60.000 news articles from 1860 to 2000 (COHA)
Gerold Schneider, Menna el Assady, Hans Martin Lehmann Page 14
p(topic | document) ⋅ p(word | topic)
2.1 Topic Matrix View The First Moon Missions
lunar module moon orbit command flight space land surface spacecraft rocket craft mission ship walk
2.1 Selected topics and their keywords
Gerold Schneider, Mennatallah El-Assady, Hans Martin Lehmann Page 17
Topics of War and Peace
2.2 Topic Evolution over Time
Topic Evolution over Time
2.3 Lexical Episodes Plots
Distant Reading
Zooming and Highlighting
Close Reading 21
Index 4 Index 17 Index 23 Index 94
Actual Distribution
Equidistance Distribution
25
Lexical Episodes
= portion within the word sequence of a corpus where a certain word appears more densely than expected from its frequency in the whole text.
episode
100
22
Lexical Episodes
100
23
Chapter Title
11.11.15
University of Zurich, Division/Office, Title of the presentation, Author Page 24
Gerold Schneider, Mennatallah El-Assady, Hans Martin Lehmann Page 25
Prohibition in the United States was a nationwide constitutional ban on the sale, production, importation, and transportation of alcoholic beverages that remained in place from 1920 to 1933. https://en.wikipedia.org/wiki/Unemployment_and_Farm_Relief_Act 1930 some pointers also to the great depression radio broadcast peak in 1930s Hydrogen bomb: 1 small peak in 1945 world peace from 1945 on tax income revenue lat 1940's: irgendetwas passierte da. government coalition early 1950's : https://en.wikipedia.org/wiki/Attlee_ministry ?? Sports 1990s TV shows 1990s
