PIER Research Methods Protocol Analysis Module
Hua Ai
Language Technologies Institute/ PSLC
Questions? "There was a significant negative
correlation between the first and third metrics used to compute a Power score for the partner's conversational contributions and this question's numeric value, and a marginal negative correlation in the case of the second metric.“What are we supposed to interpret/learn from
this statement?
Rose at el. Automatic Analysis
By Machine Learning
What is machine learning?
Machine learning is aboutautomatically finding meaningful
patterns in data
Example for medical data:Rule predicts who is more likely to have problems with their teeth as they get older.
Why Machine Learning? We use machine-learning
products every day Weather forecast Spelling checker Automated voice response system …
It has been successfully applied to many research areas Natural language processing Market analysis Bioinformatics …
Note: Search engines use machine learning to personalize search results and suggest related sites or queries.
How does machine learning work?
The simplest rule learner willlearn to predict whatever isthe most frequent result class.This is called the majorityClass.
What will the rule be in this case?
It will always predict yes.
A slightly more sophisticated rule learner will find the feature that gives the mostinformation about the result class. Whatdo you think that would be in this case?
Outlook:Sunny -> NoOvercast -> YesRainy-> Yes
<Feature Name>:<value> -> <prediction><value> -> <prediction>…
What is machine learning?
Automatically or semi-automatically Inducing concepts (i.e., rules) from dataFinding patterns in dataExplaining dataMaking predictions
Data Learning Algorithm Model
New Data
PredictionClassification Engine
What will be the prediction?
Outlook:Sunny -> NoOvercast -> YesRainy-> Yes
Model
New Data
Yes
Terminology
Concept: the rule you want to learn
Instance: one data point from your training or testing data (row in table)
Attribute: one of the features that an instance is composed of (column in table)
* Compute the predicted value.
bad
Task Assign labels to a collaborative learning
corpus using the Weinberger and Fischer’s coding scheme Text classification task
Two approach categories The Feature Based Approach
Basic feature Thread Structure feature – depthFSM featuresLIWC features (Linguistic Inquiry Word Count)
The Algorithm Based ApproachCascaded Binary ClassificationConfidence Restricted Cascaded Binary
Classification Supervised/Unsupervised
Methodology issues related to automatic corpus analysis Validity
Whether the automatic coding accomplished by the computer captures human analysts’ intention
ReliabilityHow faithfully the automatic codes match those
of human experts’ Efficiency
Are we saving time by using the automatic classifier?
Ai et al. Manual Analysis
By statistical tests Interaction between tutor’s social behaviors
and tutor’s bias towards one student’s stance or the other
Topic modeling: sub-dialog level instead of utterance level
Study design 3*3
Social: how, low, noneGoal Match: yes, no, neutral
The ccLDA model
Green Collection Power Collection
Topic 1
Topic 2 Topic 3 Topic 1 Topic 2
Topic 3
A topic is a distribution of words.
We computed a score for each utterance based on the words contained in the utterance. This score stands for to what extend this utterance is biased towards this topic.
Results Learning Gain
Student learned most in Low-social + Yes-matched
Perception (Questionnaire & Topic detection)Effect on Goal-match manipulation
Conversational DataMore social turns in social conditionsMore off-task turns in non-social conditionMore Jokes on tutor in the high social condition
Important message Understand the data is important
Design good features Both in automatic and manual analysis