Supporting Rationale Awareness in Large-Scale Online Open Participative Activities
Lu XiaoAssociate Professor
Faculty of Information & Media Studies, Department of Computer ScienceThe University of Western Ontario
London, ON, [email protected]
1
Outline
• Brief Introduction of my research background
• Rationale and Rationale Awareness – Definition and Related Studies
• Rationale Detection – Our Computational Approach
• Design for Rationale Awareness Support
• Future Work
2
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
About Me…
How to enrich computer-mediated group activities?
3
Information Science
Human-Computer Interaction
(HCI)
Data Science Social Sciences
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Between-Team Communication in the Intercultural Context
Collaborative decision-making
1. Xiao, L., & Huang, D. Y. (2015). Between-Team Communication in Cross-Cultural Context, Information, Communication & Society, http://www.tandfonline.com/doi/abs/10.1080/1369118X.2015.1067709#.VcNEDPmniDg
2. Xiao, L., Buchel, O., & Huang, D. Y. (2014). The Role of Team’s Communication Practices in Between-Team Decision Making Activities, Proceedings of iConference 2014
3. Xiao, L., Buchel, O., Martin, J., & Huang, D. Y. (2014), Towards Understanding of Contextual Factors in Information Sharing Behaviors of Inter-Team Activities, 42nd Annual Conference of Canadian Association for Information Science & Inaugural Librarians' Research Institute Symposium
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Design-based Parent-Child Mathematics Workshops
1. Xiao, L., Namukasa, I., & Zhang, Y. B. (2016). Design-based Mathematics Workshop, New Library World, to appear2. Xiao, L., Cai, T. T., & Eastmure, V. (2015). Parent-Child Dialogues and Artifact Control Behavior in Computer vs. Non-Computer Mediated
Parental Interactions, Proceedings of iConference 3. Xiao, L., Martin, J. (2012) Supporting Parent-Young Child Activities with Interactive Tabletops: A Conceptual Analysis, In Proceedings of the
ACM 2012 conference on Computer Supported Cooperative Work Companion (CSCW '12)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Curating Testimony: Living Archives of the
Rwandan Diaspora in Canada
1. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
2. Xiao, L., Yan, X. H., Miller, B., & Shrestha, A. (2014). Towards Visual Analytics for Digging into Human Rights Violations Data, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
3. Xiao, L., Elueze, I., & Kavanaugh, J. R. (2014) Human Rights Researchers' Data Analysis and Management Practices, Proceedings of the 77th Annual Meeting of Association for Information Science and Technology (ASIS&T)
4. Xiao, L., Luo, Y., & High, S. (2013). CKM: A Shared Visual Analytical Tool for Large-Scale Analysis of Audio-Video Interviews, at workshop “Big Data and the Humanities” IEEE Big Data 2013
5. Vashchilko, T., & Xiao, L. (2013). Literature review: how content analysis is used to study human rights violations in political science? Proceedings of 2013 iConference
Information Retrieval Pattern Identification
Information SharingCollaboration
Support
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
About Me…
How to enrich computer-mediated group activities?
7
Information Science
Human-Computer Interaction
(HCI)
Data Science Social Sciences
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
What is Data Science?
From the large size (Unstructured text) datasets, using scientific (Natural Language Processing + Machine Learning) methods to: 1). identify and extract (useful) data to support the online social activities2). identify language patterns to model online social behavior
Data Pattern
Heterogeneity Size, Storage and Access
SerendipityBatch, real-time
8
Science
Systematic Methods
Knowledge DiscoveryPrediction
ReliabilityValidity
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
9
Large-Scale Online Open Participative Activities
Opportunities & Challenges: social footprints: users' profile attributes, their social context and ties, and their published content
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
10
Detect Representative Rationale from Wikipedia’s Article for Deletion (AfD)
Discussions
1. Mao, W. T., Xiao, L., & Mercer, R. (2015). Toward a Knowledge Repository for Wikipedia Article for Deletion (AfD) Discussions, poster at the Third International Symposium of Chinese CHI. ACM, New York, NY, USA., to archive
2. Mao, W. T., Mercer, R., & Xiao, L. (2014a). Extracting Imperatives from Wikipedia Article for Deletion Discussions, First Workshop on Argumentation Mining at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), http://acl2014.org/acl2014/W14-21/pdf/W14-2117.pdf
3. Mao, W. T., Xiao, L., & Mercer, R. (2014b). Using Text Similarity and Sentiment Analysis to Identify Representative Rationales in Large-Scale Online Deliberations, 5th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis at the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), http://acl2014.org/acl2014/W14-26/pdf/W14-2624.pdf
4. Xiao, L., Mao, W. T., Mercer, R., & Nickerson, J. (under review). A computational approach to recognizing imperatives in online discussions
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
11
1. Khazaei, T., Xiao, L., & Mercer, R. (2015). “Change My View”: Detecting Persuasive Dialogs in Online Discussions, the 2015 Social Media & Society Conference (Jul. 27 – 29, Toronto, Canada)
2. Xiao, L., Khazaei, T., & Mercer, R. (under review), Writing to persuade: Analysis and Discovery of Persuasive Comments in Online Interactions
Detect Persuasive Texts in “Change My View” Sub-Reddit Discussions
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
12
Detect Users’ Concern for Privacy in Twitter Data
1. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (2016), Privacy Behaviour and Profile Configuration in Twitter, In WWW Workshop on Modelling Social Media (MSM '16) (April 11, Montreal, Canada)
2. Khazaei, T., Xiao, L., Mercer, R. (2016). Detecting Privacy Preferences from Online Social Footprints: A Literature Review, 2016 iConference (Philadelphia, PA, USA)
3. Khazaei, T., Xiao, L., Mercer, R., Khan, A. (2015). Social Computing and Intelligence: Exploring Opportunities for the Public and the Enterprise, 2015 IBM CASCON Emerging Technology Track
4. Khazaei, T., Xiao, L., Mercer, R., & Khan, A. (under review), Privacy Preference Inference via Collaborative Filtering
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
13
Large-Scale Online Open Participative Activities
Challenge for the Participants: To identify, interpret, and evaluate the others’ ideas and opinions
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationale: one’s justification of her ideas and opinions
Rationale awareness: one’s awareness of the others’ rationales in the activities
14
Supporting the Participants’ Rationale Awareness
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationales and Argumentation
15
“justifying a claim is making a communicative move that involves pragmatic as well as semantic constraints” (Bermejo-Luque, 2011)
Luque, L. B. (2011). Giving reasons: A linguistic-pragmatic approach to argumentation theory (Vol. 20). Springer Science & Business Media.
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
16
Rationale Studies – The Individual Level Focus
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationale Awareness in the Group Activities
17
Brainstorming activities in small groups and large online crowd contexts1. Xiao, L. (2014). Effects of rationale awareness in online ideation crowdsourcing tasks, Journal of the American Society for Information
Science and Technology (JASIST), 65, 1707-1720, doi: 10.1002/asi.230792. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-
Collaboration, 11(4), Article 2, 11-393. Xiao, L., & Carroll, J. M. (2013) The Effects of Rationale Awareness on Individual Reflection: Processes in Virtual Group Activities,
International Journal of e-Collaboration, 9(2), 78 - 954. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software ,
86(7), 1727 – 1737
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
18
Rationale Awareness in the Group Activities
Decision-Making Activities: Wikipedia’s Article for Deletion (AfD) and small group decision-making
1. Xiao, L., & Askin, N. (2014). What Influences Online Deliberation? A Wikipedia Study, Journal of the American Society for Information Science and Technology (JASIST), 65: 898–910
2. Xiao, L., & Askin, N. (2015). Rationale Sharing in Large-Scale Online Deliberations, Proceedings of iConference3. Xiao, L., & Witherspoon, R. (2015). The Effects of Pre-Discussion’s Note-Taking in a Hidden Profile Tasks, Proceedings of the 78th Annual
Meeting of Association for Information Science and Technology (ASIS&T)4. Xiao, L., & Witherspoon, R. (2015). Group Decision Making As Story Construction, Proceedings of the 78th Annual Meeting of Association
for Information Science and Technology (ASIS&T)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
19
Knowing the others’ rationales in the group activities is beneficial
Individual• Understanding and evaluating the others’ ideas and perspectives• Knowledge awareness • Contribution awareness• Reflective thinking
Group• Quality monitoring and control
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
• Authority claim detection (Bender et al., 2011; Marin et al., 2011)
• Opinionated claim detection (Rosenthal & McKeown, 2012)
• Justification identification (Biran and Rambow, 2011)- detect the “common” discourse relations in the rationale texts- Rhetorical Structure Theory (RST)- 12 selected RST relations based on the RST Tree Bank corpus
20
Rationale Detection: Related Work
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Biran, O., & Rambow, O. (2011). Identifying justifications in written dialogs by classifying text as argumentative. International Journal of Semantic Computing, 5(04), 363-381.
Rhetorical Structure Theory (RST): a theory of text organization created in the 1980s
[J .1] My first challenge is fairly general for the project, but still very much important.
2-4
Interpretation
Meeting remotely in groups can be very difficult
J oint 3-4
J oint
and having each member meet at the same time is important in
having group synergy. (37)
Purpose
Rhetorical Structure Theory
21
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationale Detection: A Corpus-based Lexical Cue Graph Model (Khazaei & Xiao, 2015;
Khazaei, Xiao, & Mercer, 2015)Step 1: Identify the RST relations that are commonly present in the rationale texts
22
Three RST relations are reported to be commonly present in the rationales: CIRCUMSTANCE, EVALUATION, and ELABORATION
1. Xiao, L., & Carroll, J. M. (2015). Shared practices in articulating and sharing rationale: An empirical study. International Journal of e-Collaboration, 11(4), Article 2, 11-39
2. Xiao, L. (2013) Do Members Converge to Similar Reasoning Styles in Teamwork? A Study of Shared Rationales in Small Team Activities, Proceedings of 2013 iConference
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationale Detection: A Corpus-based Lexical Cue Graph Model (Khazaei & Xiao, 2015;
Khazaei, Xiao, & Mercer, 2015)Step 1: Identify the RST relations that are commonly present in the rationale texts
23
Step 2: Identify the lexical cues for these RST relations based on two corpora
but it has not been sufficiently effective,
CONCESSION
EXPLANATIONEnergetic and concrete action has been taken in Colombia during the past 60 days against the mafiosi of the drug trade, because, unfortunately, it came
too late.
Corpora• RST Discourse Treebank• SFU Review Corpus
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
24
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Step 1: Identify the RST relations that are commonly present in the rationale texts Step 2: Identify the lexical cues for these RST relations based on two corpora
25
Step 3: Use the lexical cues to detect these RST relations so as to detect the rationale texts1. Khazaei, T., & Xiao, L. (2015a), Corpus-based Analysis of Rhetorical Relations: A Study of Lexical Cues, In Proceedings of the
IEEE International Conference on Semantic Computing (ICSC), pp. 417-423, 20152. Khazaei, T., & Xiao, L. (2015). Computational Analysis of Collective Intelligence –Towards Automatic Detection of
Rationales in Online Deliberations, In Proceedings of 2015 Collective Intelligence
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Step 1: Identify the RST relations that are commonly present in the rationale texts Step 2: Identify the lexical cues for these RST relations based on two corpora
26
Step 3: Use the lexical cues to detect these RST relations so as to detect the rationale textsStep 4: Use the probabilistic graph model G (V, E) to disambiguate the cues
V: the set of all possible tokens that may appear in the representations (e.g., POS tags)E: the set of all possible ordered transitions between any two tokens Weight of an edge E (i,j): the probability of a transition from token i to token j
Khazaei, T., Xiao, L., Mercer, R. (2015), Identification and Disambiguation of Lexical Cues of Rhetorical Relations across Different Text Genres, Workshop "Computational Semantics: Linking Lexical, Sentential and Discourse-level Models"(LSDSem) at 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
when
NNP
root
advcl advmod
Step 1: Identify the RST relations that are commonly present in the rationale texts Step 2: Identify the lexical cues for these RST relations based on two corpora
Our approach vs. direct usage of syntax
RST
SFU
Best performance in RST relation labeling in the literature (F-measure): 61.5% on single genre (Ji and Eisenstein, 2014) 27
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rationale Detection: A Corpus-based Lexical Cue Graph Model (Khazaei & Xiao, 2015;
Khazaei, Xiao, & Mercer, 2015)Step 1: Identify the RST relations that are commonly present in the rationale texts
28
But…
Are these three RST relations commonly present in the rationales independent of the context?
Three RST relations are reported to be commonly present in the rationales: CIRCUMSTANCE, EVALUATION, and ELABORATION (Xiao, 2013; Xiao & Carroll, 2015)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Rutgers Argument Mining Corpora (Wacholder, Muresan, Ghosh, Aakhus, 2014)
Corpora Total No. of Callouts
Total No. of Callouts that contain rationales
android_100 177 76ban_100 138 103iPad_100 234 100
Layoffs_100 200 151Twitter_100 191 97
Overview of the Rutgers Argument Mining Corpora and Our Annotations
Re: #18 You must be a shill for the RIAA and others. Truth is that file sharing in fact has been a boon to truly independent filmmakers and other non affiliated content producers, because they've become better known; it only really harms the big sharks, who have been robbing everyone blind and suppressing the little guy for decades.
Secondly, piracey is illegal. You aren't sharing anything. It's just a eufimisom for STEALING, and that's exactly what it is. It cheats the creator out of their money. 29
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
RST Annotation in the Rationale Texts (Xiao, under review)
Step 1: Rationale Region Annotation 1) Iterative open-coding process to identify the coding guidelines2) Inter-coder reliability check:
Cohen’s Kappa (the annotation of 347 comments): 70.3%3) Two researchers coded the rest data and examined each other’s results
Step 2: RST Relation Annotation• An experienced RST annotator • Intra-coder reliability check (relation labeling): 70% exact match and 7%
partial match 30
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
"Common" RST relations
Percentage in Coded RST relations
Justify 19.11%Elaboration 13.31%Conjunction 9.74%Contrast 8.81%Joint 7.73%Evaluation 5.93%Circumstance 4.75%Condition 4.61%Non Volitional Cause
3.81%
Concession 3.22%81.03%
Corpora Percentage in coded RST relations
android 77.81%ban 76.79%ipad 81.82%layoff 84.08%twitter 82.05%
Common RST relations in the callouts that have rationales
31
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
RST Relations from (Biran & Rambow, 2011)
ContrastNon Volitional CauseConcessionJustifyConjunctionJointEvaluationCircumstanceConditionElaboration
Common RST relations in the callouts that have rationales
ContrastCauseConcessionAntithesisAnalogyConsequenceEvidenceExampleExplanation – argumentationPurposeReasonResult
321. Xiao, L. (under review), Discourse Relations in Rationale-Contained Text Segments
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
33
Automatic detection of the individuals’ rationales
Data-Centered Study
When and How to present the rationales
Design •Performance measure
•Behavior change
Data-Centered Evaluation
The effects of the rationale awareness
Rationale Studies
Research Methodology: Data-Centered Approach
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
34
Shared Rationale Space
Shared Rationale
Space
WYSIWIS
1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software, 86(7), 1727 – 1737
2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on Technology for Education and Learning (TEL)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
35
Shared Rationale Space
Shared Rationale
Space
1. Xiao, L. (2012) The Effects of a Shared Free Form Rationale Space in Collaborative Learning Activities, Journal of Systems and Software, 86(7), 1727 – 1737
2. Xiao, L. (2011) Design for articulating and sharing rationales in virtual group learning activities, The IASTED International Conference on Technology for Education and Learning (TEL)
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Future Work• Improvement of the discourse relation detection approach for
identifying RST relations
• Exploration of other rationale indicators
• Design guidelines for rationale awareness tools
36
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
37
Large-Scale Online Open Participative Activities
Data Science Opportunities: Social footprints
Outline Introduction Rationale Automated Detection of Rationales
Design of Awareness Tools Future Work
Challenges• Common Ground• Skill Development (e.g., social
deliberative skill, reflection skill)• Knowledge Management
AcknowledgementColleagues at Penn State and the University of Western Ontario