Post on 28-Dec-2015
transcript
Evaluating the Effect of Social Evaluating the Effect of Social Conversation on Learning, Conversation on Learning, Interaction, and Perceived Interaction, and Perceived
Interdependence in a Collaborative Interdependence in a Collaborative Math Problem Solving EnvironmentMath Problem Solving Environment
Rohit Kumar, Gahgene Gweon, Mhesh Joshi, Rohit Kumar, Gahgene Gweon, Mhesh Joshi, Yue Cui, Adaeze Nwaigwe, Carolyn P. RosYue Cui, Adaeze Nwaigwe, Carolyn P. Roséé
Language Technologies InstituteLanguage Technologies Institute and and Human-Computer Interaction InstituteHuman-Computer Interaction Institute
Carnegie Mellon UniversityCarnegie Mellon University
Why study help?Why study help?
Offering of deep help and elaborated Offering of deep help and elaborated explanations predicts post test explanations predicts post test performance (e.g., Webb et al., 2002)performance (e.g., Webb et al., 2002)
Help behavior mediates learning (e.g., Help behavior mediates learning (e.g., Gweon et al., 2006; Gweon et al., Gweon et al., 2006; Gweon et al., 2007)2007)
Exchanging help in the context of Exchanging help in the context of collaborative learning increases collaborative learning increases identification with a learning identification with a learning community and motivation, and community and motivation, and improves race relations (Sharan, 1980)improves race relations (Sharan, 1980)
Providing help leads to feelings Providing help leads to feelings empowerment for “low status” students empowerment for “low status” students (Elbers & Hann, 2004)(Elbers & Hann, 2004)
How can we prompt helping How can we prompt helping behavior?behavior?
Explicit prompts (Gweon et al., 2006)Explicit prompts (Gweon et al., 2006) Manipulating availability of help from problem Manipulating availability of help from problem
solving environment (Gweon et al., 2007)solving environment (Gweon et al., 2007) Girls offered more help with delayed feedback from Girls offered more help with delayed feedback from
environmentenvironment Boys offered more help with immediate help from the Boys offered more help with immediate help from the
environmentenvironment Current study:Current study:
Will students follow the example of conversational Will students follow the example of conversational agents offering conceptual help?agents offering conceptual help?
Will social prompts affect how students feel and Will social prompts affect how students feel and behave?behave?
Chat AgentChat AgentTutor:Tutor: Another way to think of dividing by fractions is this: If I have Another way to think of dividing by fractions is this: If I have
1 divided by 1/2, can I turn that into something divided by 1?. If 1 divided by 1/2, can I turn that into something divided by 1?. If I want to turn the denominator, which is 1/2, into 1, what do I I want to turn the denominator, which is 1/2, into 1, what do I need to do to it? need to do to it?
Student 1:Student 1: yep yep Tutor:Tutor: Actually, you need to multiply by two. And If I multiply the Actually, you need to multiply by two. And If I multiply the
denominator by 2, what do I have to do to the numerator?denominator by 2, what do I have to do to the numerator?Student 1:Student 1: 2 2 Tutor:Tutor: Right. So 1 divided by 1/2 is the same as 2 divided by 1. Right. So 1 divided by 1/2 is the same as 2 divided by 1.
So that is one reason why the flip and multiply strategy for So that is one reason why the flip and multiply strategy for dividing by fractions works. If you multiply the denominator by dividing by fractions works. If you multiply the denominator by something to turn it into 1, you have to multiply the top by the something to turn it into 1, you have to multiply the top by the same value. So let's try another one. If I want to divide same value. So let's try another one. If I want to divide something by 2/3, what do I need to multiply the numerator and something by 2/3, what do I need to multiply the numerator and denominator by? denominator by?
Student 1:Student 1: 3 3 Tutor:Tutor: Actually, it is 3/2. You should think about why it's 3/2. Actually, it is 3/2. You should think about why it's 3/2.
Collaborative Problem Solving EnvironmentCollaborative Problem Solving Environment
Jan packed several books to amuse herself on along car ride to visit her grandma. After 1/5 ofthe trip she had already finished 6/8 of the booksshe brought. How many times more books shouldshe have brought than what she packed?
Extraneous Entertainment?Extraneous Entertainment?
Connection with VMT WorkConnection with VMT Work
Chat analysis from VMT sessionChat analysis from VMT session ““Class clown” at first appeared to be Class clown” at first appeared to be
“goofing off”“goofing off” Later reflection revealed he was a tone Later reflection revealed he was a tone
setter, putting his fellow students at easesetter, putting his fellow students at ease His light hearted behavior supported his His light hearted behavior supported his
fellow students in foraging ahead to solve fellow students in foraging ahead to solve difficult problemsdifficult problems
Experimental DesignExperimental Design
Participants: 30 6Participants: 30 6thth graders from a suburban graders from a suburban elementary schoolelementary school
Learning in pairs with or without social promptsLearning in pairs with or without social prompts 2 Condition between subjects design2 Condition between subjects design
Dependent MeasuresDependent Measures Tests and QuizzesTests and Quizzes QuestionnaireQuestionnaire Process analysis of conversational behaviorProcess analysis of conversational behavior
Experimental ProcedureExperimental Procedure
Day 1Day 1 Pre-testPre-test
Day 2Day 2 45 minute lab session45 minute lab session Quiz 1Quiz 1
Day 3Day 3 45 minute lab session45 minute lab session Quiz 2Quiz 2 QuestionnaireQuestionnaire
Day 4: [separated from Day 3 by a weekend]Day 4: [separated from Day 3 by a weekend] Post-testPost-test
QuestionnaireQuestionnaire
Perceived problem solving competence of self Perceived problem solving competence of self and partnerand partner
Perceived benefit of collaborationPerceived benefit of collaboration Perceived help receivedPerceived help received Perceived help providedPerceived help provided
All questions were answered with a 6 point likert All questions were answered with a 6 point likert scale ranging from 0 (strongly dissagree) to 6 scale ranging from 0 (strongly dissagree) to 6 (strongly agree)(strongly agree)
Questionnaire ResultsQuestionnaire Results
Control Experimental
Perceived Self Competence 4.2 (.56) 4.1 (.23)
Perceived Partner Competence
4.3 (.62) 3.9 (.49)
Perceived Benefit of Collaboration
4.5 (.74) 4.4 (.70)
Perceived Help Received* 1.8 (1.3) 3.3 (.69)
Perceived Help Provided* 1.8 (1.1) 3.1 (1.1)
Test ResultsTest Results
Very weak evidence in favor of Very weak evidence in favor of Experimental conditionExperimental condition
Consistent (but non-significant!) trend for Consistent (but non-significant!) trend for students in the Experimental condition to students in the Experimental condition to learn morelearn more
Marginal advantage for Experimental Marginal advantage for Experimental condition on Lab day 2 on Interpretation condition on Lab day 2 on Interpretation problems (p=.06, effect size .55 s.d.)problems (p=.06, effect size .55 s.d.)
Coding Scheme (Gweon et al., Coding Scheme (Gweon et al., 2007)2007)
Help provision mediates learning (Gweon et al., Help provision mediates learning (Gweon et al., 2006; Gweon et al., 2007)2006; Gweon et al., 2007)
(R)(R) Help Requests: “Help me”, “I’m stuck”, “I Help Requests: “Help me”, “I’m stuck”, “I don’t get it.”don’t get it.”
(P)(P) Help Provisions: “Find the common Help Provisions: “Find the common denominator”, “Try the flip and multiply strategy”denominator”, “Try the flip and multiply strategy”
(C)(C) Can’t help: “I don’t know”, “I’m stuck too” Can’t help: “I don’t know”, “I’m stuck too” (D)(D) Deny help: “ask the teacher”, “you’re an Deny help: “ask the teacher”, “you’re an
idiot”, “press the help button”idiot”, “press the help button” (N)(N) Nothing Nothing
Help Request met with Provision of Help Request met with Provision of HelpHelp
Student 1Student 1: What operation do we do?: What operation do we do? <student 2 tries multiplication and gets negative feedback <student 2 tries multiplication and gets negative feedback
from the problem solving environment>from the problem solving environment> <student 2 tries divide and gets positive feedback from <student 2 tries divide and gets positive feedback from
the problem solving environment>the problem solving environment> Student 2Student 2: We divide. Now look at the problem, what is : We divide. Now look at the problem, what is
the other fraction we must divide by?the other fraction we must divide by?
Student 1:Student 1: What do we put on top of the fraction? What do we put on top of the fraction? Student 2:Student 2: Did you find a common denominator? Did you find a common denominator? <student 1 correctly finds the common denominator><student 1 correctly finds the common denominator>
Help request met with Can’t HelpHelp request met with Can’t Help
Student 1:Student 1: Why 16? Why 16? Student 2: Student 2: I don’t know.I don’t know.
Student 1: Student 1: I need help.I need help. Student 2: Student 2: SameSame Student 1:Student 1: 23/2 23/2 Student 2:Student 2: What’s 23/2? What’s 23/2? Student 1:Student 1: 11.5 11.5
Help request met with Deny HelpHelp request met with Deny Help
Student 1: Student 1: I don’t get itI don’t get it Student 2: Student 2: hold onhold on <then student 1 tried something and got negative <then student 1 tried something and got negative
feedback from the problem solving environment.>feedback from the problem solving environment.> <finally student 1 tried something else, which was <finally student 1 tried something else, which was
correct, and got positive feedback from the correct, and got positive feedback from the problem solving environment>problem solving environment>
Student 1:Student 1: I don’t know what to do I don’t know what to do Student 2:Student 2: click on the help button click on the help button
Help request met with No Help request met with No ResponseResponse
Student 1: Student 1: I don’t get itI don’t get it <student 2 tries something and gets negative feedback from <student 2 tries something and gets negative feedback from
the problem solving environment>the problem solving environment> <student 2 tries something else and gets negative feedback <student 2 tries something else and gets negative feedback
from the problem solving environment>from the problem solving environment> <student 2 clicks on the help button><student 2 clicks on the help button> <student 1 tries something that is correct and gets positive <student 1 tries something that is correct and gets positive
feedback from the problem solving environment>feedback from the problem solving environment>
Student 1: Student 1: ?? <student 2 tries something and gets negative feedback from <student 2 tries something and gets negative feedback from
the environment>the environment> <student 1 tries something, which is correct, and gets <student 1 tries something, which is correct, and gets
positive feedback from the environment>positive feedback from the environment>
More Help Related Episodes More Help Related Episodes per Problem in Experimental per Problem in Experimental
ConditionCondition
F(1,15) = 16.8,p < .001F(1,15) = 16.8,p < .001
Mean Experimental = .69, Mean Control = .30Mean Experimental = .69, Mean Control = .30
Effect size = 1 standard DeviationEffect size = 1 standard Deviation
Results from Corpus AnalysisResults from Corpus Analysis
Experimental (Day 1)
Experimental (Day 2)
Control (Day 1)
Control (Day 2)
Total Episodes* 47.1 (8.2) 61.3 (12.3) 33.8 (17.9) 49.1 (26.9)
Social Prompt Episodes*
24.1 (9.9) 33.7 (16.2) 0 (0) 0 (0)
Help Episodes (P)Unsolicited
.79 (1.6)1.7 (2.1)
.36 (1.1)3.2 (6.0)
1 (1.3)2.1 (3.2)
1.4 (2.9)1.9 (3.2)
Unanswered Help Requests (C+R+N)
2.4 (2.7) 1.4 (1.9) 2.2 (1.9) 1.4 (1.4)
Non-Help Episodes* 19.9 (5.6) 35.8(9.3) 30.6 (16.3) 46.3 (25.1)
Proportion on Non-Help Related Proportion on Non-Help Related EpisodesEpisodes
Solicited Help
Unsolicited Help
UnansweredRequests
Non-help episodes
Experimental (Day 1)
Solicited Help
Unsolicited Help
UnansweredRequests
Non-help episodes
Experimental (Day 2)
Solicited Help
Unsolicited Help
UnansweredRequests
Non-help episodes
Control (Day 1)
Solicited Help
Unsolicited Help
UnansweredRequests
Non-help episodes
Control (Day 2)
80.0%
87.8%
85.2%
90.8%
What was happening in non-help What was happening in non-help episodes?episodes?
Coordination Coordination Who should go nextWho should go next Which parts of the problem each person should be Which parts of the problem each person should be
responsible forresponsible for Regulation of speed (Go! Slow down! Hurry up!)Regulation of speed (Go! Slow down! Hurry up!)
Complaining about being hungry or boredComplaining about being hungry or bored Joking about the content of the problemsJoking about the content of the problems Cheering about finishing problemsCheering about finishing problems
More talk about winning or losing in the control conditionMore talk about winning or losing in the control condition More references to “we”, “us”, and “our” in the More references to “we”, “us”, and “our” in the
experimental conditionexperimental condition Insulting each otherInsulting each other
Insults like “loser”, “you stink”, and “stupid” only occurred Insults like “loser”, “you stink”, and “stupid” only occurred in the Control condition!in the Control condition!
Negative Affect in Control ConditionNegative Affect in Control Condition
Student 1:Student 1: finally finally Student 2:Student 2: Shut up Shut up Student 1:Student 1: oooooooooo burn oooooooooo burn Student 2:Student 2: I don't like you I don't like you Student 1:Student 1: fine be that way fine be that way Student 2:Student 2: how did you get that how did you get that Student 1:Student 1: Guessing Guessing Student 2: Student 2: good, do you got it? good, do you got it? Student 1:Student 1: no no Student 2: Student 2: well too bad well too bad
What did we learn?What did we learn?
Social prompts affected student attitudes towards Social prompts affected student attitudes towards each othereach other More insults and competitive attitude in the Control More insults and competitive attitude in the Control
conditioncondition
Social prompts fostered more perceived inter-Social prompts fostered more perceived inter-dependence (based on questionnaire)dependence (based on questionnaire)
Social prompts increased the proportion of help Social prompts increased the proportion of help related episodes per problemrelated episodes per problem
Possible learning effectPossible learning effect Social factors affect learningSocial factors affect learning
TagHelper toolsTagHelper toolshttp://www.cs.cmu.edu/~cprose/TagHelper.htmlhttp://www.cs.cmu.edu/~cprose/TagHelper.html
Supporting data analysis involving Supporting data analysis involving conversational dataconversational data
Supporting on-line assessmentSupporting on-line assessment Triggering interventionsTriggering interventions
TagHelper
Labeled Texts
Unlabeled Texts
Labeled Texts
A Model that can Label More Texts
Time
Hel
p
Student1:Student1: I don’t I don’t understand what to understand what to do next.do next.Student2:Student2: You’re an idiot. You’re an idiot.Support Agent:Support Agent: Student2, Student2, it looks like your partner it looks like your partner could use some help.could use some help.
Supporting Data Analysis for Supporting Data Analysis for helping behaviorhelping behavior
Peer tutoring in Algebra Peer tutoring in Algebra LearnLabLearnLab
Data coded for high-level-Data coded for high-level-help, low-level-help, and no-help, low-level-help, and no-helphelp Important predictor of learning Important predictor of learning
(e.g., Webb et al., 2003)(e.g., Webb et al., 2003)
TagHelper achieves TagHelper achieves agreement of .82 Kappaagreement of .82 Kappa
Can be used for follow-up Can be used for follow-up studies in same domainstudies in same domain
* Contributed by Erin Walker
Example of Triggering InterventionsExample of Triggering Interventions
Collaborative idea Collaborative idea generation in the Earth generation in the Earth Sciences domainSciences domain
Chinese TagHelper Chinese TagHelper learns hand-coded topic learns hand-coded topic analysisanalysis Human agreement .84 Human agreement .84
KappaKappa TagHelper TagHelper
performance .7 Kappaperformance .7 Kappa Trained models used in Trained models used in
follow-up study to trigger follow-up study to trigger interventions and interventions and facilitate data analysisfacilitate data analysis
Example DialogueExample Dialogue
SpeakerSpeaker TextText
Student 1Student 1 People stole sand and stones to use for construction.People stole sand and stones to use for construction.
VIBRANTVIBRANT
Yes, steeling sand and stones may destroy the balance and Yes, steeling sand and stones may destroy the balance and thus make mountain areas unstable. Thinking about thus make mountain areas unstable. Thinking about development of mountain areas, can you think of a kind of development of mountain areas, can you think of a kind of development that may cause a problem? development that may cause a problem?
Student 2Student 2 Development of mountain areas often causes problems.Development of mountain areas often causes problems.
Student 1Student 1 It is okay to develop, but there must be some constraints.It is okay to develop, but there must be some constraints.
* Feedback during idea generation increases learning and idea generation productivity (except during the first 5 minutes) (Wang et al., 2007)
Pairs+Feedback
Individuals+NoFeedback
Pairs+NoFeedback
Individuals+Feedback
Process AnalysisProcess Analysis
Process loss Pairs vs Individuals:Process loss Pairs vs Individuals:F(1,24)=12.22, p<.005, 1 sigmaF(1,24)=12.22, p<.005, 1 sigma Process loss Pairs vs Individuals: Process loss Pairs vs Individuals:
F(1,24)=4.61, p<.05, .61 sigmaF(1,24)=4.61, p<.05, .61 sigmaNegativeNegative effect of Feedback: effect of Feedback:F(1,24)= 7.23, p<.05, -1.03 sigmaF(1,24)= 7.23, p<.05, -1.03 sigma PositivePositive effect of feedback: effect of feedback:
F(1,24)=16.43, p<.0005, 1.37 sigmaF(1,24)=16.43, p<.0005, 1.37 sigma
0 5 10 15 20 25 30
02
46
810
12
Time Stamp
#Uni
que
Idea
s
Unique Ideas
Nom+NNom+FReal+NReal+F Pairs+Feedback
Individuals+NoFeedback
Pairs+NoFeedback
Individuals+Feedback
Pairs+Feedback
Individuals+NoFeedback
Pairs+NoFeedback
Individuals+Feedback
Thank you for listening!Thank you for listening!