Temporal patterns of knowledge construction:

Statistical discourse analysis of a role-based online discussion

To appear in the International Journal of Computer-Supported Collaborative Learning

Alyssa WiseSimon Fraser University

alyssa_wise@sfu.cas

Ming Ming ChiuState University of New York –Buffalo

mingchiu@buffalo.edu

I appreciate the research assistance of Choi Yik Ting

Online, asynchronous forums• Can participate anywhere – no geographic limits• Can share ideas at any time – more time to think• But often disconnected, only lists of isolated ideas

Guzdial & Turns, 2000; Herring, 1999; Thomas, 2002Summaries• Connect previous ideas and develop them• But often occur at end of discussion

& Do not benefit other members De Wever et al., 2007; Schellens et al. 2005; 2007

Encourage summaries in the middle of discussions?

Motivation for the Study

Knowledge Construction (KC) FrameworkGunawardena et al.’s (1997) Five-Phase Model

Research Context for the Study Emerging Themes in Collaborative Learning Research

(e.g. Chiu & Khoo, 2005; Kapur, 2001; Reimann, 2009)

(e.g. Cress, 2008; Suthers & Teplovs, 2011)

(e.g. Arvaja, 2007; Stahl, 2004; Strijbos et al., 2004)

Possible KC Patterns

0 5 10 15 201

2

3

4

5

1a

0 5 10 15 201

2

3

4

5

1b

0 5 10 15 201

2

3

4

5

2a

0 5 10 15 201

2

3

4

5

2b

0 5 10 15 201

2

3

4

5

3

0 5 10 15 201

2

3

4

5

4

Knowledge Construction Phase

Post Number

Research Questions

• What patterns characterize knowledge construction processes during an online discussion?

• What characterizes pivotal posts that divide a discussion into distinct segments? Summaries?

• Which characteristics of a post influence the knowledge construction phase of the next post?

 

PivotalPost

FunctionsSummary (+)…

RolesSynthesizer (+)…

Individual Control variablesGenderAge…

PostControl variables# of wordsTime of post…

Time contextWeek

 

KnowledgeConstruction

FunctionsSummary (+)…

RolesSynthesizer (+)…

Individual Control variablesGenderAge…

PostControl variables# of wordsTime of post…

Time contextWeekSegment

Methods

Function

Role

Give Direction

New Idea

BringSource

Use Theory Respond Summarize

Starter X X        

Inventor   X        

Importer   X X      

Mini-me       X    

Questioner         X  

Elaborator         X  

Devil’s Advocate         X  

Traffic Director X          

Synthesizer X         X

Wrapper           X

Content Analysis

Variable Inter-rater reliability ()Knowledge construction .84

New Idea .65

Bring in Source .92

Use Theory .73

Respond .98

Give Direction .76

Summarize .88

Unit of analysis: Post / Note / Message Objectively identified unit that its author defines

Rourke, Anderson, Garrison, & Archer, 2001Inter-rater reliability Krippendorf’s (range: -1 … 1; desired: > .67)

4 types of Analytical Difficulties

• Time

• Outcomes

• Explanatory variables

• Dataset

- No missing data

Statistical Discourse Analysis

Statistical Discourse Analysis

Difficulties regarding Time

· Segments differ (S2 S4)

· Serial correlation (p8 → p9)

Branches of notes

Strategies

· Breakpoint analysis + Model· Multilevel analysis (MLn, HLM)

· Test with I2 index of Q-statistics

· Model with lag outcomes, KC (-1)

· Store path: Identify prior turn

1

2

3

8

4

5 6

7

ID Action Turn # Valid?Previous

TurnValid (-1)

Ana Do three times four. 1 – –

Ben Three times four is seven 2 X 1 Eva Three times four is nine. 3 X 2 X

Jay Three times four is twelve. 4 3 X

ID Action Turn # Valid?Respondto post?

Valid (-1)

Ana Do three times four. 1 – –

Ben Three times four is seven 2 X 1 Eva Three times four is nine. 3 X 1 Jay Three times four is twelve. 4 3 X

Statistical Discourse Analysis

Difficulties regarding Time

· Segments differ (S2 S4)

· Serial correlation (p8 → p9) Multiple topics

Branches of notes (→→ )

Strategies

· Breakpoint analysis + Model· Multilevel analysis (MLn, HLM)

· Test with I2 index of Q-statistics

· Model with lag outcomes, KC (-1)

· Store path: Identify prior turn· Vector Auto-Regression

Lag explanatory variablese.g., Valid (-1), Girl (-1) Valid (-2)

1

2

3

8

4

5 6

7

Statistical Discourse AnalysisOutcome Difficulties

Ordered outcome (KC 1-5) Infrequent outcomes (00010)

Strategies

Ordered Logit / Probit Logit bias estimator

Statistical Discourse AnalysisExplanatory model Difficulties

· People, Groups & Topics differ

· Mediation effects (X→M→Y)

· False positives (+ + + +)

Strategies

· Multilevel analysis Multilevel mediation tests

· 2-stage linear step-up procedure

Results – KC PhasesKC Phase % of Posts1) Sharing Information 60

2) Exploring Dissonance

3

3) Negotiating Meaning 16

4) Testing / Modifying 4

5) Agreeing / Applying 17

Results: Summaries as Pivotal Posts

Each discussion averaged

1 pivotal post

(2 time periods)

Results - KC Patterns

0 5 10 15 201

2

3

4

5

1a

0 5 10 15 201

2

3

4

5

1b

0 5 10 15 201

2

3

4

5

2a

0 5 10 15 201

2

3

4

5

2b

0 5 10 15 201

2

3

4

5

3

0 5 10 15 201

2

3

4

5

4

Knowledge Construction Phase

Post Number

No Regressive Segments

Pivotal Posts → Distinct Segments

No Regressive Segments

Segments Skipped

KC phases

Predicting Pivotal Posts

SynthesizerPivotal

Post

Extensive SummaryWrapper

Role Current Post

Time 2 posts ago Previous post Role Current post

Knowledge Construction

Summary

After 1st

pivotal post

New Idea (-1)after 1st pivotal

post

Respond (-2) after 1st pivotal post

Wrapper

Synthesizer

Predict Knowledge Construction

KC pattern

KC phase 1 KC phase 3 or 5

(Share) (Negotiate Meaning or Agree/Apply)

Few KC phases 2 or 4 (Dissonance, Testing)

Pivotal post

Extensive Summary often

By Synthesizer or Wrapper usually

Extensive Summary

Showed higher KC

Elevated KC of subsequent posts

Summary of Results

Teacher / Designer Assign Synthesizer Role

- Increase midway summaries and elevate KC- Simple, effective intervention

Productive online discussions do not require all phases

Researcher Empirically test Gunawardena et al’s KC model New method for analyzing online discussion

- Statistically identifies pivotal posts & segments- Test hypotheses about relationships among posts - Examine variables at multiple levels - Examine differences over Time

Implications

Further Questions• With many choices of dimensions for the

breakpoints, which one(s) should we use?

• What do identification of same vs. different breakpoints across different dimensions tell us?

• How can we do meta-analyses of multiple data sets with somewhat different codes?

• Which analyses (qualitative and/or quantitative) might be fruitful on the same data set?