Modelling Collaboration Using Point Processes

Peter. F. Halpin1, Alina. A. von Davier2,Jiangang Hao2, & Lei Lui 2

1NYU & 2ETS

IAC, 2014

1 / 27

Introduction

How can we disentangle the individual contribution from the teamcontribution?

I “Collaboration index”

Focus on Tetralogue chat data, not Basketball

2 / 27

Introduction

How can we disentangle the individual contribution from the teamcontribution?

I “Collaboration index”

Focus on Tetralogue chat data, not Basketball

3 / 27

Time Series Approach to Process Data

0 1 2 3 4 5 T

X0 X1 X2 X3 X4 · · ·

Examples:

I Bins = δ seconds; Xi = 1 if a chat message was sent; 0otherwise

I Bins = item responses; Xi = 1 if is answered correctly; 0otherwise

I In general: bins = “uniform” task components; Xi denotesthe outcomes of component i

I Methods: e.g., Brillinger, 2004; Granger & Lin, 1991; Butts,2008

4 / 27

Time Series Approach to Process Data

0 1 2 3 4 5 T

X0 X1 X2 X3 X4 · · ·

Examples:

I Bins = δ seconds; Xi = 1 if a chat message was sent; 0otherwise

I Bins = item responses; Xi = 1 if is answered correctly; 0otherwise

I In general: bins = “uniform” task components; Xi denotesthe outcomes of component i

I Methods: e.g., Brillinger, 2004; Granger & Lin, 1991; Butts,2008

5 / 27

Time Series Approach to Process Data

0 1 2 3 4 5 T

X0 X1 X2 X3 X4 · · ·

Examples:

I Bins = δ seconds; Xi = 1 if a chat message was sent; 0otherwise

I Bins = item responses; Xi = 1 if is answered correctly; 0otherwise

I In general: bins = “uniform” task components; Xi denotesthe outcomes of component i

I Methods: e.g., Brillinger, 2004; Granger & Lin, 1991; Butts,2008

6 / 27

Time Series Approach to Process Data

0 1 2 3 4 5 T

X0 X1 X2 X3 X4 · · ·

Examples:

I Bins = δ seconds; Xi = 1 if a chat message was sent; 0otherwise

I Bins = item responses; Xi = 1 if is answered correctly; 0otherwise

I In general: bins = “uniform” task components; Xi denotesthe outcomes of component i

I Methods: e.g., Brillinger, 2004; Granger & Lin, 1991; Butts,2008

7 / 27

Process Data and (Temporally) Complex Tasks

Let Xt = {Xt, Xt−1, . . . , X1}:I “Simple task:” p(Xt | Xt−1) = p(Xt), ∀ t

I “Complex task:” p(Xt | Xt−1) 6= p(Xt), ∃ t

I IRT treats tasks as simple: p(Xi | Xj , θ) = p(Xi | θ), ∀ i, j

8 / 27

Process Data and (Temporally) Complex Tasks

Let Xt = {Xt, Xt−1, . . . , X1}:I “Simple task:” p(Xt | Xt−1) = p(Xt), ∀ t

I “Complex task:” p(Xt | Xt−1) 6= p(Xt), ∃ t

I IRT treats tasks as simple: p(Xi | Xj , θ) = p(Xi | θ), ∀ i, j

9 / 27

Process Data and (Temporally) Complex Tasks

I A measure of task complexity:

KL(Xt) = EXt

(ln

p(Xt)

p(Xt)p(Xt−1) · · · p(X1)

)

= EXt

(ln∏i

p(Xi | Xi−1)

p(Xi)

)

=∑i

KL(Xi | Xi−1) (1)

10 / 27

Process Data and (Temporally) Complex Tasks

I As a function of lag u ∈ N:

KL(Xi | Xi−u) = EXt

(lnp(Xi | Xi−u)

p(Xi)

)(2)

I Summing over lags u s.t., KL(u) > 0 yields “overall” taskcomplexity:

C =∑i

∑u

KL(Xi | Xi−u) (3)

11 / 27

Example: Tetralogue Chat

0.0

0.1

0.2

0.3

IntroduceYourselves

Question1A

Question1B

QuestionsP2

QuestionsP3

PanToGarcia

ExampleSeismometerData

ExampleSeisQuestion1

ExampleSeisQuestion2

ExampleSeisQuestion3

ExampleSeismometerExplanation

ExampleSeismometerExplanation4

AlertLevelTableIntro

ReadyForFirstJob

IslandIntro

PlaceSeismometers

ReasonForTwoSeismometers

HowLongReport

ReasonForTwoMonths

NoteTaker

Lab2Trialogue1Seis

Lab2TrialogueVolcano2-2Months

Lab2FinalPredict

Congrats

Time

Information

Equation1

12 / 27

Example: Tetralogue Chat

0.0

0.1

0.2

0.3

IntroduceYourselves

Question1A

Question1B

QuestionsP2

QuestionsP3

PanToGarcia

ExampleSeismometerData

ExampleSeisQuestion1

ExampleSeisQuestion2

ExampleSeisQuestion3

ExampleSeismometerExplanation

ExampleSeismometerExplanation4

AlertLevelTableIntro

ReadyForFirstJob

IslandIntro

PlaceSeismometers

ReasonForTwoSeismometers

HowLongReport

ReasonForTwoMonths

NoteTaker

Lab2Trialogue1Seis

Lab2TrialogueVolcano2-2Months

Lab2FinalPredict

Congrats

Time

Information

[1] if its paused up on you just say something in chat. ! [2] Gotta type something to keep it from freezing up. ! So typing this for no other reason. ! [3] definite increase in lf ! [4] D, maybe? ! [5] Yeah, B is definitely true. ! [6] There's a little increase in the other, too though. ! [7] Too little to be significant? ! [8] B or D. Can't decide. ! [9] yeah i dont think theres much of a difference. ! it does go up but there were almost no lf tremors in the ! last one.![10] and now every single one of those dates has at least 4 lf ![11] So B?

13 / 27

Example: Tetralogue Chat

0.000

0.005

0.010

0.015

0 20 40 60Lag: 4 sec

Information

Equation2

14 / 27

Example: Tetralogue Chat

0.000

0.005

0.010

0.015

0 20 40 60Lag: 4 sec

Information

Equation 3

15 / 27

Separating the Complexity in a Dyadic Collaboration(Graphical version – derivation in Appendix)

0 10 20 30 40 50Minutes

0 10 20 30 40 50Minutes

} }

0 10 20 30 40 50Minutes

16 / 27

Separating the Complexity in a Dyadic Collaboration(Graphical version – derivation in Appendix)

0 10 20 30 40 50Minutes

0 10 20 30 40 50Minutes

} }

0 10 20 30 40 50Minutes

17 / 27

Separating the Complexity in a Dyadic Collaboration(Graphical version – derivation in Appendix)

0 10 20 30 40 50Minutes

0 10 20 30 40 50Minutes

} }

0 10 20 30 40 50Minutes

18 / 27

Separating the Complexity in a Dyadic Collaboration

In summary:

KLu(Zt) =∑i

(KL(Xi | Xi−u) +MI(Xi ;Yi−u)

+KL(Yi | Yi−u) +MI(Yi;Xi−u)) (4)

The complexity of a (dyadic) collaborative task has 4 parts:

1) How the actions of Person X depend on their own past actions

2) How the actions of Person X depend on past actions of person Y

3) How the actions of Person Y depend on their own past actions

4) How the actions of Person Y depend on past actions of person X

19 / 27

Example interpretation: Hawkes Process applied to chat 1

KL(X | {X}) = EZt

(∫lnµx + αxxφ(t)

µxdt

)

MI(X;Y ) = EZt

(∫lnµx + αxyφ(t) + αxxφ(t)

µx + αxxφ(t)dt

)1) How the actions of Person X depend on their own past actions →

Proportion of X’s chats that are “in response to” X’s chat behavior

2) How the actions of Person X depend on past actions of person Y →Proportion of X’s chats that are “in response to” Y’s chat behavior

1See Halpin & De Boeck, 2013; Halpin, 2013, von Davier & Halpin, 201320 / 27

Example: Tetralogue Chat

Auto Cross Total

0.00

0.05

0.10

0 20 40 60 0 20 40 60 0 20 40 60Lag: 4 sec

Information

ProcessX

Y

21 / 27

Collaboration index

I Complexity of collaborative task has a part for each personI Each person also has two parts...which parts to use for an

index (e.g., Cross or Total )

I Norm within dyads – e.g., divide by total?

3 “Proportion of variance” interpretation

7 Cannot compare over dyads unless each has same overalldependence (task design issue...)

7 Index is not independent for collaborators → incentivizes badcollaborations

22 / 27

Collaboration index

I Norm over dyads – e.g., subtract mean?

3 Can compare over dyads7 Norm referencing versus criterion referencing

23 / 27

Example: Tetralogue chat

0

30

60

90

120

0.5 1.0RX

count

Information Index

3.0

3.5

4.0

4.5

0.25 0.50 0.75 1.00RX

log

N_m

essa

ges

Relation with Number of Messages

5

6

7

8

0.25 0.50 0.75 1.00RX

log

N_w

ords

Relation with Number of Words

0.25

0.50

0.75

1.00

0.25 0.50 0.75 1.00RX

RX_Partner

Relation with Partner's Index

24 / 27

Examples: Relation to Task Outcomes

-2

0

2

4

0.25 0.50 0.75 1.00 1.25RX

Diff

eren

ce b

etw

een

Initi

al a

nd R

evis

ed T

otal

Sco

res

Relation to Task Outcomes

Obviously, more work required here!

25 / 27

Conclusions

I Can characterize temporally complex tasks in terms ofhistorical dependence

I Potential uses to inform task design and to study collaborations

I Can parse out complexity at 1) individual level 2) distinguishauto and cross dependence within an individual

I Aggregating over an interaction →“collaboration indices”

I Further steps: Reliability (standard errors and sample bias;Brillinger, 2004) and validation.

26 / 27

Thank you

peter.halpin@nyu.edu

27 / 27