Inferring Multi-agent Activities from GPS Data

Henry Kautz & Adam Sadilek

Department of Computer ScienceUniversity of Rochester

Activity Recognition from GPS Most work to date on human activity

recognition from GPS data has focused ono Activities by individuals

· E.g.: Life-loggingo Aggregate activities of groups

· E.g.: Infer popular places for tour guide apps

Example: Route Prediction

Given a user's GPS history and current GPS data, infero The user's destinationo The route the user will takeo The user's transportation plan (foot, bus, car?)

Applicationso Provide just-in-time information o Smart GPS devices – e.g., assist with use of

public transportation

GPS readingzk-1 zk

Edge, velocity, positionxk-1 xk

qk-1 qk Data (edge) association

Time k-1 Time k

mk-1 mk Transportation mode

tk-1 tk Trip segment

gk-1 gk Goal

DBN Model of Transportation Plans

Liao, Patterson, Fox, & Kautz 2003

Route Prediction

Predicted goal

Predicted path

Relationships Matter

Joint Activities Goal:

o Model & recognize multi-agent activities from GPS data

o Focus on joint activities where agents play distinct role

Assumptionso We can write a qualitative commonsense theory

of the domain of activities· Our theory is partial and inconsistent

o We have access to locations of individuals· GPS data is noisy and incomplete

Applications

Eldercare: Monitoring Caregiving ActivitiesoMary is spending time with Susano John is taking Mary to her doctoroMeasures of social and familial engagement

are indicators of physical and mental health Strategic Analysis

o Battlefields or intelligence reportsoWho is doing what with who to whom?

Capture the Flag Domain

Rich but controlled domain of interactive activitieso Very similar to strategic applications

Ruleso Two teams, each has a territoryo A player can be captured when on the opponents'

territoryo A captured player cannot move until freed by a

teammateo Game ends when a player captures the opponents' flag

Game Video

Constraints

Player location critical for recognizing eventso Capture requires players to be within an arm's

reach Consumer grade GPS loggers do not appear

to have required accuracyo Error: 1 – 10 meters, typically 3 meterso Relative error: no better!

· Differences in individual units much larger than systematic component of GPS error

Close Up: Critical Area

Difficult Example Did player 7 capture player 12 or player 13?

Can we solve this problem ourselves?

Difficult Example 40 seconds later, we see:

o 13 isn't movingo Another defender, 6 isn't trying to capture 13o 12 is moving

Therefore, 7 must have captured 13!

Approach

Solve localization and joint activity recognition simultaneously for all players

Inputs:o Raw GPS data from each playero Spatial constraintso Rules of Capture the Flag

Output:oMost likely joint trajectory of all playerso Joint (and individual) activities

Relational Reasoning

This is a problem in relational inferenceo Estimate of each player's location & activities

affects estimates for other players Rules of the game are declarative and logical

o A player might cheat, but the rules are the rules! Tool: Markov Logic (Domingos 2006)

o Statistical-relational KR systemo Syntax: first-order logic + weightso Defines a conditional random field

Markov Logic Uses FOL to compactly describe a log

normal conditional random fieldo ground clause = feature functiono first-order formulas tie the weights of their

propositional groundingsoWeights are learned from data

p(x | y) ∝ exp wini (x)i

∑⎛⎝⎜⎞⎠⎟ where

ni (x) = number of groundings of clause i that are true in x

Example∀p,q enemies(p,q)⇒ ¬ friends(p,q)

∀p,q,t enemies(p,q)∧ nearby(p,q, t)( ) ⇒ capturing(p,q, t)

∀p,q friends(p,q)⇒ friends(q,p)

p,q ∈{A,B}, t ∈{2}

enemies(A,B)

friends(A,B)

capturing(A,B,2) nearby(A,B,2)

enemies(B,A)

capturing(B,A,2) nearby(B,A,2)

friends(B,A)

Inference: MaxWalkSAT (Kautz & Selman 1995)

1. Pick a random unsatisfied clause

2.

Flip a random atom

Flip the atom that maximizes the sum of the

weights of the satisfied clauses

p

1−p

Denoising GPS Data: Snapping

Snapping

Snapping

Soft Rules for Snapping (Localization)

Hard Rules for Capturing

Soft Rules for Capturing

Comparison

Baselineo Snap to nearest 3 meter cello If A next to B on A's territory, A captures Bo Expect high recall, low precision

Baseline+Stateso Like baseline, but keep memory of players state {captured, not

captured}o Expect better precision, possibly lower recall

2-Stage Markov Logic Modelo Find most likely explanation using ML theory about locationo Use as input to ML theory about capture

Unified Markov Logic Modelo Find most likely explanation using entire axiom set

Capture The Flag Dataset

3 games 2 teams, 7 players each GPS data logged each second Games are 4, 14, and 17 minutes long

length of game

(minutes)

# GPS readings

# Captures # Frees

Game 1 16 13,412 2 2

Game 2 17 14,400 2 2

Game 3 4 3,472 6 0

Game 4 12 10,450 3 1

Total 49 31,284 10 5

Results for Recognizing Captures

Sadilek & Kautz AAAI 2010

Discovering Failed Activities

In many applications (e.g. strategic analysis) it is as important to recognize failed attempts to perform an activity, as to recognize successful activities

A failed attempt is similar to a successful attempt, but does not achieve the purpose of the activity

Can we automatically extend a theory of activities to discover failed attempts?

Learning Failed Attempts

Giveno A theory of successful activitieso One or more examples of failed activities

Determine how to weaken the definition of the activity so that it also covers the failures

The removed constraints = the intention or purpose of the activity

Add negation of purpose to definition of failed activity

Inducing a Model of Failed Attempts

Inducing a Model of Failed Attempts

Inducing a Model of Failed Attempts

Inducing a Model of Failed Attempts

Inducing a Model of Failed Attempts

Capture The Flag Dataset

Failed attempts much more common than successful activitieso SC = successful captureo FC = failed captureo SF = success freeingo FF = Failed freeing

Results

Extended Theory is More Accurate for Successful Activities

Learned Formulas

Activity Discovery

The approach just described required a domain-specific background theory and a modest amount of labeled training data

Suppose we did not know the rules of Capture the Flag?o Inductive logic programming techniques can be used

to learn clauses as well as weights from labeled datao But we still would need to know the rules in order to

label the data! How can we discover the interesting interactions

and domain-specific rules?

Activity Discovery

Speculation: we can develop a general, domain-independent theory of interesting interactions

Find and cluster interesting interactions in order to discover interaction types (in ML, the predicates)

The "failed activities" (and thus intentions) will be a cluster near the successful cluster

Interesting Interactions

Elements of a general theory of interactions:o The agents are perceptually available to each othero Behavior of one agent can be well predicted by the behavior

of the other agents during the interactiono An interaction may change some (hidden) state of an

individualo Changes in the long term behavior of an individual are

evidence of such a state change How to represent this general theory?

o As a higher-order Markov Logic theory?o As interacting time series (individuals, pairs of individuals,

triples, etc)?

Summary

Joint activities can be recognized with high precision from GPS data, even the face of overwhelming noise, by leveraging qualitative domain knowledge encoded in a statistical-relational language

The purpose of an activity can be inferred by comparing successful and failed attempts

Approach is general, extensible, and has practical applications

Interesting challenge: discovering joint activities