Making Sense of Sensors

Henry KautzDepartment of Computer Science & EngineeringUniversity of Washington, Seattle, WA

Funding for this research is provided in part by IISI and AFRL/IF

Making Sense of Sensors

or … Climbing the Data Interpretation Food-Chain

The Ubiquitous Future

Rapidly declining size and cost of sensing and networking technology makes it practical to rapidly deploy systems that monitor large environments in great detail– factories, airports, hospitals, homes– oceanic regions, cities, countryside

Problem: it is easier to collect data than make to sense of it!

Data Fusion

Traditional work in data-fusion attacks problem of recovering specific physical phenomena from the readings of homogeneous networks of noisy sensorsE.g.: given readings from underwater microphone array, determine the position of a submarine

Current Trends

Heterogeneous sensors– Instrumented environment: motion detectors, weight

detectors, video, audio, …– Instrumented personnel: smart badges, GPS phones,

metabolic sensors. …

Goal: high-level understanding– What actions are being performed?– What are the goals of the subjects?– Do we need to intervene?

Example: Security

System monitors activity in a post officeTracks common tasks performed by individuals– Mailing packages– Getting mail from PO boxes– Buying stamps

Alerts operator when abnormalities noted– Person leaves package on floor and exits– Loitering (but not waiting in line!)

Example: Guiding

Activity Compass: GPS system that– Learns daily patterns of travel – Understands walking, car, bus, bike– Integrates external information

• Real-time bus data

Predicts problems– Will user miss appointment?– Is user on the wrong bus?

Offer proactive help– E.g., suggest alternative travel plan

Triple-Use Technology

Plan-AwareComputing

Military

surveillanceaugmented cognition

CommercialSoftware

intelligent user interfaces

PatientCare

aging in placeassisted cognition

Key Issue

How to go from noisy and incomplete sensor measurements toA meaningful description of what a person is doing

• “Waiting to mail package”• “Trying to get home”

A decision by the system about whether or not to intervene … in a principled and scalable manner!

Data Interpretation Food Chain

Movement

Intentions

Behavior

Interventions

Model-Based Interpretation

General approach: build a probabilistic model of– Common user goals– Plans (complex behaviors) that achieve those goals

• Feasibility constraints • Temporal constraints• Failure (abnormality) modes

– How simple behaviors are sensed

Run model “backwards” to interpret sensed data

Million-Mile View

In principal we know how to estimate the state of the system under observation:

To make this practical, we must take advantage of the regular structure of the domain

1 1 1Bel( ) Pr( | ) Pr( | ) Bel( )t t t t t t tx z x x x x dx state at time t

observation at time t

system dynamics

Technical Foundations

Hidden Markov models– Mathematical framework for describing processes

with hidden state that must be inferred from observations

Hierarchical plan networks– Represents how a task can be broken down into

subtasksHierarchical hidden Markov models*– Key to climbing food-chain!

* Precisely speaking: factorial hierarchical hidden semi-Markov models

Video Door Sensor Motion

Location

Example

Enter PO

Wait in line

Let go package

Pay cashier

Exit PO

Mail Package

Video Door Sensor Motion

Location

Enter PO

Go to PO

boxes

Open PO box

Pick up mail

Exit PO

Retrieve Mail

Example

Video Door Sensor Motion

Location

Mail Package

PO Patron

Retrieve Mail

Outside PO

Example

Inexplicable Observations

Enter PO

Wait in line

Let go package

Pay cashier

Exit PO

Mail Package

Enter PO

Go to PO

boxes

Open PO box

Pick up mail

Exit PO

Retrieve Mail

Enter PO

Let gopackage

Exit PO

Absolute Timing Constraints

Mail Package active [9 am – 4 pm]

Enter PO

Retrieve Mail active [6 am – 8 pm]

Enter PO

Relative Timing Constraints

Go to PO

boxes

Open PO box

Retrieve Mail

Timeout

seconds

seconds

Forgot combo?Safecracking?

Summary

Commonsense knowledge base of “significant” behaviors– Hierarchically organized– Probabilistic at all levels– Many parallel ongoing activities possible– Absolute and relative timing constraints– Probabilities “tuned” by machine learning techniques for

individual users– Inexplicable observations and failure modes – points of

possible intervention

Interventions

Framework allows system to predict when an anomalous situation is likelyDifferent anomalies have different costs– Confused patron– Deliberate loitering– Planting bomb

Must avoid:

Deciding When to Intervene

(Horvitz 98)G = prediction that help is needed

Common Architecture

Activity Compass

Palm-based wireless GPS– No explicit programming – learns pattern of

transportation plans – Accesses user’s calendar, real-time bus information– Constantly tries to predict where user will go next, and

whether problems will arise– Proactive help:

• “Walk faster or you’ll miss the 9:15 bus!”• “Green St bus is late, suggest you take Elm St bus instead”

Substeps

Cleaning up GPS data– 3 meter accuracy– frequent signal loss– determine most likely path

Infer mode of transportationPredict when and where transitions in mode of travel will occurPredict points of possible failure

indoors

walk

bus

bikecar

Gathering Data

On Foot: Across Campus

By Bus: Across Seattle

Transition Prediction

Training Data:– 20,000 GPS readings gathered over 3 weeks

Inferring current mode– Input: current location, time, velocity– 98% accuracy (10 FCV)

Predicting next transition– Input: current mode, location, time, velocity– 97% accuracy (10 FCV)*

* Don is a very organized guy. Your accuracy may vary.

Predicting Transition Location

User Interface

Assisted Cognition

“Plan aware” systems to help people with cognitive disabilitiesNew project based at University of Washington – Computer Science & Engineering– UW Medical Center, ADRC– Collaborators: Intel, OGI, Elite Care

http://assistcog.cs.washington.edu/

Summary

Potential of widespread sensor networks just beginning to be tappedKey issue: interpreting data in terms of human behavior, plans, and goalsResearchers in data fusion, AI, and “ubicomp” coming together around a core set of representations and algorithms