1

Learning Temporal Relations in Smart Home Data

Vikramaditya Jakkula & Diane J. CookArtificial Intelligence LabWashington State University

2nd International Conference on Technology and Aging (ICTA) June 17, 2007

VJ AI@WSU © 2007 2

Index

Introduction Challenges for Elderly MavHome Project Experimentation Environment Temporal Relations Experimentation Process Experimentation Steps Results Conclusion & Future Work

VJ AI@WSU © 2007 3

Smart home for the elderly: Challenges

Health care• Tele-Health

and Health Monitoring

Smart Devices• Device Automation

such as smart watch to read blood pressure and other vital health parameters.

Assisted Living• Reminder

Assistant Systems

Learning and Adaptation• Prediction of activity

and detecting anomaly in activity.

• Lifestyle Patterns such as preferences of daily activities , such as exercise preferences.

Robotics• Virtual Pets• Robot enabled

wheelchair and more.

Human Computer Interface• Audio Video

based applications for event detection and analysis

VJ AI@WSU © 2007 4

MavHome: Smart Home Project

VJ AI@WSU © 2007 5

Experimentation EnvironmentMavHome Environment

MavLabMavKitchenMavPad

VJ AI@WSU © 2007 6

Experimentation Environment

MavLAB Argus Sensor Network

around 100 Sensors. include Motion, Devices, Light, Pressure, Humidity and more.

VJ AI@WSU © 2007 7

Datasets: Real vs. Synthetic

Real Dataset and synthetic datasets consist timestamp of the activity with the activity name and the state it is in.

Real Dataset (Sample):3/2/2003 12:40:0 AM, (Studio E) E9 OFF3/2/2003 2:40:0 AM, (Living Room) H9 ON3/2/2003 2:40:0 AM, (Living Room) H9 OFF3/2/2003 6:4:0 AM, (Living Room) H9 OFF3/3/2003 3:43:0 AM, (Studio C) C14 ON3/3/2003 3:43:0 AM, (Studio C) C15 ON3/3/2003 3:43:0 AM, (Studio C) C13 ON

Synthetic Dataset (Sample):2/1/2006 10:02:00 AM, off, oven2/1/2006 11:00:00 AM, on, lamp2/1/2006 11:11:00 AM, off, thermostat2/1/2006 12:02:00 PM, off, lamp2/1/2006 12:35:00 PM, off, cooker2/1/2006 1:30:00 PM, on, lamp2/1/2006 2:02:00 PM, off, fan

VJ AI@WSU © 2007 8

What is a temporal relation?

Food “Contains” Wateror

Water “Before” Pillsor

Food “Meets” Pillsor

Food “Contains” Water “before” Pills

Food Food

WaterWater

PillsPills

Time Interval

“It is common to describe scenarios using time intervals rather than time points” - James F. Allen

VJ AI@WSU © 2007 9

Allen’s 13 Temporal Relations

BeforeAfterDuringContainsOverlapsOverlapped-ByMeetsMet-byStartsStarted-ByFinishesFinished-ByEquals

VJ AI@WSU © 2007 10

Why Temporal Relations?

Reminder Assistance

• Reminder system based on temporal relations.

Anomaly Detection

• If Pills are to be taken “After” Food, we can notice violation of this activity!

Maintenance

• If Cooker is Spoiled should we call emergency or a normal repair?

Temporary Need Analysis

• If Oven used for Turkey, Is turkey at Home?

Improve Prediction

• Increase prediction accuracy with association rules!

VJ AI@WSU © 2007 11

Experimentation

Step 1: Identifying the most frequent activity using Apriori Algorithm for frequent itemset mining.

Step 2: Use Weight based Relation analysis on bounded Allen’s temporal relations to identify best relation to remove ambiguity.

VJ AI@WSU © 2007 12

Step 1:Apriori Algorithm Apriori Algorithm Pseudo-code: Ck: Candidate itemset of size kLk : frequent itemset of size kL1 = {frequent items};for (k = 1; Lk !=; k++) do begin Ck+1 = candidates generated from Lk; for each day t in datasets do increment the count of all candidates in Ck+1

that are contained in t Lk+1 = candidates in Ck+1 with min_support end return k Lk;

VJ AI@WSU © 2007 13

The Apriori Algorithm Example

Day Event23/10/2005 TV cooker lamp24/10/2005 Oven Cooker Fan25/10/2005 TV Oven Cooker Fan

26/10/2005 Oven Fan

Database D itemset sup.{TV} 2

{oven} 3{cooker } 3{lamp} 1{fan} 3

itemset sup.{TV} 2

{Oven} 3{cooker} 3

{fan} 3

Scan D

C1 L1

{TV Oven}{TV Cooker}

{TV Fan}{Oven Cooker}

{Oven Fan}

{Cooker Fan}

itemset sup{TV Oven} 1

{TV Cooker} 2{TV Fan} 1

{Oven Cooker} 2{Oven Fan} 3

{Cooker Fan} 2

itemset sup{TV Cooker} 2

{Oven Cooker} 2{Oven Fan} 3

{Cooker,Fan} 2

L2

C2C2

Scan D

C3 L3itemset{Oven Cooker Fan}

Scan D itemset sup{Oven Cooker Fan} 2

VJ AI@WSU © 2007 14

Experiment Parameters

Parameter settings for experimentation

Minimum Support vs. No of Itemset Identified

Datasets

Parameter Setting

No of Days

No of Events

No of Intervals Identified

Size of Data

Synthetic 60 8 1729 106KB

Real 60 17 1623 104KB

Minimum Support

No of Frequent Itemset

Real DatasetSynthetic Dataset

8 3 2

4 5 3

2 10 4

84

2

0

2

4

6

8

10

Real Dataset

No of Frequent Itemsets in Real vs. Synthetic datasets with different Mini-

mum Support

Real DatasetSynthetic Dataset

VJ AI@WSU © 2007 15

Step 2: Temporal Relation Formation

Table 3: Sample of Frequent Relation Pairs.

Use the above define temporal relations with the weight based rule given below to identify the best temporal relations.

VJ AI@WSU © 2007 16

Some Interesting Relations!

Frequent Relation Pairs on Real Datasets

Lamp Sensor J16 BEFORE Radio Sensor J11

Lamp Sensor I14 AFTER Lamp Sensor C9

Lamp Sensor I4 EQUALS Lamp Sensor I4

Frequent Relation Pairs on Synthetic Datasets (Daily basis)

Cooker Before Oven

Fan After Cooker

Lamp Before Cooker

These were few patterns found in the smaller datasets used for the experiment process.Help monitor interesting patterns and model behavior analysis.

VJ AI@WSU © 2007 17

Future Directions Prediction of activity. Anomaly detection mechanism using

temporal relations. Visualization temporal intervals for

monitoring daily activities and lifestyle.

VJ AI@WSU © 2007 18

Temporal Relations for Anomaly detection

Anomaly DetectionBefore After XDuring XContains Overlaps Overlapped-By XMeets Met-by XStarts Started-By Finishes Finished-By Equals

Event Sequence: X YEvidence = P(X|Y) = | Before (X,Y) + Contains(X,Y) +Overlaps(X,Y) + Meets(X,Y) + Starts(X,Y) +StartedBy(X,Y) + Finishes(X,Y) + FinishedBy(X,Y)+ Equals(X,Y) | / |Y|

Event Sequence: X A BP(B|AUX) = P(B ∩ (AUX) ) / P(AUX)= P(B ∩ A) U P(B ∩ X)/ P(A) + P(X) –P(A∩X)[Association Rule]= P(B|A).P(A) + P(B|X).P(B) / P(A) + P(X) –P(A∩X)[Multiplication Rule]

Anomaly = 1 – Evidence.

VJ AI@WSU © 2007 19

Temporal Relations for Prediction!

PREDICTIONBefore XAfter During Contains XOverlaps XOverlapped-By Meets XMet-by Starts Started-By Finishes Finished-By Equals

Prediction = Association rules + Probabilistic Model.

Probabilistic Model = P(Z|Y) = |After(Y,Z)| + |During(Y,Z)| + |

OverlappedBy(Y,Z)| + |MetBy(Y,Z)| + |Starts(Y,Z)| + |

StartedBy(Y,Z)| + |Finishes(Y,Z)| + |

FinishedBy(Y,Z)| + |Equals(Y,Z)| / |Y|

VJ AI@WSU © 2007 20

Conclusion

Unique approach for discovering patterns in smart home datasets.

Found interesting patterns which can help in prediction and anomaly detection in every day activities in smart home.

Can be used for developing time based reminder assistance system.

Can be modeled to handle patients with early onset of dementia.

Boon for elderly inhabitants enabling easy and comfort living with lifestyle analysis.

VJ AI@WSU © 2007 21

Questions!

Thank You