My Smartphone Knows I am Hungry

Fanglin Chen, Rui Wang, Xia Zhou , Andrew T. Campbell

Dartmouth College

Logging by Input

2014-06-17 2https://jawbone.com/up

2014-06-17 3

Use Devices to Track Eating

2014-06-17 4

HapiForkhttp://www.hapi.com/products-hapifork.asp

Use Devices to Track Eating

2014-06-17 5

Yatani et al. BodyScope Ubicomp 2012

Rahman et al. BodyBeat, MobiSys 2014

Need Specialized Hardware

2014-06-17 6Photo:gtresearchnews.gatech.edu

Can your smartphone unobtrusivelypredict your food eating behavior?

2014-06-17 7

Can your smartphone unobtrusivelypredict your food eating behavior?

2014-06-17 8

purchases

Our Work

unobtrusively collect behavior

data

build predictive model

predict buy food or not

2014-06-17 9

Dataset – Evidence

• 25 students, 10 weeks• Smartphone app runs 24/7 in the

background to collect:• Conversation• Physical activity• Sleep• Location• Bluetooth colocation• Wi-Fi scan log

2014-06-17 10

Dataset – Ground Truth

• We collected dinning records• Purchase time

• Location

• Account

• Cost

• We don’t know what they bought

2014-06-17 11

Purchase Time Location Account Type Cost04/11/2013 01:57:26 Novack Cafe Dining DBA Debit $3.55 04/11/2013 10:10:31 Collis Cafe Dining DBA Debit $7.45

Roadmap

• Motivation

• Design Challenges

• Approach

• Results

• Conclusion

2014-06-17 12

Why Not Just Use Location?

2014-06-17 13

2014-06-17 14

Not the Case for Campus

2014-06-17 15

Why Not Use Time?

2014-06-17 16

No Distinct Mealtimes

2014-06-17 17

Roadmap

• Motivation

• Design Challenges

• Our approach

• Results

• Conclusion

2014-06-17 18

Our Approach

• Unobtrusively monitor user behavior

• Identify food purchasing related behaviors

• Build a predictive model

2014-06-17 19

Prediction Pipeline

• Training phase

behavioral data

features classifierfeature

extraction

trainingpurchase

history

2014-06-17 20

Prediction Pipeline

• Prediction phase

behavioral data

features classifier

buy food or not

features

behavioral data

featureextraction

feature extraction

trainingpurchase

history

2014-06-17 21

Features Related to Food Purchase

• Location• Arrival time

• Location• Arrival time / Leaving time• Conversation• Physical activity

2014-06-17 22

Current LocationPrevious Location

Prediction Pipeline

• Prediction phase

behavioral data

features classifier

buy food or not

features

behavioral data

featureextraction

feature extraction

trainingpurchase

history

2014-06-17 23

Approach: Training Set Selection

• Select training data based on

One size does not fit all. Behavior changes over time.

personalization adaptation.and

2014-06-17 24

Use latest n weeks data to train

week 1 week 2 week 3 week 4 week 5 week 6 week 7

Prediction Pipeline

• Prediction phase：

behavioral data

features classifier

buy food or not

features

behavioral data

featureextraction

feature extraction

trainingpurchase

history

2014-06-17 25

Approach: Classifier

• Classification and Regression Trees (CART)

• Handles categorical data (location) and numerical data (behavioral)

2014-06-17 26

decision tree

Roadmap

• Motivation

• Design Challenges

• Approach

• Results

• Conclusion

2014-06-17 27

Evaluation Setup

• Schemes for comparison• Random guessing: flip a coin

• Generic: one size fits all model

• Personalized: personalized model but without adaptation

• Our model: personalization and adaptation

• Metrics• Accuracy: % of correct predictions

• Precision: % of correct predictions in all positive predictions

• Recall: % of identified real true cases2014-06-17 28

50.50%

68.60%73.90% 74.21%

26.60%

42.10%

49.50%52.68%50.40% 49.30%

53.60% 55.12%

0%

25%

50%

75%

Random Guessing Generic Personalized Our ModelAccuracy Precision Recall

Key Result

• Random guessing < generic < personalized < our model (personalized and adapted)

2014-06-17 29

Roadmap

• Motivation

• Design Challenges

• Approach

• Results

• Conclusion

2014-06-17 30

Conclusion and Limitation

• Proposed a simple predictive model of food purchases• 74% accuracy using smartphone sensing data

• Limitations• Cannot predict what food student would buy

• Do not cover food purchases beyond student ID card

• Do not know when they ate their food

• Limited to student body

2014-06-17 31

Future Work

2014-06-17 32

• How can we unobtrusively track what we bought?

Future Work

• How to generalize the model?

2014-06-17 33

Future Work

• How to unobtrusively detect eating?

2014-06-17 34

Future Work

• Food intervention

2014-06-17 35

Thanks!

2014-06-17 36

Backup

2014-06-17 37

What Features are more predictive

0

0.05

0.1

0.15

0.2

0.25

0.3

Current location Arrival time Prev. locationdeparture time

Physical activity Prev. locationarrival time

Conversationduration

Prev. location Conversation freq.

Feat

ure

imp

ort

ance

Features2014-06-17 38