1September 2010

Activity Recognition and Biometric Identification

Using Cell Phone

AccelerometersWISDM Project

Department of Computer & Info. ScienceFordham University

2September 2010

We are Interested in WISDM

WISDM: WIreless Sensor Data Mining Powerful portable wireless devices are

becoming common and are filled with sensors

Smart phones: Android phones, iPhone Music players: iPod Touch

Sensors on smart phones include: Microphone, camera, light sensor,

proximity sensor, temperature sensor, GPS, compass, accelerometer

3

WISDM Data Mining Problems

Completed initial stages of research on 2 tasks: Activity Recognition

What is the user doing? Biometric Identification

Who is the user? Is the user who they claim to be?

Future tasks GPS mining: learn about user routes &

interactions Use cell phones as a sensor network to

learn about the environment

September 2010

4September 2010

Accelerometer-Based Activity Recognition

The Problem: use accelerometer data to determine a user’s activity

Activities include: Walking and jogging Sitting and standing Ascending and descending stairs More activities to be added in future

work

5September 2010

Applications of Activity Recognition

Health Applications Generate activity profile to monitor

overall type and quantity of activity Parents can use it to monitor their

children Can be used to monitor the elderly

Make the device context-sensitive Cell phone sends all calls to voice mail

when jogging Adjust music based on the activity

Broadcast (Facebook) your every activity

6

Accelerometer-Based Biometric Identification

The Problem: Use accelerometer data to identify an individual user Identity prediction: map a user to one of

a set of predetermined users Authentication: determine whether a

user is who they claim to be

September 2010

7

Applications of Biometric Identification

Security & theft prevention of mobile devices

Automatic personalization of mobile devices For example, send all calls to voicemail

when jogging Identify user and load proper settings

General Security Applications Should the user be in this location? Can be used as a second level of

securitySeptember 2010

8September 2010

Our WISDM Platform

Platform based on Android cell phones Android is Google’s open source mobile

computing OS Easy to program, free, will have a large

market share Android phones now outselling iPhones

Unlike most other work on activity recognition: No specialized equipment Single device naturally placed on body (in

pocket)

9September 2010

Our WISDM Platform

Current research was conducted off-line Data was collected and later analyzed

off-line Now updating our platform to operate

in real-time In June we released real-time sensor

data collection app to Android marketplace Currently collects accelerometer and

GPS data and transmits it to our server

10September 2010

Accelerometers

Included in most smart phones & other devices All Android phones, iPhones, iPod

Touches, etc. Tri-axial accelerometers that measure 3

dimensions Initially included for screen rotation

and advanced game play

11September 2010

Examples of Raw Data

Next few slides show data for one user over a few seconds for various activities

Cell phone is in user’s pocket Earth’s gravity is registered as

acceleration Acceleration values relative to axes

of the device, not Earth In theory we can correct this given that

we can determine orientation of the device

12September 2010

Standing

13September 2010

Sitting

14September 2010

Walking

15September 2010

Jogging

16September 2010

Descending Stairs

17September 2010

Ascending Stairs

18September 2010

Data Collection Procedure

User’s move through a specific course Perform various activities for specific

times Data collected using Android phones Activities labeled using our Android app

Data collection procedure approved by Fordham Institutional Review Board (IRB)

Collected data from ~35 users (will increase)

19September 2010

Data Preprocessing

Convert time series data into examples so we can use standard classifiers (e.g., decision trees)

Use a 10 second example duration/window 3 acceleration values every 50 ms (600

total values) Generate 43 total features

Ave. acceleration each axis (3) Standard deviation each axis (3) Binned/histogram distribution for each

axis (30) Time between peaks (3), Ave resultant

acceleration (1)

20

ACTIVITY RECOGNITION RESULTS

September 2010

21September 2010

Activity Recognition Final Data SetID Walk Jog Up Down Sit Stand Total

1 74 15 13 25 17 7 151 2 48 15 30 20 0 0 113 3 62 58 25 23 13 9 190 4 65 57 25 22 6 8 183 5 65 54 25 25 77 27 273 6 62 54 16 19 11 8 170 7 61 55 13 11 9 4 153 8 57 54 12 13 0 0 136 9 31 59 27 23 13 10 163

10 62 52 20 12 16 9 171 11 64 55 13 12 8 9 161 12 36 63 0 0 8 6 113 13 60 62 24 15 0 0 161 14 62 0 7 8 15 10 102 15 61 32 18 18 9 8 146 16 65 61 24 20 0 8 178 17 70 0 15 15 7 7 114 18 66 59 20 20 0 0 165 19 69 66 41 15 0 0 191 20 31 62 16 15 4 3 131 21 54 62 15 16 12 9 168 22 33 61 25 10 0 0 129 23 30 5 8 10 7 0 60 24 62 0 23 21 8 15 129 25 67 64 21 16 8 7 183 26 85 52 0 0 14 17 168 27 84 70 24 21 11 13 223 28 32 19 26 22 8 15 122 29 65 55 19 18 8 14 179

Sum 1683 1321 545 465 289 223 4526 % 37.2 29.2 12.0 10.2 6.4 5.0 100

22September 2010

Data Mining Step

Utilized three WEKA learning methods Decision Tree (J48) Logistic Regression Neural Network

Results reported using 10-fold cross validation

23September 2010

Summary Results

24September 2010

J48 Confusion Matrix

Predicted Class

Walk Jog Up Down Sit Stand

Actual

Class

Walk 1513 14 72 82 2 0

Jog 16 1275 16 12 1 1

Up 88 23 323 107 2 2

Down 99 13 92 258 1 2

Sit 4 0 2 3 270 3

Stand 4 1 2 7 1 208

25September 2010

Activity Recognition Conclusions

Able to identify activities with good accuracy Hard to differentiate between

ascending and descending stairs. To limited degree also looks like walking.

Can accomplish this with a cell phone placed naturally in pocket

Accomplished with simple features and standard data mining methods

26

BIOMETRIC IDENTIFICATION

RESULTS

September 2010

27

Biometric Identification Data Sets

We evaluated 6 data sets (4 activities) Aggregate (all 6 activities without class

labels) Walking Jogging Ascending Stairs Descending Stairs Aggregate-Oracle (all 6 activities with

class labels) The unlabeled aggregate data set is

most realistic

September 2010

28

# of Examples per User/Activity

September 2010

ID Walk Jog Up Down Total 1 74 15 13 25 127 2 48 15 30 20 113 3 62 58 25 23 168 4 65 57 25 22 169 5 65 54 25 25 169 6 62 54 16 19 151 7 61 55 13 11 140 8 57 54 12 13 136 9 31 59 27 23 140

10 62 52 20 12 146 . . . . . .

30 35 31 28 19 113 31 64 55 17 16 152 32 34 32 0 0 66 33 64 0 0 0 64 34 59 59 0 0 118 35 55 46 19 12 132 36 87 81 23 16 207

Sum 2081 1625 632 528 4866 % 42.8 33.4 13.0 10.8 100

29

Accuracy for Person Identification (based on 10-

Second Examples)

September 2010

J48: Decision Tree LearnerStraw Man: Strategy of always predicting the most common user

30

Aggregate Data Set Confusion Matrix

September 2010

(Results for first 14 users only)

31

We Can Do Better: Majority Scheme

We know which records come from the same cell phone user

So predict the users identity based on the identity predicted most often

September 2010

32

Results Using Majority Voting

September 2010

Aggregate Walk Jog Up Down Aggregate (Oracle)

J48 36/36 36/36 31/32 31/31 28/31 36/36 Neural Net 36/36 36/36 32/32 28.5/31 25/31 36/36

33

Ratio of Records Correctly Classified to Most

Successful Imposter

September 2010

User J48 Neural Net

User J48 Neural Net

1 14:1 12:1 19 24:1 9:1 2 21:1 15:1 20 10:1 2:1 3 15:1 27:1 21 34:1 5:1 4 21:1 4:1 22 24:1 23:1 5 25:1 15:1 23 4:1 3:1 6 30:1 21:1 24 22:1 49:1 7 11:1 10:1 25 26:1 18:1 8 15:1 18:1 26 20:1 7:1 9 27:1 23:1 27 16:1 9:1

10 8:1 5:1 28 17:1 16:1 11 17:1 13:1 29 17:1 7:1 12 25:1 4:1 30 24:1 2:1 13 16:1 16:1 31 18:1 16:1 14 5:1 20:1 32 6:1 18:1 15 16:1 46:1 33 14:1 5:1 16 17:1 13:1 34 30:1 26:1 17 21:1 17:1 35 6:1 7:1 18 17:1 49:1 36 13:1 12:1

34

Authentication Results

Positive authentication rate: % of test examples coming from a user that are correctly classified as belonging to that user

Negative authentication rate: % of test examples from an imposter that are correctly identified as not belonging to the userSeptember 2010

35

Biometric Identification Conclusions

Very accurate models for person identification using data mining of accelerometer data

Generally perfect performance for identification when using majority scheme

Can get good biometric results without knowing the specific activity the user is performing

September 2010

36September 2010

Related Work At least a dozen papers on activity

recognition using multiple sensors, mainly accelerometers Typically studies only 10-20 users

Activity recognition also done via computer vision

Actigraphy uses devices to study movement Used by psychologists to study sleep

disorders, ADD A few recent efforts use cell phones

Yang (2009) used Nokia N95 and 4 users Brezmes (2009) used Nokia N95 with real-time

recognition One model per user (requires labeled data from each

user)

37September 2010

Future Work

Add more activities and users Add more sophisticated features Try time-series based learning

methods Deploy higher level applications:

activity profiler Can be used to encourage healthy

behaviors Can benefit the young and the elderly

38

Future Work (cont.)

Generate results in real time Activity recognition

Uses a universal model so no need to train per user

Send results to server and get response back and on Web

Alternatively do everything on the phone Biometric Identification

Need a model per user so would need to train model

Not too hard, just collect unlabelled activity dataSeptember 2010

39

Future Work (cont.)

GPS Data Mining Find spatio-temporal patterns from GPS

data Location based on time of day, day of

week Identify friends and people you spend

time with Identify things about the environment

Where are the pedestrian paths on campus? How busy is the cafeteria and when? Where do people congregate …

September 2010

40

For More Information

See the WISDM web site: http://storm.cis.fordham.edu/~gweiss/

wisdm/ See the two published papers on

activity recognition and biometric identification

Meetings usually Thursday at 6:30 pm Talk to me There are quite a few benefits to

undergraduate research!September 2010

41September 2010

Thank You

Questions?