A benchmark dataset to evaluate sensor displacement in activity

recognition

SAGAWARE 2012, Pittsburgh (USA)

Oresti Baños1, Attila Toth Máté2, Miguel Damas1, Héctor Pomares1,

Ignacio Rojas1, and Oliver Amft2 1Department of Computer Architecture and Computer Technology, CITIC-UGR, University of Granada, SPAIN

2ACTLab, Signal Processing Systems, TU Eindhoven, NETHERLANDS

EU Marie Curie Grant #264738 DG-Research Grant #228398

Problem statement

Collect a training dataset

Train and test the model

The AR system is “ready”

2

Problem statement

INVARIANT SENSOR SETUP (IDEALLY) GOOD RECOGNITION

3

Problem statement

SENSOR SETUP CHANGES RECOGNITION PERFORMANCE MAY DROP

4

Problem statement

SENSOR SETUP CHANGES

• Technical anomalies (decalibration, battery failure, etc.) ‘Easy’ to model

• Sensor displacement Hardly synthesizable

5

Problem statement

SENSOR SETUP CHANGES

• Technical anomalies (decalibration, battery failure, etc.) ‘Easy’ to model

• Sensor displacement Hardly synthesizable

6

Concept of sensor displacement

• Categories of sensor displacement

– Static: position changes can remain static across the execution of many activity instances, e.g. when sensors are attached with a displacement each day

– Dynamic: effect of loose fitting of the sensors, e.g. when attached to cloths

• Sensor displacement new sensor position signal space change

• Sensor displacement effect depends on

– Original/end position and body part

– Activity/gestures/movements performed

– Sensor modality (ACC, GYR, MAG)

7

Sensor displacement = rotation + translation (angular displacement) (linear displacement)

Sensor displacement in AR: related work

• Features invariant to sensor displacement

– Heuristic method which achieved higher recognition rates for within a body part sensor displacement (Kunze08)

– Genetic algorithm for feature selection (Förster09a)

• Feature distribution adaptation

– Covariate shift unsupervised adaptation based on EM (Bayati09)

– Online-supervised user-based calibration (Förster09b)

• Sensor fusion

– Output classifiers correlation (Sagha11)

– Hierarchical-weighted classification (Banos12)

K. Kunze and P. Lukowicz. Dealing with sensor displacement in motion-based onbody activity recognition systems. In 10th international conference on Ubiquitous computing, pages 20–29, Seoul, South Korea, September 2008.

K. Förster, P. Brem, D. Roggen, and G. Tröster. Evolving discriminative features robust to sensor displacement for activity recognition in body area sensor networks. In Intelligent Sensors, Sensor Networks and Information Processing (ISSNIP), 2009 5th International Conference on, pages 43–48, dec. 2009.

H. Bayati, J. del R Millan, and R. Chavarriaga. Unsupervised adaptation to on-body sensor displacement in acceleration-based activity recognition. In Wearable Computers (ISWC), 2011 15th Annual International Symposium on, pages 71–78, june 2011.

K. Förster, D. Roggen, and G. Tröster. Unsupervised classifier self-calibration through repeated context occurences: Is there robustness against sensor displacement to gain? In Proc. 13th IEEE Int. Symposium on Wearable Computers (ISWC), pages 77–84, Linz, Austria, September 2009. IEEE Press.

H. Sagha, J. R. del Millán, and R. Chavarriaga. Detecting and rectifying anomalies in Opportunistic sensor networks 8th Int. Conf. on Networked Sensing Systems, IEEE Press, 2011, 162 - 167

O. Banos, M. Damas, H. Pomares, and I. Rojas. On the Use of Sensor Fusion to Reduce the Impact of Rotational and Additive Noise in Human Activity Recognition. Sensors, 2012, 12, 8039-8054

8

Study of sensor displacement

• Analyze

– Variability introduced by sensors self-positioning with respect to an ideal setup

– Effects of large sensor displacements

• Scenarios

– Ideal-placement

– Self-placement

– Mutual- or induced-displacement

Ideal Self Mutual

9

Dataset: activity set

• Activities intended for: – Body-general motion: Translation | Jumps | Fitness

– Body-part-specific motion: Trunk | Upper-extremities | Lower-extremities

10

Dataset: Study setup

• Cardio-fitness room

• 9 IMUs (XSENS) ACC, GYR, MAG

• Laptop data storage and labeling

• Camera offline data validation

http://crnt.sourceforge.net/CRN_Toolbox/Home.html 11

Dataset: Experimental protocol

• Scenario description

• Protocol

Round Sensor Deployment #subjects #anomalous sensors

1st Self-placement 17 3/9

2nd Ideal-placement 17 0/9

- Mutual-displacement 3 {4,5,6 or 7}/9

12

Preparation phase (sensor positioning & wiring, Xsens-Laptop

bluetooth connection, camera set up)

Exercises execution (20 times/1 min. each)

Battery replacement, data downloading

Data postprocessing (relabeling, visual

inspection, evaluation)

Round

Dataset: Experimental protocol

• Scenario description

Shading spots identify the de-positioned sensors for each subject

Self-placement Mutual-displacement

13

Sensor displacement evaluation

• Considerations

– Domain: ACC (X,Y,Z)

– Features: MEAN and STD

– ALL sensors

• Statistical analysis

• Classification

14

Sensor displacement evaluation

• Considerations

– Domain: ACC (X,Y,Z)

– Features: MEAN and STD

– ALL sensors

• Statistical analysis

• Classification

RCIDEAL LCIDEAL= LCSELF

15

RCSELF

Sensor displacement evaluation (statistical analysis)

• Considerations

– Domain: ACC (X,Y,Z)

– Features: MEAN and STD

– ALL sensors

• Normalized variance along the features PC across all subjects:

Ideal-placement

Activitie

s

Features PCA

5 10 15

5

10

15

20

25

30

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Self-placementA

ctivitie

s

Features PCA

5 10 15

5

10

15

20

25

30

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

16

See the paper for the mutual-displacement!

Sensor displacement evaluation (statistical analysis)

• Considerations

– Domain: ACC (X,Y,Z)

– Features: MEAN and STD

– ALL sensors

• Average variance marginalized over the activities and feature dimensions:

Ideal Self Mutual (4) Mutual (5) Mutual (6) Mutual (7)0

0.2

0.4

0.6

0.8

1

1.2

1.4x 10

-3Average variance (subjects 2, 5 and 15 and all activities)

Ideal Self-Placed0

0.2

0.4

0.6

0.8

1

1.2x 10

-3 Average variance (all subjects and activities)

17

Sensor displacement evaluation (classification)

• Considerations

– Domain: ACC (X,Y,Z)

– Features: MEAN and STD

– ALL sensors

• Classification

– Ideal: 5-fold cross validated, 100 times

– Self/Mutual: tested on a system trained on ideal-placement data

– 6 sec. sliding window

NCC KNN DT0

20

40

60

80

100

Accura

cy (

%)

Ideal Self Mutual(7)

18

Conclusions and future work

• CONTRIBUTIONS – Concept for categorising inertial sensor displacement conditions

• Ideal default deployment • Self-placement reflects a users perception of how sensors could be

attached • Mutual-displacement could represent boundary conditions for

recognition algorithms – A dataset to compare performance of different methods and conditions

• With the large set of annotated activities, the dataset will lend itself primarily for activity classification problems

• A wide variety of sensors and modalities were considered for displacements to capture potential effects on a recognition methods’ feature extraction and recognition

– An statistical and classification performance analysis • Demonstrates the shift into the feature space due to sensor displacement • Shows the performance drop from the ideal to the self-placement and

mutual-displacement conditions

19

Conclusions and future work

• NEXT STEPS

– Evaluate the state-of-the-art methods and techniques proposed to deal with sensor displacement

– Propose new methodologies that may be eventually tested on this dataset

– Share the dataset with the community allowing for the benchmarking of the current and future solutions in the regard of sensor displacement (available at http://www.ugr.es/~oresti/datasets)

20

Thank you for your attention. Questions?

Oresti Baños Legrán Dep. Computer Architecture & Computer Technology

Faculty of Computer & Electrical Engineering (ETSIIT) University of Granada, Granada (SPAIN)

Email: oresti@ugr.es Phone: +34 958 241 516 Fax: +34 958 248 993

Work supported in part by the HPC-Europa2 project funded by the European Commission - DG Research in the Seventh Framework Programme under grant agreement no. 228398 and by the EU Marie Curie Network iCareNet under grant no. 264738 and the FPU Spanish grant AP2009-2244. We want to specially thank the participants who helped us to collect this dataset.

21