Recognizing C omplex Human A ctivities – From Top Down to Bottom Up and Back

Recognizing Complex Human Activities –From Top Down to Bottom Up and Back

Dr.-Ing. Ulf Blanke Wearable Computing Lab | ETH ZürichSamsung Jul 25, 2014

Vita

2001-2006 Dipl. (M.Sc.) Informatik, TU Darmstadt

2007-2011 PhD, Multimodal Interactive Systems Group, TU Darmstadt- 3y scholarship, German Research Foundation- Prof. Dr. Bernt Schiele

Post-Doc, Max Planck Institut for Informatics, Saarbrücken- Computer Vision and Multimodal Computing

2011-2012 Senior Researcher at AGT International (R&D Division)- Integrated safety and security solutions- Headquarter: Switzerland, R&D: Darmstadt

2012 ... Senior Scientist and Pioneer Fellow at ETH-Z - Wearable Computing Lab, Prof. Gerhard Tröster

Ulf Blanke | Recognizing complex human activities – From Top Down to Bottom Up and Back | 2

Overview


Composite Activities- Challenges - Discovering and combining relevant events (LoCA09)- Transfer and recombine relevant events (ISWC10)

Overview of current projects

……

t t

1 2 3 1 2 3

Composite Activities


time

Recognizing composite activitiesby decomposition into isolated activity events Excellent work addressing isolated activity recognition

Only little work on composite activities

Data from wearable sensors

ChallengesAtomic activity events


t

1. Inner-class variability and intra-class similarity2. Large corpus of irrelevant and ambiguous data

Screwing

Drilling

Variabilitywithin activity

Similarity acrossdifferent activities

Screwingtime

ChallengesComposite Activities


1. Variation (duration, irrelevant data, e.g. by interruptions)2. Changing order of underlying activity events

timeOther challenges laterTowards less supervision

Recognizing Composite ActivitiesOne way of doing it

Layer 2 …

…

Layer 3…

…Composite

t

Layer L1

Data


Compositeactivities

Sensor data

Which low level events are important for composite activities?

Learning(automatic Selection)

walking eating eating

dinnerlunch

Recognizing Composite ActivitiesLearning relevant events

picking up food

Prepfood

Doing dishes


eating eating

Compositeactivities

Recognizing composite activities by activity spotting feasible?

Activity Spotting

dinnerlunch

Recognizing Composite ActivitiesSpotting and combining relevant events

Sensor data

walkingpicking up food

Prepfood

Doing dishes


Can we learn distinctive parts of composite activities?

Can we gain computational efficiency by reducing todata important for recognition?

Research QuestionsActivity spotting for composite activities

1

2


Approach

compositeactivities

Histogram calculationK-means clustering

Joint boosting

Feature-CalculationSensor data


Low Level Activity Selection (Joint)Boosting

(1)Combinationof low level activities to infer high-level activities

(2) Automatic Selection of most discriminative low level activitiesBoosting (Friedman2000)

(3) Sharing features(i.e. low level activities) across high level activities

JointBoosting (Torralba2004)

+

others

lunch

lunch

dinner

lunch dinner


Experimental SetupDataset

7 days of a life from a single person (Huynh08) Two layers of annotation

4 high level routines, more than 20 low level activities

Pocket

Wrist

working working dinner

lunch commutingcommuting

2 acceleration sensors

walkingstandingin line

having a coffee

Lunch

walkingeating


Experimental SetupFixed Parameters

High-level activities

Low level activities

Sensor data

Doing dishes

Feature-Calculation

K-means clusters

Joint boosting

Mean and Variance - over 0.4s window- on (x,y,z)-acceleration- of pocket and wrist

Histograms- over 30min

window


Results

80 10 4 (Huynh08)Precision in % 88 83 83 77Recall in % 90 88 84 66Data used in % 74 18 13 100

Cluster Soft Assignments

80 10 486 77 7390 82 8345 5 2

Cluster Hard Assignments

Amount of data used

Recall

Precision

Number of classifiers204060801000

1020

90

80

7060

504030

100

30

01020

9080

7060

5040

100

204080100 60

in %

in %

Number of classifiers

Observed data reduced dramatically at superior performance


DinnerCommuteLunch

sitting/desk activities (47.24%)

driving car (32.90%),

driving car (21.71%)

Time

walking (99.23%)

sitting / desk activities (97.86%)

walking (96.09%)

driving bike (47.86%) walking (22.51%) picking up food (16.81%)

queuing in line (43.86%) picking up food (14.59%)

driving bike (16.76%)

sitting/desk activities (31.20%)

3664248

291353

Lunch WorkCommuteDinner

Time

Time

Distribution of low level labels for clustering

ResultsWhich low level activities are used?


Overview




……

t t

1 2 3 1 2 3

Composite activitiesKnowledge transfer

Work on knowledge transfer (Zheng09), (Kasteren10), (Banos12), … different aspects of transferring knowledge

Here: “Partonomy” (Miller&Johnson-Laird76, Tversky90…) Borrowed from object perception relationship between sub-parts

Composite C2Composite C1

New Composite?


Research Questions

Does a partonomy-approach improve state of the art?

Can we transfer knowledge of activity events to learn and recognize new activities with minimal training?

Can we use composition knowledge to improve recognition of underlying activity events?

For composite activity recognition…

1

2

3


L2-Composite L2-CompositeLayer 2 …L2-Composite L2-Composite

…

L3-CompositeLayer 3 L3-Composite ……

Layer nLn-Composite

t

Bottom-up “construction” to hierarchy of multiple layers

Layer L1

DataFrom step 1: scores x

Partonomy-Based Activity RecognitionLn-Composite Activity Modeling


Raw data stream + Segm.

(2) Classifier

(1) Feature Calculation(e.g. mean, var, FFT)*

Sensors

Spotting atomic activitiesPipeline

Central time t of segment Normalized confidences of event classes U per segment s

Groundtruth


Ln-compositeactivity

x1x0

z0 z1

y

Unary potentials: scores of individual events

Pairwise potentials:Co-occurrence of relevant events (temporal dist. and class of event)

Probability for composite model

The right events, combined at the right time

y y

z0 z0z1 z2 z2z1 z3

x2x1 x2x0 x0 x1 x3

t

Step 2: Ln-Composite Activity ModelingConditional Random Field

L1-activityevents


Output of Step 2

Before Non-Maximum surpression


ExperimentsBookshelf Dataset

5 Xsens IMU’s at upper body

10 subjects 6 L2-composite activities

Make back partJoin 2 parts Assemble box …

…L1

L2

… …


Partonomy vs. single-layer for L2-composites Joint boosting, sliding window Leave-one-subject out cross-validation

Reduction: Single-layer:-

10% Partonomy: -2%

Training samples

Avg.

EER

for a

ll 6

class

es

83%

62%

85%72%

ExperimentsResults on Bookshelf-Dataset

vs.


Research Questions

Does a partonomy-approach improve state of the art?

Can we transfer knowledge of activity events to learn and recognize new activities with minimal training?

Can we use composition knowledge to improve recognition of underlying activity events?

For composite activity recognition…

1

2

3


ExperimentsBookshelf Dataset

5 Xsens IMU’s at upper body

10 subjects 6 L2-composite activities

Make back partJoin 2 parts Assemble box …

…L1

L2

… …


ExperimentsTarget-Dataset Mirror

5 Xsens IMU’s at upper body 6 subjects 10 L1-events, 6 L2-composites and

4 L3-composites

……

prepare backside 2Join 2 parts ……

Finish backpart

Prepare frames 14x

L2

L3

…

…

L1


Hanging up on wall

Mark Drill Screw Hang up26% 50% 99% 50%

100%

Fix side frame

Mark Drill hammer26% 50% 65%

90 %1 - E

qual

err

or ra

teExperimentsL2-composite activities

With transferred activity event detectorscomposite activity recognition possible with minimal training

71%


Contribution and Conclusion

Discovering and combining events (LoCA09) automatic discovery of relevant parts using Joint boosting Efficient method, outperforms approach using all data

Transferring and recombining events (ISWC10) Outperforms direct approach Knowledge transfer possible Improves lower level recognition

Ulf Blanke | Recognizing Composite Human Activity | 36

Today…

User Independent, Multi-Modal Spotting of Subtle Arm Actions with Minimal Training Data. G. Bauer, U. Blanke, P. Lukowicz, and B. Schiele. 10th Percom Workshop (ComoRea 2013). IEEE

South by South-East or sitting at the desk. Can orientation be a place?U. Blanke, R. Rehner and B. Schiele, (ISWC 2011). IEEE

Remember and Transfer what you have Learned - Recognizing Composite Activities based on Activity Spotting.U. Blanke and B. Schiele, (ISWC 2010), IEEE.

Towards Human Motion Capturing using Gyroscopeless Orientation Estimation.U. Blanke and B. Schiele, (ISWC 2010), IEEE.

Visualizing Sleeping Trends from Postures.M. Borazio, U. Blanke and K. Van Laerhoven, (ISWC 2010), IEEE.

All for one or one for all? – Combining Heterogeneous Features for Activity Spotting.U. Blanke, B. Schiele, M. Kreil, P. Lukowicz, B. Sick and T. Gruber (CoMoRea in conj. with Percom 2010), IEEE.

An Analysis of Sensor-Oriented vs. Model-Based Activity Recognition.A. Zinnen, U. Blanke and B. Schiele, (ISWC 2009). IEEE.

Daily Routine Recognition through Activity Spotting.U. Blanke and B. Schiele, (LoCA 2009), Springer.

Sensing Location in the Pocket.U. Blanke and B. Schiele, (Ubicomp 2008, adjunct proceedings)

Scalable Recognition of Daily Activities with Wearable Sensors.Tâm. Huynh, U. Blanke and B. Schiele, (LoCA 2007), Springer.

Selected Publications

Overview




……

t t

1 2 3 1 2 3

Collective Crowd behavior

1M visitors

GPS

Supervised projects


Parkinson’s Disease Activities, travel purposes, places towards less supervision

Past projects

Place detection


Past projects


Sleep studies

Co-Supervision of PhD Students


Sinziana Mazilu Long-Van Nguyen-Dinh Zack Zhu

Development team (Project Züri Fäscht)

Sascha Negele

Tobias Franke

David Bannach

Robin Guldener

Torben Schnuchel

Enes Poyarez

Dominik Riehm

William Ross

KellyStreich

PhD Students

Thank you for your kind attention.

Date post:	22-Feb-2016
Category:	Documents
Upload:	dixie
View:	22 times
Download:	0 times

Recognizing C omplex Human A ctivities – From Top Down to Bottom Up and Back

Documents