1. Coupling Eye-Motion andEgo-Motion Features forFirst-Person Activity Recognition

2. First-Person Activity Recognition Recognize tasks from[Sun 09]wearable sensors Cameras, accelerometers, and IMUs[Fathi 11] Previous: kinematic motions Changes in attention couldprovide another source ofinformation 2 3. Previous WorkActivity recognition using changes in attention1.Eye-motion First person activity recognition using eye-motion [Bulling 11] Independent use of eye-motion cannot representbroader changes of attention.Reading a book electrooculography3 4. Previous WorkActivity recognition using changes in attention1.Eye-motion 2.Ego-motionReading a book Copying text from one display to another4 5. Eye-motion Characteristics [Bulling 11] Gaze-x Micro level Intermittent jumps Saccade Macro level Activities involves characteristicsaccade sequences Histogram of motion sequence WordbookTime 6 6. Eye-motion and Ego-motionEye-motionRight Reading a book LeftRightEgo-motion Copying text fromone display toanother LeftBoth of two motions haveIntermittent motion primitives and characteristic sequences7 7. Our ApproachOur method: Inside-out camera system provideeye-motion and ego-motion at once.Eye-motionEgo-motion 8 8. Overview Inside OutsideGaze coordinatesFramesMotion ExtractionEye-motionEgo-motion WordbookAnalysisWordbook Feature Classification12 9. Overview Inside OutsideGaze coordinatesFramesMotion Saccade ExtractiondetectionEye-motionEgo-motion WordbookAnalysisWordbook Feature Classification12 10. Eye-motion Primitives [Bulling 11]Inside Gaze coordinates Saccade detectionEye-motion PrimitiveTime Threshold & QuantizationEye-motion CharactersNOUAB Kr b Mn u bL l rCRK j d f ETimeJ HDG F11 11. Eye-motion Primitives [Bulling 11]Inside Gaze coordinatesSaccade detectionEye-motion Primitive (Saccade) Time Threshold & QuantizationEye-motion Characters NOUAB K r b Mn u bb C L l r r RKK j d f J HDG E F Time 11 12. Overview Inside OutsideGaze coordinatesFramesMotion Saccade Global Extractiondetection Optical flowEye-motionEgo-motion WordbookAnalysisWordbook Feature Classification12 13. Ego-motion PrimitivesOutside FramesGlobal optical flowEgo-motion Primitive Time Threshold & QuantizationEgo-motion CharactersNOUABMn u bCU rb r AL l rK j d fRE Time J HDG F11 14. Overview Inside OutsideGaze coordinatesFramesMotion Saccade Global Extractiondetection Optical flowEye-motionEgo-motion Wordbook n-gramWordbooksAnalysisWordbook Feature Classification12 15. Generating Wordbook [Bulling 11]Motion primitives U rb rA TimeWordbook features Current FrameRetrieve 1-4 gram wordbooks 1-gram Word count r100 L50 H1 16 16. Generating Wordbook [Bulling 11]Motion primitives U rb rA TimeWordbook features Current FrameRetrieve 1-4 gram wordbooks 1-gram4-gram Word count Wordcount r100 rrrr5 L50 Lrrr2 H1 lrlr1 17 17. Generating Wordbook [Bulling 11]Motion primitives U rb rATimeWordbook features Current FrameRetrieve 1-4 gram wordbooks 1-gram4-gramStatistical feature Word count Wordcountmax r100 rrrr5 L50 Lrrr2size H1 lrlr1range, var, mean 18 18. Generating Wordbook [Bulling 11]Motion primitives U rb rA TimeWordbook features Current FrameRetrieve 1-4 gram wordbooks 1-gram4-gramStatistical feature Word count Wordcountmax r99rrrr5 L50 Lrrr2size H1 lrlr1range, var, mean 19 19. Generating Wordbook [Bulling 11]Motion primitives U r b rA TimeWordbook features Current FrameRetrieve 1-4 gram wordbooks 1-gram4-gram Statistical feature Word count Wordcount max r100 rrrr5 L49 Lrrr3 size H1 lrlr1 range, var, mean20 20. Overview InsideOutsideGaze coordinates FramesMotion Saccade Global Extractiondetection Optical flowEye-motionEgo-motion Wordbook n-gramWordbooksAnalysisWordbook Feature2 types,4 wordbooks,5 features Classification=40dim feature Linear SVM12 21. Experiment1.READ 2.VIDEO 3.WRITE4.COPY5.BROWSE 123 4 51234 530 sec. 120 sec. 25 min. / subjectTrainingTest 5 subjects Case 1 Workspace 1 Workspace 2 2 sets / subject Case 2 Workspace 2 Workspace 122 22. Comparing the Feature Performance Saccade Wordbook (SW)Eye-motion feature [1] Motion Histogram (MH)Ego-motion feature [2] Motion Wordbook (MW) Ego-motion feature (proposed) GIST Visual feature [3] [1] Bulling, 11 [2] Kitani, 11 [3] Oliva, 01 19 23. Performance Over Tasks MOWORD feature improves the performance Joint feature improves the performance complementaly 20 24. SummaryJoint use of eye-motion and ego-motionimproves classification performanceFuture Work Apply to a wider range of tasks Ego-motion feature might do more on dynamic tasks Combined of object detection Hand, face, or other objects