40
EEMCS - HMI 1 Dance Movement Patterns Recognition (Part I) Israel Solà Cerdán
EEMCS - HMI 1�
Dance Movement Patterns Recognition (Part I)
Israel Solà Cerdán
EEMCS - HMI 2�
AcknowledgmentsAcknowledgments•• This project would not has been possible without the This project would not has been possible without the
collaboration of our project directors: collaboration of our project directors: –– Anton Anton NijholtNijholt–– Dennis Dennis ReidsmaReidsma–– Jordi CarrabinaJordi Carrabina
•• Thanks for the received aid of:Thanks for the received aid of:–– HerwinHerwin van van WelbergenWelbergen–– Ronald Ronald PoppePoppe
•• Thanks to our family, friends, girlfriendsThanks to our family, friends, girlfriends (MLG) (MLG) and and partners.partners.
/ 40
EEMCS - HMI 3�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 40
EEMCS - HMI 4�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 5�
PresentationPresentation•• Joint projectJoint project
•• The Virtual DancerThe Virtual Dancer
•• Obtaining a greater interactionObtaining a greater interaction
•• Recognizing dance movement patternsRecognizing dance movement patterns
/ 39/ 40
EEMCS - HMI 6�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 7�
GoalsGoals1.1. Starting up The Virtual Dancer application (computer vision partStarting up The Virtual Dancer application (computer vision part
and artificial intelligent part)and artificial intelligent part)
2.2. Analyzing the application and developing improvements Analyzing the application and developing improvements strategies strategies
3.3. Recognition of simple dance movementsRecognition of simple dance movements
4.4. Recognition of dance movement patternsRecognition of dance movement patterns
5.5. Automatic generation of dance movement search patternsAutomatic generation of dance movement search patterns
6.6. Finding patterns without a referenceFinding patterns without a reference
/ 39/ 40
EEMCS - HMI 8�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 9�
Organization of the ExpositionOrganization of the Exposition•• Virtual Dancer applicationVirtual Dancer application
•• The computer vision system The computer vision system ““ParleVisionParleVision””
•• The work The work environment with the environment with the setup application and the used materialsetup application and the used material
•• The study carried out at the beginning of the projectThe study carried out at the beginning of the project
•• The implemented modifications in the computer vision system The implemented modifications in the computer vision system ““ParleVisionParleVision””
•• Dance Movement Patterns Recognition (Part II)Dance Movement Patterns Recognition (Part II)–– HMMHMM–– Recognizing Simple/Complex MovementsRecognizing Simple/Complex Movements–– Auto Generation of Complex Patterns GraphsAuto Generation of Complex Patterns Graphs–– Test BenchTest Bench–– ConclusionsConclusions
/ 39/ 40
EEMCS - HMI 10�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 11�
The Virtual DancerThe Virtual Dancer
/ 39/ 40
EEMCS - HMI 12�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 13�
ParlevisionParlevision
/ 39/ 40
EEMCS - HMI 14�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 15�
MaterialMaterial•• HardwareHardware
–– 2 computers with the following properties2 computers with the following properties•• Pentium 4 2.8GHzPentium 4 2.8GHz•• 512 MB of RAM512 MB of RAM•• Microsoft Windows XP with the SP 2Microsoft Windows XP with the SP 2
–– 1 camera1 camera•• Philips ToUcam XSPhilips ToUcam XS
•• SoftwareSoftware–– ParleVisionParleVision–– Virtual DancerVirtual Dancer
•• OthersOthers–– Human DancerHuman Dancer–– A black backgroundA black background–– tt--shirts with different colorsshirts with different colors
/ 39/ 40
EEMCS - HMI 16�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 17�
SetupSetup
4
5
LAN
3
6
1 2
Port 7521
1 Computer Vision PC
2 Artificial Intelligent PC
3 Camera
4 User
5 Background
6 LAN
/ 39/ 40
EEMCS - HMI 18�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 19�
Interaction DimensionsInteraction Dimensions
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 20�
Interaction DimensionsInteraction Dimensions•• Temporal dimensionTemporal dimension
–– Before the songBefore the song–– During the songDuring the song–– After the songAfter the song–– Between songsBetween songs
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these
goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 21�
Interaction DimensionsInteraction Dimensions•• Goal of the interactionGoal of the interaction
–– To attract the attentionTo attract the attention–– To entertainTo entertain–– To leave a good memoryTo leave a good memory
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these
goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 22�
Interaction DimensionsInteraction Dimensions•• ActionsActions
–– To attract the attentionTo attract the attention•• To say helloTo say hello•• To invite the userTo invite the user•• To be seenTo be seen
–– To entertainTo entertain•• To start the danceTo start the dance•• To encourage the userTo encourage the user•• To do a choreography / chorusTo do a choreography / chorus•• To enrich the danceTo enrich the dance•• To explain some stepTo explain some step
–– To leave a good memoryTo leave a good memory•• To congratulateTo congratulate•• To say goodbyeTo say goodbye
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these
goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 23�
Interaction DimensionsInteraction Dimensions•• Cause of the actionCause of the action
–– ReactiveReactive–– DeliberativeDeliberative
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these
goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 24�
Interaction DimensionsInteraction Dimensions•• Way to do this actionWay to do this action
–– VerbalVerbal–– Non VerbalNon Verbal
GOAL OF THE INTERACTION
(What we want to do)
ACTIONS(What to do to get these
goals)
CAUSE OF THE ACTION(Why are we doing this action?)
WAY TO DO THIS ACTION(How we want to do it)
TEMPORAL(When occurs the fact)
/ 39/ 40
EEMCS - HMI 25�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 26�
Interaction TreeInteraction TreeInteraction
Before the song During the song After the song
To attract theattention
To say hello
Reactive / Deliberative
To invite to dance
Reactive / Deliberative
To be seen
To entertain
To do a Choreography/ Chorus
To enrich thedance
To attract the attention
To be seen
Reactive / Deliberative
Reactive / Deliberative
Reactive / Deliberative
To entertain
Reactive / Deliberative
To leave a goodmemory
To congratulate
Reactive / Deliberative
To say good bye
Reactive / Deliberative
To start the dance
Reactive / Deliberative
Between songs
To entertain
To encourage theuser
To explain somestep
Reactive / Deliberative
Reactive / Deliberative
To encourage theuser
Reactive / Deliberative
To encourage theuser
Reactive / Deliberative
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
Verbal / Non Verbal
/ 39/ 40
EEMCS - HMI 27�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 28�
Waiting a User (1)
To be seen(2)
To say hello(3)
To invite todance (4)
To start thedance (6)
To encouragethe user (5)
Dance time (7)
To congratulate(12)
To say goodbye (15)
To encouragethe user (13)
To explainsome step (14)
CV detection
Pad & CV detection
Yes
No
User says Yes User says No
User says No
User says Yes
To start the song Chorus detected
Random
Absent-mindeduser
To finish the song
User go out
Random
Interaction State MachineInteraction State Machine
To enrich thedance (9)
To encouragethe user (10)
To be seen(11)
To do a choreography
(8)
To detect passive movements
/ 39/ 40
EEMCS - HMI 29�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 30�
Variables, Animations & TransitionsVariables, Animations & Transitions•• What animations we could use in each stateWhat animations we could use in each state
•• New animationsNew animations
•• Conditions required for changing the state Conditions required for changing the state
•• Available VariablesAvailable Variables
•• New variablesNew variables
/ 39/ 40
EEMCS - HMI 31�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 32�
Possible Improvements•• AnimationsAnimations
–– To say hello moving the hand, looking at the screen centerTo say hello moving the hand, looking at the screen center–– To bow to the userTo bow to the user–– To clap to the userTo clap to the user–– To say hello putting the head upTo say hello putting the head up
•• OutputsOutputs–– That our dancer could speakThat our dancer could speak
•• InputsInputs–– To recognize usersTo recognize users’’ speechspeech–– To recognize headTo recognize head’’s movementss movements
•• Pattern RecognitionPattern Recognition–– Simple movementsSimple movements’’ recognitionrecognition–– Patterns recognitionPatterns recognition–– Creation of patterns automaticallyCreation of patterns automatically–– Search of PatternsSearch of Patterns
/ 39/ 40
EEMCS - HMI 33�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 34�
New Added FeaturesNew Added Features
/ 39/ 40
EEMCS - HMI 35�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 36�
New Created Pipelines• Save Video Pipeline
• From Video Pipeline
/ 39/ 40
EEMCS - HMI 37�
New Created Pipelines• Pipeline with Radial Histogram
/ 39/ 40
EEMCS - HMI 38�
ContentsContents•• IntroductionIntroduction
–– PresentationPresentation–– GoalsGoals–– Organization of the ExpositionOrganization of the Exposition
•• The Virtual Dancer The Virtual Dancer •• ParlevisionParlevision•• Work EnvironmentWork Environment
–– MaterialMaterial–– SetupSetup
•• Previous StudyPrevious Study–– Interaction DimensionsInteraction Dimensions–– Interaction TreeInteraction Tree–– Interaction State MachineInteraction State Machine–– Variables, Animations & TransitionsVariables, Animations & Transitions–– Possible ImprovementsPossible Improvements
•• The Built System (CV)The Built System (CV)–– New Added FeaturesNew Added Features–– New Created PipelinesNew Created Pipelines–– New Sent VariablesNew Sent Variables
/ 39/ 40
EEMCS - HMI 39�
New Sent VariablesNew Sent Variables•• cumulativeHorActivityNormalizedcumulativeHorActivityNormalized•• cumulativeVertActivityNormalizedcumulativeVertActivityNormalized•• lHandlHand•• rHandrHand•• cumulativeRActivityNormalizedcumulativeRActivityNormalized•• rr•• xx•• yy
/ 39/ 40
EEMCS - HMI 40�
Dance Movement Patterns Recognition Dance Movement Patterns Recognition (Part II) (Part II) ……
Bedankt voor uw aandacht !Bedankt voor uw aandacht !
/ 39/ 40
