Inaugural Meeting

7th April 2008Adaptive Systems Research Group

Prof. Chrystopher L. NehanivDr. Joe Saunders

Prof. Kerstin Dautenhahn

UH - Where are we?

Human-Robot Interaction Research @ UH

• Current European FP6 Robotics Projects:– Cogniron (IP): Cognitive

Robot Companion– Robotcub (IP):

Developmental Robotics (iCub)

– IROMEC (STREP): Robotic Toys for Children

• New FP7 projects: – LIREC (IP)– ITalk (IP)

• Networks: Euron, euCognition

• Researchers:– Academics:

• Prof. Kerstin Dautenhahn• Prof. Chrystopher L. Nehaniv• Dr. Rene te Boekhorst• Dr. David Lee

– Postdocs:• Dr. Ben Robins• Dr. Ester Ferrari• Dr. Joe Saunders• Dr. Hatice Kose-Bagci• Dr. Aris Alissandrakis• Dr. Khenglee Koay• Dr. Nuno Otero

– Research Assistants/PhD students• Michael L. Walters• Assif Mirza• Qiming Shen• Dag Syrdal• Josh Wainer

UH - Related EU Projects

RobotCub – Social Interaction, Developmental Robotics (FP6 IP, September 2004 – August 2009)

Cogniron – Robot Social Spaces, Proxemics, Learning and Development, Imitation

(FP6 IP, Jan 2004 – Feb 2008)

LIREC - LIving with Robots and intEractive Companions (FP 7 IP, March 2008 – June 2012)

UH - The Team

Principal Scientists

Prof. Chrystopher L. Nehaniv Prof. Kerstin Dautenhahn

Researchers

Dr. Joe Saunders

+

* Own Effort

Research Assistant/

FellowNew Hire

Dr. Caroline Lyon*

Hello

Frank FörsterPhD Student*

UH - Approach to ITALK

A robot will use what it learns individually and from others sociallythrough grounded sensorimotor interaction

to bootstrap the acquisition of language via aspects of ideomotor theory,

and will use its language abilities in turn to drive its learning of social and manipulative abilities.

This creates a positive feedback cycle between using language and developing other cognitive abilities.

Like a child learning by imitation of its parents and interacting with the environment around it, the robot will master basic principles of structured

grammar, like negation, by using these abilities in context.

“In the beginning, there was the act.”- Goethe

UH - Approach to ITALK

Major Open Areas for Robot/Agent Development of Language:

1) deixis, gesture and reference;2) predication;3) negation;4) emergence of syntactic categories;5) compositionality

Current Work and Open Problems: A Roadmap for Research into the Emergence of Language of Communication. In C. Lyon, C.L. Nehaniv, A. Cangelosi (editors), Emergence of Communication and Language, pages 1-27, Springer-Verlag, 2007

UH - Viewpoint

Extending C.S. Peirce, latter L. Wittgenstein, & R. Garrett Millikan

To understand (and create!) language use capacity bottom up:(proto-)representations need NOT separate “what-to-do” from “what is-the-case”

(pushmi-pullyu reps.)activity of manipulating and communicating socially in a context of embodied action is

necessarymajor purpose of language is to manipulate the environment, including the social

environment, via communication(other purposes: to bootstrap learning, to regulate interaction, encoding

temporally/spatially remote experience stories – cf: Ho et al. 2008,Mirza et al., 2008)to understand the grammar of a word/linguistic construction look how it is usedmeaning is created in an embodied process of semiosis with interpretant, signal (sign),

signified.

Outline and UH Role

Five Major Work Packages

Action DevelopmentConceptualisationSocial Interaction – Work Package LeaderLanguage EmergenceIntegration and Bootstrapping of Cognition

Robotic Experiments and Demonstrations

Based on iCub robotSimulation (where necessary)Prior to iCub on Kaspar

Outline and UH Role

• Development of artificial embodied agents that are able to acquire complex behavioural, cognitive and linguistic skills through individual and social learning.

Specifically•To learn to handle and manipulate objects and tools autonomously

•To cooperate and communicate with other robots and humans

•To adapt their abilities to changing internal, environmental and social conditions

Aim

UH will build upon our social learning architecture - ROSSUM (RObot Self-Imitation and Scaffolding Utility Mechanism)

Saunders J, Nehaniv C.L, Dautenhahn K. and Alissandrakis A.,(2007), Self-imitation and Environmental Scaffolding for Robot Teaching, International Journal of Advanced Robotics Systems, Vol.4, Issue 1, pp.109- 124

Scaffolded Skills <-> Scaffolded Language

Outline and UH Role

AbilitiesAgents learn to handle objects

UH:

Investigate, by extending existing studies on Ideomotor theory applied to physical robots, how to develop more complex compositional and hierarchical actions via scaffolded learning.

Outline and UH Role

AbilitiesAgents learn to handle objectsAgents create and use embodied concepts

UH:

Our current studies on robot teaching and interaction ensure that the robots experiences are grounded in its own internal and external perceptual states. The crucial aspects of these states are selected using information theoretic approaches.

AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skills

UH:

Using our existing social learning and teaching robotic architecture (called ROSSUM) which is now operating on our humanoid robot (called Kaspar) we intend to study relationships between deixis, gesture and reference. Exploit concept of shared meaning arising through similar gestures and also study ideas of ‘negation’.

Outline and UH Role

AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilities

UH:Investigate relationships between actions and objects via hierachical scaffolding

Outline and UH Role

AbilitiesAgents learn to handle objectsAgents create and use embodied conceptsAgents develop social skillsAgents develop linguistic abilitiesAgents transfer knowledge between cognitive domains

UH:

Study the HRI aspects of verbal or gestural communication.

Outline and UH Role

UH Existing Work on ROSSUM and its Relation to Language

Teaching via Assisted/self

ImitationImitation

CorrespondenceProblem

Affordances andEfffectivities

EmbodiedGrounded

Sensory-motorConcepts

Steps toLanguage Acquisition

How can a child discover affordances and effectivities for action?:

• by trial-and-error – may be slow• by observational imitation – but correspondence problem exists• by assisted/self-imitationCaregivers invite infants to imitate – by putting infants on both sides of the correspondence problem

Scaffolded Teaching via Assisted Imitation

Assisted/Self-Imitation

• Putting through of infants movements• Acting in tandem• Linking effectivities of the infants body with affordances for action• Scaffolding taught competencies

(* Affordances, Effectivities and Assisted Imitation - Patricia Zukow-Goldring, Michael Arbib (2007).)

UH Existing Work on ROSSUM and its Relation to Language

Correspondence Problem and Difficulties in Observational Learning

Matching Movements?

Matching Goals/Effects/States?

Matching Combinations of Movements?

What does matching mean?

The Correspondence Problem*

*C.L.Nehaniv & K.Dautenhahn (2002) – The Correspondence Problem

UH Existing Work on ROSSUM and its Relation to Language

The Correspondence Problem in Imitation

Relative Position

Demonstrated Effect Corresponding Effects

Relative Displacement

Absolute Position

(also see Aris Alissandrakis, Chrystopher L. Nehaniv, Kerstin Dautenhahn and Joe Saunders (2005))

UH Existing Work on ROSSUM and its Relation to Language

UH Existing Work - Robots Learning Socially

using ROSSUM

Teaching via Assisted/self

Imitation

Inverse andForward models

ExtendedIdeomotor

Theory

Imitation

CorrespondenceProblem

Affordances andEfffectivities

ReafferencePrinciple

EmbodiedGrounded

Sensory-motorConcepts

Steps toLanguage Acquisition

UH Existing Work on ROSSUM and its Relation to Language

From Analyse der Empfindungen – Ernst Mach - 1922

Body is visually perceived in exactly the same way as the environment – representation

is the same

Distinction between body and environment must be

based on a non-visual criteria

Distinction achieved by voluntary control

Actions are considered to be creations of the will

UH Existing Work – Ideomotor Theory

Ideomotor Theory : Lotze (1852), James (1890) and Műnsterberg (1888)

How can voluntary behavior or ‘will’ be explained?

All actions are held as ‘images’ of the sensory feedback they produce.

Sensory stimulations result from internal feedback they produceplus

external feedback from the effects on the environment

Actions are generated based on imagining their sensory consequences

UH Existing Work – Ideomotor Theory

Why is this important for a robotic learning system?

One powerful way of learning is through imitation

Extend the Ideomotor Idea to allow imitation – Extended Ideomotor Theory*

If an action can be invoked from thinking about it…..

then

the same action could be invoked by seeing it performed by someone else

*An Ideomotor Approach to Imitation - Wolfgang Prinz (2005)

Extended Ideomotor theory thus supports the idea of learning through experience and

learning through observation

UH Existing Work – Ideomotor Theory

UH New Work – extending ROSSUM

Teaching via Assisted/self

Imitation

Inverse andForward models

ExtendedIdeomotor

Theory

Imitation

CorrespondenceProblem

Affordances andEfffectivities

ReafferencePrinciple

EmbodiedGrounded

Sensory-motorConcepts

Learning viaObservational

Imitation

“Putting oneself In the shoes of

Another”TOM?

Learning viaAssociation,

Inference andReferential intent

Adultto

Child?(Teaching)

Childto

Adult?(Learning) Steps to

Language Acquisition

Perceived State Action State Resulting From Action Execution

Action State Resulting From Action Execution

Action State Resulting From Action Execution

Action State Resulting From Action Execution

.

.

.

ROSSUM Behavioural Component / Memory Model

‘Ideas of Movement’From Previous Learning Effects of Movement

Perceived State

Perceived State

Perceived State

Acts as an Inverse Model Acts as a Forward Model

Similarity

Matching

Current State

Similarity

Matching

Model State

Goal-directed Task

Sequence

Primitive

Behaviour

Primitive Primitive Primitive

Goal-directed Task

Primitive Primitive

Example of a Trained Hierarchy

State

Primitive

Behaviour

Using task scaffolding to build competencies

UH The First Year – Experimental and Theoretical

Research Questions What is the relationship between deixis, gesture and reference ----------- D3.1 (M24)

- Related to hypothesis on the emergence of language based on mirror neurons- Shared meaning arises through similar affordant gestures- the human tutor reveals to the robot via deixis, gesture and reference relevant effectivities and affordances

How can negation arise though the mechanisms of communicative social interaction?---D3.2 (M48)

-Experiments to be carried out where facial and manual gestures used in social interaction to indicate negation- This would involve inhibition signals operating on existing scaffolded behaviours

UH The First Year – Technical Considerations for ROSSUM

Implement Forward ModelForward model side of ROSSUM to be implemented (relates to D1.1)

Gaze Following and Observational ImitationImplement direct imitation via human body tracking either as combination of TOF depth sensor/magnet field trackers/ eye gaze tracker (as Yu/Ballard, S. Knoop)

Object DetectionExtend capabilities for enhanced object recognition by robot using colour segmentation, shape, texture (as Yu/Ballard, Deb Roy).

Word/Phoneme Detection?Study possibility of word/phoneme recognition (as per Deb Roy)

Assisted or Self ImitationExtend current model to further process dynamic movements

Thank you

Adaptive Systems Research Group