EU – FP7

AMARSi Adaptive Modular Architectures for Rich Motor Skills

ICT-248311

D 1.1

October 2010 (6 months)

Comparative evaluation of notions of modularity in human motor control and of existing algorithms for the

identification of motor primitives

Martin Giese1, Andrea d’Avella2, Tamar Flash4 , Yuri P. Ivanenko2, Thomas Schack3, Enrico Chiovetto1, Albert Mukovskiy1.

1- Section for Computational Sensomotorics, Department of Cognitive Neurology, Centre for Integrative Neuroscience, University Clinic Tübingen, Tübingen, Germany.

2- Department of Neuromotor Physiology, Santa Lucia Foundation, Rome, Italy. 3- Neurocognition and Action Research Group, Faculty of Psychology and Sport

Science, University of Bielefeld, Bielefeld, Germany. 4- Department of Computer Science and Applied Mathematics, Weizmann

Institute of Science, Rehovot, Isreal. Due date of deliverable 1st September 2010 Actual submission date 15th October 2010 Lead Partner Tübingen University Revision Final Dissemination level Public

Comparative evaluation of notion of modularity in human motor control and of existing algorithms for the identification of motor primitives

There exists a broad range of biological approaches to movement primitives that covers different

functional levels and different methodological approaches. Many of the existing approaches are

represented in the AMARSI consortium. This deliverable will, on the one hand, provide a brief

overview of the approaches towards movement primitives that have been established by the different

partners. Possible relationships that might form a basis to establish collaborations between the groups

and specifically with the technical groups in robotics will be briefly discussed. On the other hand, in the

second part of this deliverable, some quantitative work will be presented that compares between

methodologically closely related approaches of different groups. Such comparisons are possible only

between approaches that conceptualize movement primitives in similar ways, and at the same

representation level.

1 Overview of different biological approaches towards motor primitives The concept of movement primitives is quite popular in research in neuroscience. At the same time,

the use of this concept is very heterogeneous, and many different conceptualizations of motor

primitives appear in the biological literature. Some of these conceptions address phenomena at similar

levels, but using different methodologies. Others signify phenomena at very different representation

levels, such as muscle synergies or cognitive action plans. The AMARSI consortium comprises groups

that have worked on different approaches for the characterization of movement primitives, and which

cover a large part of the existing approaches. Some of these approaches are closely related, while

others address completely different representation levels. Table 1 gives an overview of the

approaches and conceptualizations of motor primitives that are represented within the AMARSI

consortium.

Partner People involved Level Theoretical / experimental

approaches Section

UniBi

T. Schack, B.Land, A. Krause, B. Blaesing

Cognitive representations

Volitional initiation control strategies

1.1

WI T. Flash, A. Barliya, R. Fuchs, Y. Meirovich

Kinematics, strokes / trajectory segments

Differential geometry, invariants, kinematic analysis

1.2

SFL A. d’Avella, Y. Ivanenko, G. Cappellini

Neuromuscular synergies / kinematics

Unsupervised learning, EMG, motion capture

1.3

UniTu

M. Giese, E. Chiovetto, A. Mukovskiy, N. Taubert

Kinematics, movement sequences

Unsupervised and supervised learning, dynamical systems, motion capture

1.4

Table 1. Biological approaches for the investigation of movement primitives within the AMARSI

consortium.

In the following, we will first sketch the definition and approaches for the identification and modelling of

motor primitives provided by the different partners that are involved in the analysis of biological data.

Finally, we will provide a comparison that highlights similarities and complementarities that might help

to establish fruitful interactions between the different partners.

1.1 Neurocognitive perspective on motor primitives

A very general definition of motor primitives is underlying the work by the partner UniBi. This

conception includes a variety of levels form low to high-level representations, and it addresses

specifically the cognitive aspects of the representation and control of complex movements. A particular

focus of this approach is the planning and memory of complex movements and changes associated

with skill learning and during development.

1.1.1 Notion of Motor Primitives (MP)

Motor Primitives are conceptualized as basic building blocks in a modular motor control architecture

(cognitive architecture of human motion). These building blocks are functionally relevant elementary

components or transitional states of (complex) movements. In our point of view it is possible to

understand a movement as a serial and functional order of significant and goal related body postures.

Such body postures and motion sequences are bi-directional linked with perceptual (movement-)

effects and typically stored in memory in a hierarchical order.

A main topic of our research is the cognitive architecture of human action, showing how it is organized

over several levels and how it is built up. Alongside Bernstein's (1947) approach to the construction of

action, there have been several formulations of the idea that movement control is constructed

hierarchically. In contrast to most of these approaches, the model proposed here views the functional

construction of actions on the basis of a reciprocal assignment of performance-oriented regulation

levels and representational levels (see Table 2). These levels differ with respect to their central tasks

on different regulation and representation levels. Each level is assumed to be functionally

autonomous.

Code Level Main function Subfunction Means

IV Mental Control

Regulation Volitional initiation control strategies

Goals; goal posture representation; strategies

III Mental representation

Representation Effect-oriented adjustment

Basic action concepts

II Sensorimotor representation

Representation Spatial-temporal adjustment

Perceptual effect representations

I Sensorimotor control

Regulation Automatization Functional systems; Transitional states

Table 2. Levels of Complex Motion in Humans (Schack, 2004).

We identify Basic Action Concepts (BACs) as major building blocks at the representation level. BACs

are based on the cognitive chunking of body postures and movement events concerning common

functions in realisation of action goals. Their characteristic set of features results from the perceptive

and functional properties of action effects: They tie together their functional and sensory features.

These functional features are derived from action goals (goal postures), and this connects BACs to

Level IV. However, BACs also integrate sensory features of sub-movements, for example, through

chunking. As a result, they also refer to the perceptual effects of movements. This connects BACs with

Level II. All together BACs can be viewed as the mental counterparts of functionally relevant

elementary components or transitional states of complex movements. They are characterized by

recognizable perceptual features. They can be described verbally as well as pictorially, and can often

be labelled with a linguistic marker. "Turning the head" or "bending the knees" might be examples of

such basic action concepts in the case of, say, a complex floor exercise. Based on our research we

learned that such movement representations might provide the basis for action control in skilled

voluntary movements in the form of cognitive reference structures.

With respect to robot control, current work is largely focused on a low level of abstraction that is very

close to the sensors and actuators. In contrast, human actions are heavily informed by huge amounts

of representations about the goal and the characteristics of the anticipated movements, the

encountered objects, and about how to counteract the numerous disturbances and mistakes that

usually occur during even moderately complex movements. Therefore one major goal of our research

is to design experiments and simulations concerning the formation and cooperation of motion building

blocks at a cognitive level (memory) and to learn about the integration of biomechanical and motor

constraints into cognitive motor representations and cognitive motor hierarchies.

1.1.2 Methods and experimental evidence

1.1.2.1 Experiments to study the cognitive representation of MP’s (BACs) in memory

We used different experimental methods to study:

cognitive structures in long term memory in different types of movements

movement based chunking in short term memory

representations of power- and precision grasps in memory

representations of manual action after stroke

the overlap between kinematical and cognitive movement structures

Summarizing the expertise studies, we found that in high-level experts the representational

frameworks were organized in a distinctive hierarchical tree-like structure, with remarkable similarities

between individuals. These frameworks were well matched with the functional and biomechanical

demands of the task. In comparison, action representations in low-level athletes and non-athletes

were organized less hierarchically, were more variable between persons, and were less well matched

with functional and biomechanical demands.

Figure 1. Representation structures for two chosen tennis expertise groups, respectively experts and

non-players (A and B), based on the hierarchical cluster analysis of basic action concepts (BACs) in the

tennis serve. The horizontally aligned numbers denote the BACs (for the code, see text); the vertical numbers

specify the Euclidean distances. The lower the numbers, the lower the distances between BACs in long term

memory. For all groups holds n = 11; p = 0.05; dcrit = 3.46 (from Schack & Mechsner, 2006).

Results from two different lines of research addressing the mental representation level showed that

not only the structure formation of representations in long-term memory but also chunk formation in

working memory are built up on BACs and relate systematically to movement structures. Experiments

were designed to assess both the structure of mental representations in LTM (determined with the

SDA-M) and chunking in working memory (determined with Cognition and Movement Chronometry,

CMC). Results confirmed the interaction between long-term memory and short-term memory,

demonstrating that cognitive systems interact to produce complex movements. Our experiments have

shown that both, the order formation in LTM and the chunking in working memory build on the

topological (spatiotemporal) structure of the movement. This provides experimental evidence that

structures in movement and memory mutually overlap.

1.1.2.2 Experiments to study the relationship of cognitive motor representation and

biomechanical constraints

To accomplish a better understanding of the cognitive architecture of complex movements, it is not

only interesting to know whether LTM and working memory cooperate horizontally on, for example,

one level of mental representations. It is, for instance, also crucial to know whether there is a vertical

cooperation between the level of mental representations and the level of sensorimotor control. One

could ask whether biomechanically relevant features can be found in the structure of mental

representations. Experimental studies (Schack, 2004; Schütz et al., 2009) showed that

representational frameworks were organized in a hierarchical tree-like structure and revealed a good

match with the biomechanical demands of the task. After measuring kinematic parameters, we

investigated the relationship between the structure of motor representation and the kinematic

parameters of different movements. Our studies have revealed significant correlations between

kinematic parameters (time structure, angles according to the take-off-phase, tilt angle, angular

velocities, etc.) of movement and the corresponding parts of mental representations.

Other studies are done to find out the relationship between anticipated goal states (end-state-comfort)

and the representation of functional grasp constellations in children (Weigelt & Schack, 2010; Stöckel,

& Schack, 2010). All together our experimental results support the hypothesis that voluntary

movements are executed and stored in memory directly through representations of their anticipated

perceptual effects. We are combining experiments concerning the cognitive representation structure in

motor memory with PCA or a new kind of spatiotemporal analysis (Bläsing & Volchenkov, 2010) to

identify MP’s and to learn about the relationship of cognitive and biomechanical motor constraints

(MP’s).

1.1.2.2 Simulations and neural network architectures

In order to model the learning and generation of complex movements, a hierarchical, modular

neuronal network architecture is currently under development. The architecture will be able to learn

demonstrated movements (kinaesthetic teaching) and - after training - trigger adaptive movement

execution based on anticipated sensory data from the robot and the environment.

The idea is to have - on a lower level - a battery of recurrent networks controlling the execution of

automatically learned movement primitives in a tight sensory motor loop. The dynamics of those

networks is controlled through bifurcation by inputs (similiar to the work by Jun Tani regarding RNNPB

with parametric bias units) by a higher level network that selects- and interpolates between patterns

and generates long motor sequences. On the top level of the architecture, one or multiple hierarchical

self-organising maps (HSOM) represent the cognitive structures of the complex movements (BAC's,

see Schack & Mechsner, 2006). The HSOMs can analyse and cluster sensory data, detect proper

situations when to perform suitable movements and adjust task affordances (Krause et al., 2010).

In addition, other computational approaches for the simulation of the structure of mental

representations are being developed (Tscherepanow et al., 2010). Such biologically inspired models

of movement representations are to be contrasted with other, less biologically plausible models, such

as discussed in WP4 for motor primitives.

1.1.2 Implications for robotics

Because the production of manual actions is affected by a number of factors, such as biomechanical

constraints (Weigelt et al., 2009), a line of our studies focuses on the question of how structures of

sensory-motor representations are established and changed in a stepwise manner under the

consideration of task constraints (Bläsing et al. 2010b; Maycock et al., 2010). In these studies it is of

interest to learn about the relationship between the structure of mental representations and the

performance in manual actions, especially in situations in which actions result in errors. To this end,

we combined methods from cognitive psychology, biomechanics, and computer sciences (i.e.

augmented reality / virtual reality scenarios) to assess the structure of sensory-motor representations,

to introduce experimental manipulations for manual actions, and to examine the resulting performance

changes (including changes in erroneous performance). The insights gained in these first experiments

will be implemented on robotic platforms (i.e. a 7 DOF robot arm set-up) within a longer research

perspective.

Our research concerning the cognitive architecture of human action (Bläsing et al., 2010a; Schack,

2004; Schack, 2010) is of interest for the construction of robot architectures and vice versa. In different

research project, we translated our findings in studies of cognitive factors in human motor control into

models that can guide the implementation of cognitive robot architectures. Focusing on the issue of

manual action control, we illustrated some results in the context of grasping with a five-fingered

anthropomorphic robot hand (Schack & Ritter, 2009).

A research project about cognitive planning of action sequences and sensorimotor adaptation

(cooperation with the CoR-Lab and HRI Europe) addresses the cognitive and perceptual dimensions

of goal posture planning in humans and is related to grasp optimization and trajectory planning in

robots. The aim here is to integrate the research results in humans (e.g. Weigelt & Schack, 2010) into

a comprehensive framework, and posture based movement representation, that allows an efficient

realisation of a fluent robot grasping.

Our studies about cognitive representation of complex motor actions (e.g. Schack & Hackfort, 2007;

Schack & Mechsner, 2006), are of relevance for a control of full body movements in robots. We

created a research lines from an experimental analysis to a computational modelling of full body

movements (Krause et al., 2010). In this research line, we want to investigate complex movements of

the whole human body, as they occur in sports, dance or other expert tasks (Bläsing et al., 2010a), but

also in every-day life, in order to understand how these movements are controlled, learned and

reproduced under changing conditions. Our aim is to apply different biomechanical and psychological

methods to analyse the kinematics, force profiles, muscle activity, mental representations and other

cognitive control structures, and to develop a computational model that integrates the results of the

different measures into a simulation of the movement. Results of these studies could be helpful on

many levels for the general understanding of human movement control, but also for movement control

in artificial agents and robots, especially if we are interested in “human-friendly” interaction between

humans and technical systems.

Cognitive representations are reference frames in the implementation and control of human motor

action (Schack & Ritter, 2009; Zentgraf et al., 2009). For a better understanding of the structure and

functionality of such reference frames we planned in cooperation with Rolf Pfeiffer and Rüdi Füchslin

(ai-Lab, Zürich) different experiments about principles of tool use and tool manipulation in humans. In

a next step we are going in direction of a computational modelling and a technical implementation of

functionally structured motor representations in robotic platforms. The aim is here to address in a

longer research perspective the “frame of reference problem” in robotics from a new point of view.

1.2 Motor primitives at the level of kinematic invariants

A number of groups in the AMARSI consortium address the problem of movement primitives at the

level of kinematics. A very generic approach is followed by the partner WI who characterizes

movement primitives by their geometrical invariants. This approach works at a very basic level and

does not require higher cognitive representation levels. This conception is particularly suitable for the

segmentation and modelling of action streams and of complex movements that are modelled by

sequencing of simpler segments.

1.2.1 Notion of Motor Primitives (MP)

In dealing with the modularity of the motor control system the focus of the Weizmann group is on the

level of trajectory planning, but studies are also carried out which are dealing with motion planning at

the joint level. The term motor (or motion) primitive refers to elementary movements or building blocks

from which more complicated movements are constructed. In studying the issue of motor

compositionality one of the main goals is to be able to infer motion primitives from movements that are

apparently continuous. Hence, the main questions in investigating the nature of motion primitives and

compositionality are as follows. a) Can we identify an alphabet of motor primitives from which more

complex behaviors are constructed? b) What is the nature and internal representation of such

primitives? c) What generation rules are used to generate or span an entire motor repertoire of either a

single or several motor tasks from a limited set of elementary movements? d) What syntactic rules are

used in joining together motor elements?

In particular in attempting to identify and characterize motor primitives our approach is based on taking

advantage of the existence of motor invariants and templates at the level of trajectory planning. These

invariants consist, for example, of the invariance of hand paths and velocity profiles during reaching

movements or of the two-thirds power law observed for curved and drawing movements. These

kinematic laws of motion can also be accounted for by several optimization models, specifically the

minimum jerk model (Flash and Hogan, 1985), but also by more general maximum smoothness

models (MSD models). Earlier it was suggested that the observation of a piecewise segmented

power law and the change in the value of the velocity gain factor can be considered as evidence for

motor segmentation. This claim, however, is problematic (Flash and Hochner, 2005) and is not

consistent with the findings reported in Richardson and Flash (2002) who have shown that global

optimization gives rise to a similar piecewise constant relationship between the logarithms of speed

and curvature without any assumption about segmented control. More recent studies, however, have

suggested a somewhat alternative explanation for the power law and hence an alternative approach to

segmentation, i.e., the approach based on equi-affine differential geometry and specifically on the

observation that the two-thirds power law is equivalent to moving at a constant equi-affine speed

(Flash and Handzel, 2007). In particular, when this new framework was applied to the analysis of hand

trajectories it has led to a new focus on new geometric metrics and invariants which have included the

equi-affine arc-length and equi-affine curvature which remain invariant under equi-affine

transformations.

This approach was then extended to 3D drawing and curved movements by showing that the

hypothesis that equi-affine speed is kept piecewise constant also during 3D hand trajectories leads to

the formulation of a generalized power law whereby hand speed depends both on movement

curvature and torsion (Pollick et al., 2009; Maoz et al., 2009).

The equi-affine analysis has also served as a basis for developing a more general group theoretical

approach to the study of motor invariants and segmentation and to the suggestion of several new

ideas (Flash and Handzel, 2007; Polyakov et al., 2009a, 2009b): 1) Movement states within the motor

space might be characterized by their equi-affine differential invariants 2) A large variety of

movements might be achieved by applying equi-affine transformations on such limited number of

states and by combining them.

This theoretical approach has also led to a series of combined behavioral, neurophysiological and

computational studies that gave some validity to this idea (see below, Polyakov et al, 2009a, 2009b).

An important regularity observed during human movement which was not accounted for by the equi-

affine framework is the isochrony principle, i.e., movements of different lengths (amplitudes) having

nearly the same movement duration (Viviani and Flash, 1995). Also no principled rule for the selection

of the velocity gain factor was suggested. Hence more recently we have generalized the equi-affine

description to a new and broader theory based on geometrical invariance. The new notion is that

movement duration and compositionality arise from cooperation among Euclidian, equi-affine and full

affine geometries where each geometry possess a canonical measure of distance along curves, an

invariant arc-length parameter (Bennequin et al., 2009).

The theory was mathematically formulated and its predictions were tested on three data sets:

drawings of elliptical curves, locomotion and drawing trajectories of complex figural forms. This new

theory suggests new notions about motion compositionality and segmentation.

Figure 2. Equi-affine parameters for a parabolic-like recorded movement segment. a. Equi-affine velocity

(dots) and curvature (asterisks) for a scribbling segment. b. The actual drawing made by the monkey. Since

parabolas are characterized by zero equi-affine curvature, the motion segment can be well-fitted by parabolas,

(dashed lines). Figure taken from Polyakov et al., 2009b.

1.2.2 Methods and experimental evidence

The following pieces of experimental evidence support the discussed theory:

a) Earlier evidence for the existence of motion primitives at the kinematic levels comes from work

concerning the principle of superposition of elementary point-to-point to point movements from

a study of trajectory modification (Flash and Henis, 1991; Henis and Flash, 1995).

b) A co-articulation phenomenon was observed in a study involving continuous practice during

the generation of hand-writing like trajectories. The kinematic properties of well-practiced

movements were successfully accounted for by the maximum smoothness principle and newly

formed curved primitives were shown to emerge following sufficient practice (Sosnik et al.,

2004).

c) In relation to the above geometric approach to movement segmentation and compositionality

based on the equi-affine framework (Handzel and Flash, 1999; Flash and Handzel, 2007;

Pollick and Sapiro, 1997) we have derived a necessary mathematical condition on paths for

which the predictions of both the 2/3 power-law and the constrained minimum-jerk model

coincide (Polyakov et al., 2009a). Such paths must obey the following relation 06 rr

(differentiation w.r.t. the equi-affine arc-length). Using this condition and requiring the

invariance of these paths with respect to equi-affine transformations, we have shown that only

for parabolic shapes, the 2/3 power law and the criterion of smoothness maximization are

mathematically reconciled on the geometric level yielding trajectory elements that are invariant

to equi-affine transformations. We have also demonstrated how complex piece-wise parabolic

trajectories can be generated from a single parabolic template – based on equi-affine

geometric transformations and uniform scaling. We have then examined the validity of our

theoretical analysis by fitting free monkey scribbling movements with basic parabolic strokes

(see Figure 2) and found that following practice, these drawing movements could be

decomposed into only 3-4 well separated clusters of parabolic segments (see Figure 3).

Defining a movement primitive as an elementary stroke that cannot be intentionally stopped after its

initiation, we also found that when the monkey's motor performance was disrupted by giving a reward

at certain locations, the monkey indeed tended to decelerate and stop their movements but not before

the completion of parabolic-like path segments. In additional studies, the neural activities of multiple

single-units, underlying scribbling movements, were recorded in parallel from M1 and PMd during 8

recording sessions and have been segmented in an unsupervised way based on a Hidden Markov

Modeling analysis (as in Abeles et al., 1995). In many cases, the movements corresponding to the

identified states of neural activities formed clusters of similar geometric shapes; some clusters

consisting of parabolic-like segments. By applying partial cross-correlation analysis (Stark et al.,

2007), we have found a stronger representation in the activities of several cells of equi-affine speed

rather than of Euclidian speed.

1.2.3 Implications for robotics

Obviously, all the above computational problems and approaches may apply also to robotic systems

where algorithms and approaches for the selection of motion primitives at the end-effector and joint

levels and their blending arise also in relation to robot arm movements, locomotion and multi-effector

movements especially for robot humanoids, but also to more conventional manipulators or mobile

robots. An example is work in WP4 that models human motion using both, joint-level and end-effector

levels of control resulting in more effective reaching (see Hersch et al., 2008; Calinon et al., 2010).

Figure 3. Emerging parabolic clusters and dimensionality reduction. A. Typical histograms for the fitted

parabolic segments. In the one-dimensional histogram (left), the segments are counted according to their

orientation. In the color histogram (right), they are counted in distinct bins according to the orientation and focal

parameter of the parabola. B. Location of the vertex and orientation of the parabola for every 10th parabolic

segment for the recording sessions. Locations of the vertices of the similarly oriented parabolas are also

clustered. The clusters are marked by ellipses and the mean orientations of the parabolas within each cluster are

depicted by arrows (Taken from Polyakov et al., 2009b).

Another important application for robotics is the segmentation of natural movements into simpler

segments, which then can be supplied to learning algorithms and as basic segments for the design of

controllers. The proposed methods have the advantage that they do not require any prior knowledge

or training data, such as supervised segmentation algorithms, as the ones discussed in section 1.3.4.

020

-200

-100

0

100

Orienta

tion,

[deg.]

Start of practice

0 50

# of strokes

0

2

4

01000

16th

practice session

0 50

# of strokes

0

100

200

300

01020

-100

0

100

Orienta

tion,

[deg.]

# of strokes

0 50

p, [mm]

0

2

4

0100200

# of strokes

0 50

p, [mm]

0

5

10

15

0500

# of strokes

0 50

# of strokes

p, [mm]

0

50

100

-100

-50

0

50

Vertices and orientations

Positio

n -

y,

[mm

]

Vertices and orientations

-100 0 100

-100

-50

0

50

100

Position - x, [mm]

Positio

n -

y,

[mm

]

-100 0 100

Position - x, [mm]

-100 0 100

Position - x, [mm]

Vertices and orientations

1

2

3

1

2

1

23

1

2

3

2

1

3

2

1

Overtrained behaviorA

B

Mo

nke

y O

Mo

nke

y U

Mo

nke

y O

Mo

nke

y U

1.3 Motor primitives at the neuromuscular level

One of the lowest levels for the definition of movement primitives is the level of muscle synergies.

Such synergies can be identified by application of dimension reduction methods to muscle activity

patterns or to kinematic data. The partner SLF has been leading in the research within this field within

the last decade. For selected classes of movements this work has accurately characterized then

organization of the associated muscle activation patterns and their changes associated with different

tasks. In some sense, this characterization of movement primitives in biological systems is closest to

their implementation at the level of biomechanics and the control at the muscle level.

1.3.1 Notion of Motor Primitives (MP)

Two notions of motor primitives have been used by the SLF group to capture the modular organization

of the muscle patterns observed during human locomotion and reaching movements. The first notion

is that of a basic temporal component in the muscle activity patterns (Ivanenko et al., 2004, 2006). The

second notion is that of a time-varying muscle synergy (d'Avella et al., 2003; d'Avella et al., 2006;

d'Avella et al., 2008).

Both notions are based on the assumption that the central nervous system (CNS) generates the

muscle patterns appropriate for performing a task by superposition of a few basic motor programs.

However, each notion emphasizes different invariant features of these motor programs.

According to the first notion, the time-varying muscle activation vector observed in a specific task

condition is generated by the combination of a few basic temporal components each associated

with the synchronous activation of groups of muscles through a constant weighting vector :

∑

(1)

As the task conditions change, the temporal components are invariant while the weighting matrix is

adjusted appropriately.

According to the second notion, the same time-varying muscle activation vector is generated by the

combination of a few time-varying muscle synergies, i.e. time-varying muscle weights ,

appropriately scaled in amplitude shifted in time:

∑

(2)

where is the amplitude scaling coefficient and ti the onset delay for the i-th synergy. In this case, the

time-varying synergies are invariant across task conditions and the changes in the muscle patterns are

captured by changes in the scaling and timing coefficients.

1.3.2 Methods and experimental evidence

Experimental evidence supporting both notions of motor primitives rely on unsupervised learning

(decomposition algorithms) to identify either temporal components and synchronous muscle

weightings or time-varying synergies and combination coefficients from multi-muscle

electromyographical recordings obtained during the performance of a motor task in many different

conditions.

Muscle patterns recorded during human walking at different speeds, with different body weight

unloading, can be reconstructed by five basic temporal components identified with factor analysis (see

Figure 4 and Ivanenko et al., 2004, 2006). Similar components are identified by other algorithms such

as non-negative matrix factorization or independent component analysis (Ivanenko et al., 2005), see

Figure 5. Finally, five temporal components also capture the temporal organization of running muscle

patterns with only one of these components significantly different from those for walking (Cappellini et

al., 2006), see Figure 6.

The spatiotemporal organization of the phasic muscle patterns for fast reaching movements in

different directions in vertical planes (d'Avella et al., 2006) can be reconstructed by a small number of

time-varying muscle synergies identified by an iterative optimization algorithm developed for this

purpose (Figure 7). The reconstruction of muscle synergies is robust against changes in posture and

load, and the amplitude coefficients show cosine directional tuning.

Moreover, modulation of phasic and tonic time-varying muscle synergies captures the variations in the

muscle patterns observed in reaching movements in different directions and with different speeds, as

shown in Figure 8 (d'Avella et al., 2008). These results suggest that the central nervous system might

be using a simple scaling strategy for generating the joint torque profiles appropriate for moving along

a given trajectory with different speeds. As the equation of motion for an articulated arm are invariant

for changes in the movement time scale (r) if the dynamic components of the joint torques are scaled

in amplitude by r2 and anti-gravity torque components are not changed. Thus, if a torque profile

adequate for reaching a given spatial target at one speed is known, a simple scaling rule allows to

generate the torque profiles for reaching that target, along the exact same path, at different speeds. A

low-dimensional representation of the muscle patterns for reaching in terms of phasic and tonic

muscle synergies would greatly simplify the implementation of such a control strategy. Indeed, the

amplitude coefficients of the phasic synergies were found to scale with speed to a relationship close to

quadratic.

Figure 4. Locomotion program as a characteristics timing of muscle activation (Ivanenko et al., 2004).

Similar five activation components, identified with factor analysis, account for about 90% of variance in the leg

EMG patterns during walking at different speeds and loads.

95%

75%

50%

35%

0%

stance swing

1 km/h

2 km/h

3 km/h

5 km/h

averaged

comp 1

comp 2

comp 3

comp 4

comp 5

stance swing

speed body weight support

LocomotionMotor Program

LocomotionMotor Program

comp 1 32 4

CPG

5 1

TD TDLO

time

…comp 1 32 4

CPG

5 1

TD TDLO

time

…

Figure 5. Basic components. Basic temporal components identified by different statistical approaches (factor

analysis, independent component analysis, non-negative matrix factorization and fitting by Gaussian

components) from normalized EMG patterns during normal walking (Ivanenko et al., 2005).

1.3.3 Implications for robotics

Most models of adaptive modules proposed in the robotics literature or currently being developed as

building block for the AMARSI architecture use a representation of kinematic variables, since robot

dynamics is usually taken care of by low-level feedback control loops. In contrast, the notions of motor

primitives employed by the SLF group are mainly at the muscular, thus dynamic, level. While the

fundamental differences in the nature of the actuators and sensors between biological and robotic

systems might suggest that a notions of motor primitive at the dynamic level are not immediately

relevant for robot control, such statement might not apply to novel biologically inspired robot platform

with compliant actuators. Indeed, one of the features of the muscle-based actuator systems is that

they are naturally compliant and that they allow for adjusting the mechanical impedance by regulating

the amount of co-contraction. How the control of the impedance of multi-joint systems is regulated by

the nervous system and whether the representation and control of impedance relies on motor

primitives (such as, for example, muscle synergies which generate zero output force/torque but control

impedance of the limb along specific directions) are novel and open questions in human motor control

that might be highly relevant for the control of robots with compliant actuators.

Along these lines, a first attempt to combine estimation of the dynamics of reaching movements

through dynamical system with estimation of the impedance parameters required to adapt to

uncertainties during transport was conducted by the technical partner EPFL in Gribovskaya et al.,

2011 (see also AMARSI Deliverable 4.1: Ch.8, Stable Estimator of Dynamical Systems).

77EMGs

SPLE

BIC

TRIC

DELTA

DELTP

TRAPS

TRAPI

LD

RA

OE

OI

EST1

EST9

ESL2GM

Gmed

ILIO

TFL

ADDL

SART

BF

ST

RF

Vmed

Vlat

MG

LG

PERL

SOL

FDB

TA

STER

Factor Analysis

comp 1

comp 2

comp 3

comp 4

comp 5

Gaussian fitting

stance swing

Independent

Component

Analysis

Non-Negative

Matrix

Factorization

EMGsdecomposition

temporal components

N

i

ii residualtft1

)()( Wm

PV ~ 90%

Figure 6. Five basic temporal components for human running (Cappellini et al., 2006). The timing of the

main peak of components is similar for walking and running except for the second component that is shifted in

running.

1.4 Motor primitives defined by learned kinematic components

The partner UniTu takes the viewpoint of machine learning. The work has mainly focused on the

analysis and modelling of kinematics of body movements by the application of learning methods,

which are inspired by the concept of motor primitives form neuroscience. The derived primitive-based

representations have been exploited, on the one hand, for the analysis of motor behaviour and the

perception of body movements. On the other hand, we have developed methods that transform such

primitive-based representations into algorithms that are suitable for the offline and online synthesis of

body movements. In addition, we have used primitive-based representations for the study of

perception of body motion.

B7 km/h

1

2

3

4

5

walking running

6

B7 km/h

1

2

3

4

5

walking running

6

7 km/h

A walking runningSTER

SPLE

BIC

TRIC

DELTA

DELTP

TRAPS

TRAPI

LD

RAS

OE

OI

EST1

EST9

ESL2

GM

ILIO

TFL

ADD

SART

BF

ST

RF

Vmed

Vlat

MG

LG

PERL

SOL

FDB

TA

Gmed

7 km/h

A walking runningSTER

SPLE

BIC

TRIC

DELTA

DELTP

TRAPS

TRAPI

LD

RAS

OE

OI

EST1

EST9

ESL2

GM

ILIO

TFL

ADD

SART

BF

ST

RF

Vmed

Vlat

MG

LG

PERL

SOL

FDB

TA

Gmed

STER

SPLE

BIC

TRIC

DELTA

DELTP

TRAPS

TRAPI

LD

RAS

OE

OI

EST1

EST9

ESL2

GM

ILIO

TFL

ADD

SART

BF

ST

RF

Vmed

Vlat

MG

LG

PERL

SOL

FDB

TA

Gmed

walking running

Figure 7. Reconstruction of muscle patterns for fast-reaching movements by combinations of time-

varying muscle synergies (adapted from d’Avella et al., 2006). Five time-varying synergies (A) extracted from

the phasic muscle patterns recorded during point-to-point reaching movement to targets in different directions on

frontal and sagittal vertical planes explain the variation in the muscle patterns across directions (B) as due to the

selection of a small number of amplitude coefficients (represented by the height of the rectangles below the EMG

traces) and time-shifts (horizontal position of the rectangles). The amplitude coefficients (C, polar representation)

show a simple dependence to movement direction, approximated by a cosine function.

Figure 8. Reconstruction of muscle patterns for reaching movements in different directions and speeds

by combinations of phasic and tonic time-varying muscle synergies (adapted from d’Avella et al., 2008).

Thee phasic and three tonic time-varying synergies (A) extracted from the muscle patterns recorded during point-

to-point reaching movement to targets in different directions on the frontal plane and with different speeds capture

the variation in the muscle patterns (B) by modulation in amplitude and timing of the phasic synergies and in

amplitude only the tonic synergies. The cosine directional tuning of the amplitude coefficients for both synergies

(C) is modulated by speed for the phasic synergies.

1.4.1 Notion of Motor Primitives (MP)

In their learning-based approach UniTu distinguishes temporal primitives in the sense of individual

actions in longer sequences, and spatial primitives that permit the modelling of complex movements

by superposition of a small number of source components, which might include only subsets of the

available degrees of freedom.

1.4.1.1 Temporal primitives

Temporal primitives have been investigated in the context of complex action sequences, such as

forms in martial arts. A key problem in practice is the automatic segmentation of natural action

sequences in temporal segments that correspond to meaningful single actions. This problem has been

addressed by supervised learning approaches. For this purpose, the trajectories where characterized

by a sparse set of robust trajectory features (e.g. zeros of the velocity in individual degrees of

freedom). These features were optimized for segmentation. Individual temporal primitives were then

characterized by training examples and were modelled as a discrete sequence of key features that

corresponded to such vectors of such robust features. Such feature sequences form templates which

then could be matched in a robust manner to the feature sequences extracted from the trajectories by

a special dynamic programming technique (Figure 9). This algorithm made the matching process

robust against missing or additional key features in the matched sequence compared to the template.

After the automatic segmentation the individual temporal primitives were modelled by spatio-temporal

morphable models. This technique permits the very accurate modelling of complex movement

trajectories by interpolation in space-time. Inspired by image morphing techniques that are meanwhile

common in computer graphics, the underlying idea is to characterize classes of similar trajectories by

their space-time shifts against a reference trajectory (like the average of a set of prototypes). Such

space-time shifts can be computed by dynamic time warping. The correspondence shifts of a set of

prototypes relative to a reference trajectory form a basis of a vector space. This permits to model

intermediate movement trajectories by linear combinations of such correspondence shifts (Figure 10).

The technique of spatio-temporal morphable models provides an efficient low-dimensional

parameterization of movement style. It was shown to be efficient for the analysis as well as for the

synthesis of simple movements (Poggio and Giese, 1999; Giese and Poggio, 2000) and also for

movement parameterization form the study of movement perception in psychophysical experiments

(Giese and Lappe, 2002; Giese et al., 2008).

Figure 9. Supervised segmentation of action streams. The original movement is scanned with a time

window. The trajectory segment within the window is compared with template sequences (template primitives)

using an algorithm that is based on dynamic programming. (a) Trajectory segment with robust features s

i . The

segment can be characterized by a sequence of these key features (b) The prototypical primitive can be

characterise by a sequence of corresponding features m

i . Dynamic programming makes the matching robust

against additional or missing key features. (See Ilg et al., 2004 for details).

Figure 10. Modelling of trajectories by linear combination in space-time. (a) Spatio-temporal

correspondence is established between the individual prototype trajectory and a reference trajectory. Each

prototype is characterized by the functions ζn(t) and τn(t) that characterize the spatial and the temporal shifts

against the reference trajectories. (b) New trajectories can be generated by linear combination of the spatial and

temporal shifts, and space-time warping of the reference trajectories with the resulting shifts. (c) Generation of an

action sequence for a humanoid agent by linear combination of a neutral and a happy style. (For details see

Giese & Poggio, 2000; Ilg et al. 2004).

1.4.1.2 Spatial primitives

The approach for the extraction of spatial primitives from trajectory data of UniTu is closely related to

the approaches developed by SLF. The developed methods aim at an approximation of joint angle

trajectories by a minimum set of learned components or source signals that are combined using a

nonlinear mixing model, corresponding to an anechoic mixture. This model has been found to be

most appropriate by a detailed analysis of the approximation of sets of periodic and non-periodic body

movements, comparing different blind source separation algorithms (Omlor and Giese, 2006a; Giese

et al., 2009). The applied model is characterized by the equation:

∑

(3)

Here denotes the angle trajectory, and are the source functions or mixture components.

These functions are interpreted as invariants over different types of movements. The mixing weights

and the delays are the model parameters which are movement specific and which are modified

as part of movement planning and control. The weights determine the amplitude (gain) of the

corresponding source and the delays between the components highlight the importance of timing

for the coordination of joint movements. Mixture models of the type (3) are called anechoic. Detailed

analyses showed that various classes of complex body movements could be approximated by these

models with high accuracy and very few source components, typically less than five (Omlor and Giese,

2006a; Giese et al., 2009).

(a) (b)

(c)

The described model mainly aims at a minimal mathematical parameterization of the trajectories. The

interpretation of model components as primitives is, therefore, less direct than in some biologically

motivated methods:

The source functions components form a function basis that spans the trajectories. For

periodic movements these functions are similar to a Fourier basis. It remains to be clarified

how such functions are related to primitives in terms of controlled components.

Both, weights and delays are movement specific and therefore are modified as part of

movement control. Approximating sets of trajectories by the model (3) and analyzing the

model parameters shows that typically these parameters cluster for individual movement types

and styles (e.g. Roether et al. 2009). In addition the algorithm can be modified in order to

enforce identical parameter values within individual clusters. A primitive is thus best modeled

by a combination of the source function and specific motion-type specific changes of the

parameters and the delays . This approach has been successfully applied for the

modeling and analysis of emotional movement styles and the identification of emotion-specific

movement primitives (Omlor and Giese, 2006b, 2007; Roether et al., 2009 see Figure 10a).

It is relatively easily possible to modify the algorithm in a way that embeds additional priors

and sparseness constraints. This might make it possible to extend the approach for the

extraction of more sources which map directly onto physiologically relevant movement

primitives or physiological modules. Which exact priors are required for maximum

physiological interpretability remains to be discovered. One possibility is priors that enforce

primitives that are spatially localized on the moving face or skeleton of moving bodies.

In an additional set of studies we have extended the described approach for movement synthesis in

real-time (Figure 11). The underlying idea is that solutions of nonlinear dynamical systems, called

dynamic primitives in the following, are mapped onto the source signals identified by unsupervised

learning from the trajectories (Park et al., 2008a, 2008b, 2009). The resulting architecture consists of a

number of dynamic primitives, which fulfill a similar function as CPGs in biological organisms. The

coordination of the generated patterns in presence of noise can be stabilized by introduction of

dynamic couplings between the dynamic primitives. In this context it turns out that compact

representations, based on a small number of sources, show more controllable dynamical behavior

than models, e.g. based on PCA, that include a relatively large number of coupled components or

dynamic primitives (Park et al., 2008a, 2008b; Giese et al., 2009). In addition, we investigated for

simple examples the integration of non-periodic and periodic movement primitives within the same

architecture.

The same architecture can be extended by introduction of dynamical approaches for navigation or the

synchronization between multiple interacting agents in a scene (Giese et al., 2009). This makes the

approach suitable for an online synthesis of the body movements of individuals and also of the

coordinated movements of groups of agents.

Due to the simplicity of the resulting architecture, the framework is accessible for the analysis of the

dynamical stability of the resulting architecture. This has been exploited for the design of the dynamic

couplings between the individual primitives (Park et al., 2008a, 2008b) as well as in recent work for the

analysis and the design of the stability of collective order formation scenarios realized by groups of

human agents (Park et al. 2009; Mukovskiy et al., 2008, 2010). Present work focuses on the analysis

and design of the dynamical stability properties exploiting concepts from contraction analysis (e.g.

Lohmiller and Slotine, 1998; Pham and Slotine, 2007; Mukovskiy et al., 2010).

Figure 11. Application of the extraction of movement primitives by blind source separation. (a) Automatic

extraction of emotion-specific features form gait trajectories, using a blind source decomposisiton model

combined with sparse regressions (from Roether et al., 2009). Colors indicate the joint angle amplitudes

compared to normal walking for angry, fearful, happy and sad walking for the joints indicated along the vertical

axis. Each emotion is characterized by a characteristic profile of changes in joint angle amplitudes. The extracted

features closely match results from psychophysical studies (indicated by the black and white signs). (b)

Architecture for the online synthesis of complex body movements based on learned primitives. Based on the

learned mixing model that is determined by blind source decomposition of a set of training trajectories movements

are generated online. The source signals are generated online by mapping of the solutions of nonlinear dynamical

systems (dynamical primitives) onto the source signals using kernel methods. (c) By coupling of the dynamic

primitives of different agents coordinated crowd behaviour can be self-organized. This example shows the

autonomous formation of a crowd with agents that synchronize their step phases. Taken from Giese et al., 2009.

1.4.2 Methods and experimental evidence

A hierarchical model based on temporal primitives combined with a modelling of individual primitives

by spatio-temporal morphable models has been successfully applied for the modelling and imitation

learning of movements, such as writing or martial arts techniques (Ilg et al., 2004; Mezger et al.,

2005). The approach has been successful not only for movement synthesis but also for the analysis of

movement styles, such as the estimation of skill level from Karate sequences (Ilg et al., 2004). The

low-dimensional parameterization of movement styles by spatio-temporal morphable models has been

shown to be very useful in psychophysical and fMRI experiments on body movements (Giese and

Lappe, 2002; Giese et al., 2008; Jastorff et al., 2006, 2009), as well for the study of the perception of

facial movements (Knappmeyer et al., 2004).

Modelling of movements in terms of spatial primitives has been successfully applied for the study of

the execution and perception of emotional body movements (Roether et al., 2008, 2009; Omlor and

Giese, 2006b, 2007). The synthesis of trajectories based on dynamic primitives derived from learned

spatial primitives has been applied for the online synthesis of human locomotion and non-periodic

movements (Park et al., 2008a, 2008b). This work includes integration of primitives for non-periodic

(a) (b)

(c)

and periodic motion. Also this work demonstrates that the modelling of body movements in terms of

dynamic primitives is highly suitable for the modelling of the behavior of groups of human agents for

example locomoting groups (Figure 11C) or dancing (Giese et al., 2009). More recent work has started

to apply the framework of contraction theory (Lohmiller and Slotine, 1998, Wang and Slotine, 2005) for

the development of stability bounds for systems based on dynamic primitives (Park et al., 2009;

Mukovskiy et al., 2010). This work suggests that this approach might be suitable for guaranteeing the

stability of systems including large numbers of nonlinear dynamic primitives.

1.4.3 Implications for robotics

The concept of temporal primitives has been successfully implemented for the realization of imitation

learning on a Mitsubishi robot arm (Ilg et al., 2004). The concept of learned dynamic primitives is

suitable for an online synthesis of complex full-body movements and the design of complex networks

of dynamic primitives. This makes it potentially interesting for the modelling of complex motor

behaviour on robot platforms. However, this framework at the moment lacks the integration of sensory

feedback signals. This step seems crucial for a transfer of this methodology to a variety of robot

systems and is a central problem of present work. In addition, it remains to be explored in how far

primitives learned from kinematic data can be approximated by control primitives on individual robot

platforms. Contraction theory as a tool for stability analysis in complex systems is very general and

has been applied extensively in the context of robotics and nonlinear control (Lohmiller and Slotine,

1998; Lohmiller and Slotine, 2000; Slotine, 2006; Chung and Slotine, 2009).

1.5 Relationships between the different approaches In order to address possible correspondences between the architecture of movement representations

in humans and in robots, it seems helpful to investigate MPs using maximally similar tasks.

A very general conception of movement primitives (MPs) is given by the approach of UniBi, that

envisions different levels of MPs, some of which overlap with the levels of description of the other

partners. Which levels are relevant depends on the task. For example, particular tasks might primarily

address the higher cognitive levels, while others more require a treatment on the level of muscle

synergies. In addition, the complexity of MPs might change dependent on the level of action

organisation. It seems possible that interactions exist between MPs at a more biological (motor-

driven), perceptually driven and cognitive (intentionally driven) levels of action organisation.

The general approach of UniBi offers many opportunities to interact with the other partners. For

example, the work on MPs based on the perception and the kinematics of body postures of UniTu can

be nicely combined with the approaches of UniBi for the analysis of kinematics and the cognitive

representation of movement keypoints in memory. A combination of these different approaches might

help to learn more about the relationships between kinematic, perceptual and the memory side of

MPs. The same question is related to the kinematic and muscle-based analysis of primitives by SLF.

Specific questions of such interactive research may be how body postures are represented in memory,

or what is the relationship between muscle-activation (SLF), body postures and kinematics (as

modelled by UniTu), and functional-biomechanical constraints in the memory representations of MPs.

A further set of important questions in this context is the interplay of learning processes and the role of

expertise, dependent development and structure formation of MP's in the memory (UniBi) and at the

neuromuscular levels (as treated by SLF).

In general, the approach of UniBi offers a tool for the comparison of representation structures for

complex movements in the long term memory of experts and novices on the basis of different sets of

building blocks, or motor primitives, defined via different methods. Similar approaches can likely be

applied to motor primitives that are defined via trajectory planning or movement planning on joint level,

as analyzed by Weizmann, to muscle activity patterns and muscle synergies, as analyzed by SLF, or

kinematic primitives such as extracted by the methods of UniTu.

Another set of connections exists in the domain of motion segmentation. Differential geometrical

approaches by Weizmann for the determination of invariants might be combined by methods for

supervised and unsupervised segmentation, such as developed by UniTu and also by UniBi. A related

interesting topic, presently being investigated at UniBi, is how human observers spontaneously

segment “abstract”, not object-related, complex movement sequences in different conditions, and how

their decisions are influenced by learning and expertise (Bläsing et al., 2010a). Results of such

experiments could be compared to results of algorithmic methods for movement segmentation in order

to compare human and statistical methods for movement segmentation.

A further problem is the development of methods for measuring of the similarity of different movement

and clustering. UniBi has compared motion segments by applying Procrustes analysis, which is widely

used in shape matching. The resulting movement segments can be applied to the SDA method was

used to analyze the mental representations of movements in long term memory. Other methods, e.g.

based on spatio-temporal correspondence, have been proposed by UniTu. Further similarity measures

might be developed exploiting concepts from differential geometry, such as applied in the work by

Weizmann.

Close connections also exist between the work of Weizmann and UniTu. Motion primitives at the level

of hand or COM trajectories during locomotion can be defined based on the hypothesis that kinematic

strokes at the task level are represented in terms of non-Euclidian variables, and that motion primitives

are invariant under certain group of non-linear transformations. Earlier work, partially carried out in

collaboration between Weizmann and UniTu (Dayan et al., 2007; Casile et al., 2010), has indicated

that similar invariant properties of movement apply also to motion perception. At the same time, work

by SLF (Ivanenko et al., 2010) suggests that internal models of automatic postural responses might

influence perception as well as motor control. This supports that primitives are represented at different

levels, such as formulated in the theory of UniBi.

Work involving compositionality principles at the joint level has also been carried out by Weizmann

group in collaboration with UniTu (Barliya et al., 2009) showing that using a relatively simple oscillatory

model at the joint level it is possible to account for the inter-segmental law of coordination observed

during human locomotion (Borghese et al., 1996; Ivanenko et al., 2008). This work is discussed in

more detail in section 2.2.

Tight connections exist also between the work of SLF and UniTu. The central algorithms for the

unsupervised learning of primitives from EMG and trajectory data are very similar, and a presently

ongoing computational study investigates how the underlying models are mathematically related, and

what are the advantages and disadvantages of the underlying algorithms for practical applications,

and especially for the analysis of data in motor control. Preliminary esults from this study are

presented in section 2.1. Another important aspect is the investigation of the relationship between the

neuromuscular and the kinematic level of MPs. Basic temporal components might be linked to specific

kinematic events, such as the onset of foot lift in walking (Ivanenko et al., 2006). Moreover, scaling in

amplitude and duration of phasic time-varying muscle synergies might underlie the path invariance

observed for reaching movements with different speeds (d'Avella et al., 2008). Here, interesting

connections between the approaches of SLF, UniBi, Weizmann and UniTu might be established.

The approach for the online synthesis of trajectories by nonlinear dynamical systems by UniTu is

closely related to the primitive-based control approaches developed at EPFL (e.g. Ijspeert et al., 2002;

Buchli et al., 2006). Challenges to be solved are to embed the proposed learned dynamic primitives for

complex movements into control architectures and to constrain them in a way that makes them

suitable for the embedding in the existing robot platforms. Here, learning-based approaches and the

concept of stability design in modular systems modelling highly complex motor behaviour, such as

investigated by UniTu, might form a basis for fruitful interactions with the approaches of EPFL that are

based on dynamic primitives that are implementable on the available robot platforms. (C.f. AMARSI

Deliverable 4.1: Ch.2, Dynamical Movement Primitives; Ch.8, Stable Estimator of Dynamical

Systems). Several discussed biological models were based on the notion of dynamical systems in

order to drive the generated motion. Notions of stability and tractability that are easily created by the

non-biological models such as DMP and SEDS (see AMARSI Deliverable 4.1: Ch.2, Ch.8). However,

such models cannot be easily transferred to the analysis of complex biological movements, as some of

the ones described in the previous sections. While this may be an impediment for robotics application,

further development of biologically plausible models may nonetheless contribute to improving the

complexity of the more mathematically oriented approaches in robotics. Such developments would

represent a nice desirable outcome of the AMARSI project.

2 Quantitative comparisons between selected approaches

Beyond the general considerations in the first part of this deliverable, some of the approaches are so

closely related that it makes sense to compare them mathematically or quantitatively based on

empirical or simulated data sets. This applies specifically to the approaches for the identification of

primitives by unsupervised learning by SLF and UniTu, and to the characterization of invariants of

locomotion patterns as studied by WI and UniTu. In the following we give a short report about these

more quantitative comparisons.

2.1 Comparison between the unsupervised learning approaches by SLF and UniTu

2.1.1 Introduction As discussed in the first part of this deliverable, there are currently two main different definitions of

motor primitives at muscle level used by the SLF group: the first one is that of basic temporal

components in the muscle activity patterns (Ivanenko et al., 2004, 2006). The second notion is that of

time-varying muscle synergies (d'Avella et al., 2003; d'Avella et al., 2006; d'Avella et al., 2008). Both

notions rely on the decomposition of EMG datasets in terms of a linear combination of motor

primitives, scaled in amplitude by some scalar weights. In particular, the first one is based on the

following equation

∑

(4)

where is a vector of muscle activations, defining the basic temporal components (scalar

functions of time), being real vectors of scaling weights, and where n is the total number of

primitives.

In contrast, the second notion is based on the following equation

∑

(5)

for which and have the same meaning as above, and where the specify time delays relative

to the vectors . Note that this time the are time-dependent variables, whereas the weights are

not. Furthermore, the elements of the vectors and all scaling coefficients were always

constrained to be non-negative in both models, given the non-negativity of the elements of the vector

. Interestingly, the models in (4) and (5) can be seen as two special cases of the model used by

the UniTu group to study spatial primitives (Omlor and Giese, 2007). This model is described by the

equation

∑

(6)

with representing either a joint-angle time series or, in the other case, an EMG signal at instant t.

It is indeed straightforward to see that each row of (4) can be re-expressed in the form of equation (6)

by equating , , and by considering each coefficient as the i-th

element of the j-th vector . Similarly, each row of (5) can be re-expressed as a specific case of

equation (6) by taking , for , and . Note

however that, for the same dataset , to express eq. (5) with eq. (6) a much larger number of

source signal may be required with respect of the case of expressing eq. (4) with eq. (6). In general,

each muscle (index i) of a time-varying synergy might require a different source , so that

.

Therefore, given an EMG dataset is given, it can be decomposed according to one of the models

presented above. However, the number of the parameters to be identified and their identification

methods differ for model to model. Coefficients and primitives of model (4) were identified by using

standard methods such as Principal or Independent Components Analysis and Non-Negative Matrix

factorization (Ivanenko et al., 2005). Parameters of model (5) were identified by using an iterative

optimization algorithm that identifies shift-invariant multidimensional bases using Matching Pursuits

and NMF (see (d'Avella et al., 2003; d'Avella et al., 2006). For parameter extraction, Omlor and Giese

(2007) exploited an extraction algorithm based on a time-frequency transformation (Wigner-Ville

distribution). Also the computation time is dependent of the model and the identification method. For

all these reasons, it is hard to establish specific criteria that are suitable for choosing between these

models with respect to a given application. In order to establish a benchmark and for a comparison

between these approaches for primitive extraction we chose the following proceeding: Starting from

the generative models (4), (5) and (6), three different EMG-like datasets were generated.

Subsequently, different algorithms for dimensionality reduction based on these three models were

applied to the generated datasets and their results were compared. This quantitative comparison

might help to reveal non-obvious differences between the models and should suggest advantages and

disadvantages of the individual identification methods for synergy extraction from muscle activity.

2.1.2 Methods

2.1.2.1 Generation of artificial data sets

Generation of source signals: For the quantitative validation simulated datasets were generated

automatically. These data sets tried to replicate coarsely the properties of real EMG signals, recorded

from a number of muscles during NumT trials with executions of motor behavior. Different

generative models were tested (see below). All generative models derived the data from a set of

statistically independent EMG-like waveforms (referred to as source signals, or simply as sources, in

the following). For the generation of these signals we used an autoregressive moving average (ARMA)

model, which was based on the following equation

∑ ∑

(7)

where is the order of the autoregressive part of the model (AR), is the order of the moving

average part (MA), are the coefficients of the recursive linear filter, and where are the coefficients

of the non-recursive linear filter. The signal signifies white noise. The ARMA model can be

interpreted as a discrete linear system, with the white noise input and the output signal . The

AR and MA coefficients were first estimated (using the MATLAB function armax.m with , )

from real EMG data. This data was recorded during from the biceps muscle of a subject performing an

elbow flexion. Muscle activity was at 1 KHz for a period of 400 ms. The raw signals had been

amplified, rectified, low-pass filtered with a cut-off frequency of 5 Hz. In addition, it was resampled in

order to the fit a sample time window with time steps). output signals were then obtained

by varying the input noise of the model (MATLAB function sim.m). The noise input was given by a

normally distributed random vector of length . Positive Independent component analysis (a particular

version of independent component analysis with non-negativity constraints in the output) was applied

to the output signals of the ARMA model in order to obtain a set of non-negative statistically

independent waveforms. Finally, these signals were normalized by division by their maximum values.

Generative models: The generated source signals were subsequently used to generate datasets

simulating the EMG activities of muscles during hypothetical experimental trials. The simulated

EMG-like datasets were characterized by an intrinsic modular structure. The goal of the analysis using

dimension reduction methods was to extract the basic primitives, and the underlying structure from this

simulated data. The simulated datasets were generated by mixing together a set of primitives

derived directly from the source signals assuming different mixture models that reflected the structures

of the generative models which were underlying the different unsupervised learning methods. In the

following each algorithm will be described along with the model that underlies the corresponding

simulated EMG datasets.

2.1.2.2 Tested dimensionality reduction algorithms

1) Non-Negative Matrix Factorization (NMF): NMF is a standard algorithm for multivariate analysis

where a matrix is factorized into two matrices, and , so that

(8)

The key aspect of this method is that all the elements of the matrices and are imposed to be non-

negative. Therefore, if is thought to be a matrix in which each row represents an EMG signal.

According to equation (8) each EMG signal in results from the linear combination of the rows of the

matrix scaled by the elements of the rows of the matrix .

According to this mixing model, one simulated EMG dataset could be generated by multiplying a

matrix of dimension by (in which the rows were independent source signals) by a

by matrix of positive random numbers, drawn by a uniform distribution in the interval

(0,1). different datasets were then obtained by changing each time the elements of . Note

that in this case each primitive coincides with a source signal.

2) Blind source separation (or anechoic demixing, which from now on we will refer to as An) is an

unsupervised learning algorithm that approximates signals by linear superposition of components with

signal-specific time delays (Omlor and Giese, 2007). Given a simulated EMG matrix , each row

signal can be expressed as

∑

(9)

where are scalar weighting factors, each is a time source signal and the are time delays

between source signals and the signals . Note that no non-negativity constraints are imposed by

this algorithm and that also in this case each primitive coincides with a source signal with .

Based on the modular organization proposed by this model, simulated EMGs were generated by

combining linearly independent delayed sources weighted by weight coefficients .

Delays were chosen randomly and drawn from an uniform discrete distribution in the interval .

3) Positive anechoic demixing: The positive anechoic demixing (pAn) algorithm works exactly as An,

but additional non-negativity constraints are imposed to the coefficients and the sources .

4) Time-varying synergies: The time-varying synergies extraction algorithm (TV) is a method for

primitive extraction that was described in previous works of the partner SLF (d’Avella and Bizzi, 2005;

d’Avella et at., 2006). The method is based on the following generative model

∑

(10)

where is a vector of real numbers, each component of which represents a specific simulated EMG

activation at time . The vector signal represents the muscle activations for the primitive,

and is a time delay and a non-negative scaling coefficient. All parameters of the model are

constrained to be non-negative. Therefore, given a matrix of simulated EMG signals, the algorithm

identifies, through an iterative optimization process, all the non-negative parameters of the second

member of equation (10).

In our work, to generate simulated EMG dataset having modular structure of the TV model we

considered sources and pooled them in primitive matrices of dimension

by . In this case, a primitive coincided with one of such matrix. We then computed random

numbers that were used as weighting coefficients (whose values were taken form a uniform

distribution with comprised between 0 and 1) and random integer numbers form a uniform

discrete distribution in the interval that specified the time delays . Finally, the primitives were

combined according to equation (10).

2.1.2.3 Performance measures

The approximation quality of the models for the data was characterized by computing the explained

variance of the data that was captured by the fitted dimension reduction model. This measure was

given by the coefficient

‖ ‖

‖ ̅ ‖ (11)

where was the matrix of the actual dataset, the reconstructed values by the fitted model, and

where ̅ is a matrix with the mean values of the data over trials.

To assess the approximation qualities of the algorithms dependent on the compatibility of the

algorithm with the generative model of the data, we simulated three datasets according to equations

(4), (5) and (6). To each one of them we applied NMF, An, pAn and TV. We finally computed the

index for all combinations of simulated models and dimensionality reduction algorithms.

As a second measure, we assessed the similarity between original and extracted primitives. This was

done by computing the maximum of the scalar products between original and recovered primitive over

all possible time delays. In detail, we considered the two sets of original and reconstructed primitives

for a specific model (NMF, An or TV). Then, given two normalized primitives and , we computed

the maximum of their scalar product over all possible delays for the second primitive, that is

(12)

where indicates the second primitives delayed in time by j time steps. For NMF and An, and

were already vectors (of length ). Differently from NMF and An, for TV and were matrices of

dimension by . In this case, before computing the scalar product, and were rearranged

the entries in form of vectors by concatenating them in rows with a length of . By definition,

can only adopt values between 0 and 1. For all possible pairs of primitives in the two groups the

corresponding values were computed. The pair with highest similarity was selected and the

corresponding primitives were removed from the two groups of primitives. The similarities between the

remaining primitives was then computed, and the best matching pair of primitives was selected and

removed from the original and reconstructed model. This procedure was iterated until all primitives

were matched.

2.1.3 Preliminary results and discussion

In the following, we present results obtained with the dimensionality reduction algorithms applied to

datasets with , , and . Table 1 shows the R2 values that quantify the

reconstruction accuracy for the generated datasets with the different algorithms. Table 3. Approximation quality for the different generative models obtained with the different dimensionality reduction algorithms. Rows correspond to the generative models and columns to the algorithms.

Data simulated with the NMF generative model was perfectly reconstructed by NMF. However, this

failed for the accurate reconstruction of datasets derived from generative models with specific

temporal structure (An and TV). The anechoic algorithms (An and pAn) resulted in general in the

highest performance. However, for the dataset derived from an anechoic mixture model (second row

in Table 3) pAn performed better than An. This makes sense since this algorithm implies non-

negativity constraints for the estimation of the parameters that match the constraint of the generating

model. The TV algorithm performed perfectly for the reconstruction of the corresponding generated

dataset, but failed for the other datasets. The most interesting aspect is whether the algorithms are

able to retrieve the original primitives by applying dimensionality reduction to the simulated datasets.

This was evaluated by assessing the similarities between the primitives that were used to generate a

dataset, based on a specific modular structure (i.e NMF, An or TV), and the primitives extracted by the

corresponding algorithm that is based on the same generative model. One example is shown in Figure

12A, showing that NMF could perfectly retrieve the original primitives (corresponding to the very high

value of the measure S). A different situation arises for the An dataset, for which the sources signals,

weights and delays where all non-negative. In this case the An algorithm, which does not exploit this

positivity constraint results in reasonable approximation of the original data, but a small similarity

between the original and the recovered primitives (Figure 12B). This is different for the algorithm pAn,

which results in high approximation (Table 3) and in addition perfectly retrieves the original source

components (Figure 12C). The same is true if this algorithm is applied to the dataset TV (Figure 12D).

While the presented results are still preliminary, they seem to represent a useful starting point for a

further validation of the different algorithms. In addition, this comparison points to interesting

NMF An pAn TV

NMF 1,00 0,99 1,00 0,63 An 0,74 0,92 0,99 0,43 TV 0,84 0,97 0,98 0,99

theoretical questions, such as why An, and in particular pAn, always perform well for many of the

tested datasets. As expected from statistical learning theory, a good capability of reconstructing the

original dataset is not always a guarantee a good recovery of the original primitives. A good

understanding of these issues seems essential for any analysis of biological data using such statistical

methods.

Some works based on the study of muscle synergies have demonstrated that scaling and temporal

parameters associated with muscle synergies can be task-dependent and differentiate between

different behaviors (Overduin et al, 2008), or healthy and impaired motor function (Cheung et al,

2009). Therefore, an thorough understanding whether such considerations can be made, independent

on the applied model for muscle synergies, It seems thus to be of essential importance for scientist in

the motor control to assess which claims can be derived from the parameters of such learned models,

Figure 12. Comperison between actual and extracted primitives. A. Primitives extracted by NMF from the

NMF generated dataset. B. Primitives extracted by An from the dataset generated by the An model. Remark

that all generated sdources were constreained to be non-negative. The algorithm An extracts malso negative

signals because no constraints were imposed on the the extracted sources. C. Primitives extracted by pAn from

the An generated dataset. D. Primitives extracted by TV from the TV the generated dataset.

2.2 Inverse kinematics and computational constraints at the joint level (joint work of WIS and UniTu)

Work involving compositionality principles at the joint level conducted by the Weizmann group in

collaboration with UniTu (Barliya et al., 2009) shows that based on the use of a relatively a simple

oscillatory model at the joint level, it is possible to account for the inter-segmental law of coordination

observed during human locomotion (Borghese et al., 1996; Ivanenko et al., 2008). Application of the

blind source separation analysis developed by UniTu to gait trajectories resulted in models for

locomotion that closely resemble the mathematical model developed by the group at the WI. Earlier work shows that the redundancy in the control of walking can be efficiently parameterized if the

trajectories of the lower limb are characterized by elevation angles, i.e. the angles between the

segments and the cardinal axes in the external frame of reference. In this case, the trajectories of the

elevation angles of thigh, shank and foot are lying within a two-dimensional plane, effectively

eliminating one available degree of freedom (Borghese et al., 1996; see Figure 13). The analytical model for the temporal variation of the elevation angles is described in detail in Barliya

et al. (2009). It turns out that these angles were well approximated by the simple sinusoidal time

dependence:

tAa sin (13)

In addition, it was shown that under the assumption that the natural frequencies fulfill the condition

FST (T=thigh, S=shank, F=foot),the orientation of the plane is very well approximated by

2121

3131

3232

sin

sin

sin

AA

AA

AA

n (14)

The above expression for the normal vector to the plane is a function of the amplitudes and phases of

the sinusoids that describe the elevation angles, but not their frequencies. This predicted normal to the

plane differs from the actual normal as computed by using PCA by less than 3°. Figure 14 illustrates

this result.

Applying the blind source separation algorithm developed by UniTu (Omlor and Giese, 2006a) it was

found that the sources used to describe joint rotations during human locomotion are similar to the

Fourier based description of joint rotations of the different leg segments applied by Barliya et al.

(2009).

A very important question that is still open in the motor control area is: given the kinematic redundancy

that exists in the mapping from task to joint levels, how are elementary primitives or strokes identified

at the task level map into possible sets of MP or elementary motions at the joint level. In another study

currently carried out in collaboration between Weizmann and UniTu (Barliya et al., 2010) it was found

that the mapping from hand to joint spaces can be relatively simple and may involve the anechoic

mixture of similar sources both at the joint and task spaces, but only when using quite specific

kinematic representations of the joint coordinates which is described in an absolute extrinsic frame of

reference. Moreover, such similarity between task and joint related sources was not found for

alternative representations. The fact that two spaces share the same set of sources might be

exploited.

One direction that is currently explored is how this set of sources can serve as a mediator between the

task and joint spaces. The observation that both spaces share one set of sources only when the joint

space is represented in a particular manner has led us to further investigate another old question, that

of representations. The tendency for “laws” to emerge when biomechanical systems are represented

in an extrinsic frame of reference is consistently observed. A concrete example is the law of

intersegmental coordination which was just described above and holds only for elevation angles that

Figure 13. Elevation angles of the different leg segments during one gait cycle as function of time (left),

and as angle-angle plot (right). The trajectories lies in a two-dimensional plane, reducing the effective number

of degrees of freedom. (From Barliya et al., 2009.)

Figure 14. Two planes illustrating the average difference (2.73°) between the plane constrain derived

from the original data by a PCA analysis and the plane predicted by the analytical model (from Barilya et

al., 2010).

are described in an external frame of reference. This work has laid the basis for future studies

addressing the question what are the relations between motor primitives at the task and joint levels.

References:

Abeles M., Bergman H., Gat I., Meilijson I., Seidemann E., Tishby N., Vaadia E. (1995) Cortical activity flips

among quasi-stationary states. Proc. Natl. Acad. Sci. USA, 92, pp. 8616–8620

Barliya A., Omlor L., Giese M.A., Flash, T. (2009) An analytical formulation of the law of intersegmental

coordination during human locomotion. Experimental Brain Research, 193(3), pp. 371–385.

Barliya A., Omlor L., Giese M.A., Flash T. (2010) The human arm redundancy problem revisited. Abstract

for the Society of Neural Control of Movement, NCM 2010.

Bennequin D., Fuchs R., Berthoz A., Flash T. (2009) Movement Timing and Invariance Arise from Several

Geometries. PLoS Comput Biol., 5(7). Published online.

Bernstein, N.A. (1947). O postrojenii dvizenij (Über den Aufbau der Bewegungen). Moscow, Megdiz.

Bläsing B., Puttke M., Schack, T. (Eds) (2010a) Neurocognition of dance. Psychology Press: London.

Bläsing B., Maycock J., Bockemühl T., Ritter H., Schack T. (2010b) Motor synergies and object

representation in virtual and real grasping. International Conference on Applied Bionics and Biomechanics,

ICABB, in press.

Bläsing, B. & Volchenkov (2010). Spatio-temporal Analysis of Full-body

Movement in Classical Dance. Scientific Computing for the Cognitive Sciences - SCCS10, Ruprecht-Karls-Universität, Heidelberg.

Borghese N.A., Bianchi L., Lacquaniti F. (1996) Kinematic determinants of human locomotion. J. Physiol.

494, pp. 863–879.

Buchli J., Righetti L., Ijspeert A. J. (2006) Engineering entrainment and adaptation in limit cycle systems -

from biological inspiration to applications in robotics. Biological Cybernetics, 95, 6, pp. 645–664.

Cappellini G., Ivanenko Y.P., Poppele R.E., Lacquaniti F. (2006) Motor patterns in human walking and

running. J. Neurophysiol., 95, pp. 3426-3437.

Calinon S., D'halluin F., Sauser E., Caldwell D., Billard A. (2010) Learning and reproduction of gestures by

imitation: An approach based on Hidden Markov Model and Gaussian Mixture Regression. IEEE Robotics

and Automation Magazine, 17, pp. 44-54.

Casile A., Dayan E., Caggiano V., Hendler T., Flash T., Giese M.A. (2010) Neuronal Encoding of Human

Kinematic Invariants during Action Observation. Cerebral Cortex, 20(7), pp. 1647-1655.

http://cercor.oxfordjournals.org/cgi/content/full/20/7/1647

Cheung V.C.K, Piron L., Agostini M., Silvoni S., Turolla A., Bizzi E. (2009) Stability of muscle synergies for

voluntary actions after cortical stroke in humans. Proc Natl Acad Sci USA 106: 19563–19568.

Chung S.J., Slotine J.J.E. (2009) Cooperative Robot Control and Concurrent Synchronization of

Lagrangian Systems. IEEE Transactions on Robotics, 25(3), pp. 686-700.

d'Avella A., Saltiel P., Bizzi E. (2003) Combinations of muscle synergies in the construction of a natural

motor behavior. Nat. Neurosci., 6, pp. 300-308.

d'Avella A., Bizzi E. (2005) Shared and specific muscle synergies in natural motor behaviors. Proc Natl

Acad Sci U S A, 102(8):3076–3081.

d'Avella A., Portone A., Fernandez L., Lacquaniti F. (2006) Control of fast-reaching movements by muscle

synergy combinations. J. Neurosci., 26, pp. 7791-7810.

d'Avella A., Fernandez L., Portone A., Lacquaniti F. (2008) Modulation of phasic and tonic muscle synergies

with reaching direction and speed. J. Neurophysiol., 100, pp. 1433-1454.

Dayan E., Casile A., Levit-Binnun N., Hendler T., Giese M. A. (2007) Neural representations of kinematic

laws of motion: Evidence for action-perception coupling. Proc. Nat. Acad. Sci., 104(51), pp. 20582-20587.

Endres D., Christensen A., Giese M.A. (2010) Bayesian binning for motion segmentation. Submitted to NIPS.

Flash T., Hogan N. (1985) The Coordination of Arm Movements: An Experimentally Confirmed

Mathematical Model. The Journal of Neuroscience, 5(7), pp. 1688-1703.

Flash T., Henis E. (1991) Arm trajectory modification during reaching towards visual targets. Journal of

Cognitive Neuroscience, 3, pp. 220-230.

Flash T., Hochner B. (2005) Motor primitives in vertebrates and invertebrates. Current Opinion in

Neurobiology, 15(6), pp. 660-666.

Flash T., Handzel A.A. (2007) Affine differential geometry analysis of human arm movements. Biological

Cybernetics, 96(6), pp. 577-601.

Giese M.A., Poggio T. (2000) Morphable models for the analysis and synthesis of complex motion patterns.

International Journal of Computer Vision, 38(1), pp. 59-73. http://www.compsens.uni-tuebingen.de/pub/pdf/giesepoggio00.pdf

Giese M.A., Lappe M. (2002) Measurement of generalization fields for the recognition of biological motion.

Vision Researech, 38, pp. 1847-1858. http://www.compsens.uni-tuebingen.de/pub/pdf/genfpf.pdf

Giese M.A., Thornton I.M., Edelman S. (2008) Metrics of the perception of body movement. Journal of

Vision, 8(9):13, pp. 1-18. http://www.compsens.uni-tuebingen.de/pub/pdf/Giese-2008-jov-8-9-13.pdf

Giese M.A., Mukovskiy A., Park A., Omlor L., Slotine J.J.E. (2009) Real-Time Synthesis of Body Movements

Based on Learned Primitives. In Cremers D, Rosenhahn B, Yuille A L (eds): 'Statistical and Geometrical

Approaches to Visual Motion Analysis', Lecture Notes in Computer Science, 5604, pp. 107-127. Springer Verlag. http://www.compsens.uni-tuebingen.de/pub/pdf/LNCS5604proc2009.pdf

Gribovskaya E., Kheddar A., Billard A. (2011) Motion Learning and Adaptive Impedance for Robot Control

during Physical Interaction with Humans. IEEE Int. Conf. on Robotics and Automation. Submitted.

Handzel A.A., Flash T. (1999) Geometric methods in the study of human motor control. Cogn. Stud., 6, pp.

309–321.

Henis E., Flash T. (1995) Mechanisms underlying the generation of averaged modified trajectories.

Biological Cybernetics, 72, pp. 407-419.

Hersch M., Guenter F., Calinon S., Billard A. (2008) Dynamical System Modulation for Robot Learning via

Kinesthetic Demonstrations. IEEE Transactions on Robotics, 24, pp. 1463-1467.

Ijspeert A., Nakanishi J., Schaal S. (2002) Learning attractor landscapes for learning motor primitives. Adv.

in Neural Inf. Process Systems, 15, pp. 1547–1554.

Ilg W., Bakir G.H., Mezger J., Giese M.A. (2004) On the representation, learning and transfer of spatio-

temporal movement characteristics International Journal of Humanoid Robotics, 1(4), pp. 613-636.

http://www.compsens.uni-tuebingen.de/pub/pdf/humano04_ilg.pdf

Ivanenko YP, Poppele RE, Lacquaniti F (2004) Five basic muscle activation patterns account for muscle

activity during human locomotion. J Physiol (Lond) 556:267–282.

Ivanenko Y.P., Cappellini G., Dominici N., Poppele R.E., Lacquaniti F. (2005) Coordination of locomotion

with voluntary movements in humans. J. Neurosci., 25, pp. 7238-7253.

Ivanenko, Y.P., Poppele, R.E. & Lacquaniti, F. (2006) Motor control programs and walking. Neuroscientist,

12, 339–348.

Ivanenko Y.P., d’Avella A., Poppele R.E., Lacquaniti F. (2008) On the origin of planar covariation of

elevation angles during human locomotion. J. Neurophysiol., 99, pp. 1890–1898.

Ivanenko Y.P., Dominici N., Daprati E., Nico D., Cappellini G., Lacquaniti F. (2010) Locomotor body scheme.

Hum. Mov. Sci. Submitted.

Jastorff J., Kourtzi Z., Giese M.A. (2006) Learning to discriminate complex movements: Natural vs.

artificial trajectories. Journal of Vision, 6(8):3, pp. 791-804.

http://www.compsens.uni-tuebingen.de/pub/pdf/Jastorff-2006-jov-6-8-3.pdf

Jastorff J., Kourtzi Z., Giese M.A. (2009) Visual learning shapes the processing of complex movement

stimuli in the human brain Journal of Neuroscience, 29(44), pp. 14026-38.

http://www.jneurosci.org/cgi/reprint/29/44/14026

Knappmeyer B., Bülthoff H.H., Giese M.A. (2004) Facial motion and the carivature effect. In: Kerzel D,

Franz F, Gegenfurtner K (eds): Beiträge zur 46. Tagung experimentell arbeitender Psychologen. Pabst Science

Publishers, Lengerich, Germany, 141.

Krause A.F., Dürr V., Bläsing B., Schack T. (2010) Evolutionary Optimization of Echo State Networks:

multiple motor pattern learning. 6th International Workshop on Artificial Neural Networks and Intelligent Information Processing, (peer reviewed, accepted, 10 page full paper).

Lohmiller W., Slotine J.J.E. (1998) On contraction analysis for nonlinear systems. Automatica, 34(6), pp.

683–696.

Lohmiller W., Slotine J.J.E. (2000) Control System Design for Mechanical Systems Using Contraction

Theory. IEEE Trans. Aut. Control, 45(5), pp. 984-989.

Maoz U., Berthoz A., Flash T. (2009) Complex Unconstrained Three-Dimensional Hand Movement and

Constant Equi-Affine Speed. Journal of Neurophysiology, 101(2), pp. 1002-1015.

Maycock J., Dornbusch D., Elbrechter C., Haschke R., Schack T., Ritter H. (2010) Approaching Manual

Intelligence. Künstliche Intelligenz, in press.

Mezger J., Ilg W., Giese M.A. (2005) Trajectory synthesis by hierarchical spatio-temporal correspondence:

comparison of different methods. ACM Symposium on Applied Perception in Graphics and Visualization

2005, A Corona, Spain, August 26- 28, 2005, pp. 25-32. ACM press

http://www.compsens.uni-tuebingen.de/pub/pdf/APGV05p.pdf

Mukovskiy A., Park A., Omlor L., Slotine J.J.E., Giese M.A. (2008) Self-organization of character behavior

by mixing of learned movement primitives. Proceedings of the 13th Fall Workshop on Vision, Modeling, and

Visualization (VMV), October 8-10, Konstanz, Germany, pp. 121-130. http://www.compsens.uni-

tuebingen.de/pub/pdf/VMV08.pdf

Mukovskiy A., Slotine J.J.E., Giese M.A. (2010) Design of the dynamic stability properties of the collective

behavior of articulated bipeds. In press: In Proc. of 10th IEEE-RAS Int. Conf. on Humanoid Robots,

Humanoids 2010.

Omlor L., Giese M.A. (2006a) Blind source separation for over-determined delayed mixtures. Advances in

Neural Information Processing Systems, 19, pp. 1074-1081.

http://www.compsens.uni-tuebingen.de/pub/pdf/OmlorGiese_NIPS06.pdf

Omlor L., Giese M.A. (2006b) Unsupervised learning of spatio-temporal primitives of emotional gait.

Perception and Interactive Technologies 2006, Lecture Notes in Artificial Intelligence, 4021, pp. 188-192. Springer, New York. http://www.compsens.uni-tuebingen.de/pub/pdf/PIT06p.pdf

Omlor L., Giese M.A. (2007) Extraction of spatio-temporal primitives of emotional body expressions.

Neurocomputing, 70(10-12), pp. 1938-1942. http://www.compsens.uni-

tuebingen.de/pub/pdf/Neurocomputing_LO07.pdf

Overduin S.A., d'Avella A, Roh J., Bizzi E. Modulation of muscle synergy recruitment in primate grasping.

J Neurosci. 23;28(4):880-92.

Park A., Mukovskiy A., Omlor L., Giese M.A. (2008a) Self organized character animation based on learned

synergies from full-body motion capture data. Proceedings of the 2008 International Conference on Cognitive

Systems (CogSys), University of Karlsruhe, Karlsruhe, Germany, 2-4 April, Springer-Verlag, Berlin.

http://www.compsens.uni-tuebingen.de/pub/pdf/CogSys08.pdf

Park A., Mukovskiy A., Omlor L., Giese M.A. (2008b) Synthesis of character behaviour by dynamic

interaction of synergies learned from motion capture data. Skala V. (ed): Proceedings of the 16th

International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision

(WSCG), 4-7 Feb, Plzen, Czech Republic, pp. 9-16.

http://www.compsens.uni-tuebingen.de/pub/pdf/WSCG08.pdf

Park A., Mukovskiy A., Slotine J.J.E., Giese M.A. (2009) Design of dynamical stability properties in

character animation. In: The 6th Workshop on Virtual Reality Interaction and Physical Simulation, VRIPHYS

09, Nov 5-6, Karlsruhe, Proc., pp. 85-94. http://www.compsens.uni-tuebingen.de/pub/pdf/VriPhys2009Fin.pdf

Pham Q. C., Slotine J. J. E. (2007) Stable concurrent synchronization in dynamic system networks. Neural

Networks, 20(3), pp. 62–77.

Poggio T., Giese M.A. (1999) Synthesis and Recognition of Biological Motion Patterns Based on Linear

Superposition of Prototypical Motion Sequences. In: Proceedings of the IEEE Workshop on Multi-View

Modeling and Analysis of Visual Scene, Fort Collins, CO, June 21-23, 1999, pp. 73-80.

http://www.compsens.uni-tuebingen.de/pub/pdf/cvpr99.pdf

Pollick F.E., Sapiro G. (1997) Constant affine velocity predicts the 1/3 power law of planar motion

perception and generation. Vision Res., 37(3), pp. 347-53.

Pollick F.E., Maoz U., Handzel A.A., Giblin P.J., Sapiro G., Flash T. (2009) Three-dimensional arm

movements at constant equi-affine speed. Cortex, 45(3), pp. 325-339.

Polyakov F., Drori R., Ben-Shaul Y., Abeles M., Flash T. (2009a) A Compact Representation of Drawing

Movements with Sequences of Parabolic Primitives. PLoS Comput Biol., 5(7). Published online.

http://dx.plos.org/10.1371/journal.pcbi.1000427

Polyakov F., Stark E., Drori R., Abeles M., Flash T. (2009b) Parabolic movement primitives and cortical

states: merging optimality with geometric invariance. Biological Cybernetics, 100(2), pp. 159-184.

Richardson M.J.E., Flash, T. (2002) Comparing smooth arm movements with the two-thirds

power law and the related segmented-control hypothesis. J. of Neurosci., 22, pp. 8201-8211.

Roether C.L., Omlor L., Giese M.A. (2008) Lateral asymmetry of bodily emotion expression. Current Biology, 18, pp. R329-330. http://www.compsens.uni-tuebingen.de/pub/pdf/RoetherCB08.pdf

Roether C.L., Omlor L., Christensen A., Giese M.A. (2009) Critical features for the perception of emotion

from gait. Journal of Vision, 9(6), 15, pp. 1-32. Association for Research in Vision and Ophthalmology,

Rockville. http://www.compsens.uni-tebingen.de/pub/pdf/Roether-2009-jov-9-6-15.pdf

Schack T. (2004) The Cognitive Architecture of Complex Movement. International Journal of Sport and

Exercise psychology, 2(4), pp. 403-438.

Schack T., Mechsner F. (2006) Representation of motor skills in human long-term-memory. Neuroscience

Letters, 391, pp. 77-81.

Schack T., Hackfort D. (2007) An action theory approach to applied sport psychology. In G. Tenenbaum &

R. C. Eklund (Eds.), Handbook of Sport Psychology (3rd Edition). NJ:Wiley, pp. 332-351.

Schack T., Ritter H. (2009). The Cognitive Nature of Action – Functional Links between Cognitive

Psychology, Movement Science and Robotics. Progress in Brain research: Mind and Motion - The

Bidirectional Link between Thought and Action. Elsevier. pp. 231-252.

Schack T. (2010) Die kognitive Architektur menschlicher Bewegungen – innovative Zugänge für

Psychologie, Sportwissenschaft und Robotik. Reihe „Sportforum“. Aachen: Meyer & Meyer.

Schuetz, C., Klein-Soetebier, T., & Schack, T. (2009). Modeling of

Biomechanical Parameters Based on LTM Structures. In H. Ritter, G.

Sagerer, R. Dillmann & M. Buss (Eds.), Human Centered Robot Systems.

Cognition, Interaction, Technology. (pp. 161-171). Berlin Heidelberg: Springer-Verlag.

Slotine J.J.E. (2006) Modularity, Synchronization, and What Robotics May Yet Learn from the Brain. In:

Kawamura S., Svinin M. M., Arimoto S., Editors, "Advances in robot control: from everyday physics to

human-like movements" Springer, Berlin, New York , pp. 180-200.

Sosnik R., Hauptmann B., Karni A., Flash T. (2004) When practice leads to co-articulation: the evolution of

geometrically defined movement primitives. Exp. Brain Res., 156(4), pp. 422-438.

Stark E., Drori R., Asher I., Ben-Shaul Y., Abeles M. (2007) Distinct movement parameters are represented

by different neurons in the motor cortex. Eur. J. Neurosci. 26, pp. 1055–1066.

Stöckel, T. & Schack, T. (2010, under review). Goal posture representation and end-state comfort planning in

children. Journal of Experimental Child Psychology.

Tscherepanow M., Kortkamp, Kühnel, Helbach, Schütz C., Schack T. (2010) Simulating the Structure of

Mental Representations Using Computational Features. NIPS, under review.

Viviani P., Flash T. (1995) Minimum-jerk, two-thirds power law and isochrony: converging approaches to

the study of movement planning. Journal of Experimental Psychology: Perception and Performance, 21, pp.

32-53.

Wang W., Slotine J.J.E. (2005) On Partial Contraction Analysis for Coupled Nonlinear Oscillators.

Biological Cybernetics, 92(1), pp. 38-53.

Weigelt M., Rosenbaum D.A., Hülshorst S., Schack T. (2009). Moving and memorizing: Motor planning

modulates the recency effect in serial and free recall. Acta Psychologica, 132, pp. 68-79.

Weigelt M., Schack T. (2010) The development of end-state comfort planning in preschool children.

Experimental Psychology, 6, pp. 1-7.

Zentgraf K., Green N., Munzert J., Schack T., Tenenbaum G., Vickers J.N., Weigelt M., Wolfensteller U.,

Heekeren H.R. (2009) How are actions physically implemented. Progress in Brain research: Mind and

Motion - The Bidirectional Link between Thought and Action. Elsevier. pp 305-320.