Hand-Eye Coordination and Vision-based Interaction

Martin Jagersand

Collaborators: Zach Dodds, Greg Hager, Andreas Pichler

Vision-based Tasks

Everyday Physical Interactions:Installing a Light Bulb

Sensory-Motor Coordination

• Humans interact effortlessly with the world through vision and physical contact

• Fundamental: Sensory-Motor transforms

Different issues

• What tasks can be performed?– Camera models, geometry, visual encodings

• How to do vision guided movement?– H-E transform estimation, feedback,

feedforward motion control

• How to plan, decompose and perform whole tasks?

Tasks and Image Specifications

task space error

image space error

image space satisfied

task space satisfied

Task ambiguity

• Will the scissors cut the paper in the middle?

Task ambiguity

• Will the scissors cut the paper in the middle? NO!

Task Ambiguity

• Is the probe contacting the wire?

Task Ambiguity

• Is the probe contacting the wire? NO!

Task Ambiguity

• Is the probe contacting the wire?

Task Ambiguity

• Is the probe contacting the wire? NO!

Camera Models• Internal calibration: Longuet-Higgins 1981

• Weak calibration: Faugeras 1992, Hartley 1992

• Affine calibration: Horaud et al. 1998, 1999

• Stratification of stereo vision: Faugeras 1995

- characterizes the reconstructive certainty of weakly, affinely, and internally calibrated stereo rigs

Caff

C inj

Cproj

C sim

reconstruction up to a bijection of task space

up to a projective transformation of task space

up to an affine transformation of task space

up to a similarity (scaled Euclidean transformation)

Visual Invariance

inj injinjinj

proj

sim

aff aff

proj proj

Task Invariance

A task T(f)=0 is invariant under a group G of transformations, iff x

f V , g G with g(f ) V T(f )=0 T(g(f ))=0n

xn

If T(f ) = 0 here,

T(f ) must be 0 here, if T is G invariant

T(f ) must be 0 here, if T is G invariant

T(f ) must be 0 here, if T is G invariant

T(f ) must be 0 here, if T is G invariant

proj

inj

sim

aff

JacobianEstimation

Estimating a Force Frame

Composite Task: Solving a Puzzle

Modulating Basis Vectors

Human Arm Animation

Training sequence Simulated Animation