Utility = f(Vision)- A Review

Perception

“To perceive is also about how to approach and what to do with an object …”

“Perception/cognition is determined by aspects and form of the agent (Embodiment) …”

Affordances

“An affordance is an intrinsic property of an object, allowing an action to be performed with the object. It also depends on the embodiment of the agent performing the action …”

“Objects which are cars for residents of Lilliput, are merely toys for Gulliver… ”

A Condition for Survival

“One of the most basic function of all organisms is the cutting up of environment into classifications by which non-identical stimuli can be treated as equivalent …”

Clustering Visual Input• Click to edit Master text styles

• Second level• Third level

• Fourth level• Fifth level

Tremendous variation in shape !(Hard for state of art algorithms based on appearance to recognize them)

BUT

All are sittable surfaces!(for humans)

Or, dimensionality = 1 in affordance space.

So, the question to ask is:

What are the affordances an object can support given its visual features such as shape, texture and color ?

Why to answer this question ?

• Obtaining semantic clustering of objects Generalization !

• Building vision perception for robotic platforms.

• Generating scene descriptions in a utilitarian framework Visual aid devices for blind !

• For the sake of science !

Points to Note

“The proposition is to use appearance cues as a supplement to affordance learning and not to totally ignore them…”

Shape AffordancesIs not one to one

Continued… Implicit and Explicit Knowledge

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level“Shapes can only represent explicit knowledge ..”

“Knowledge about hooks/fixture is implicit in (b)…”

“If I have seen further it is by standing on ye sholders of Giants”-Isaac Newton

A Survey of efforts in the past

Affordance Learning

From Activity

From Shape From Simulation

Hand Activity

Local Features

Body Activity

Global Features

Interactive Robot

Freeman & Newell [1971]

• Structure is a unit that provides a set of functions.

• Laid down a formalism for• When• How

Can structures be combined to provide required functions.

The first efforts ! (Winston, Binford et al

[1983])

• Functional description of an object cup• ako: A kind of• hq: Has quality

Input to System:

First Vision System using functional information (Connel & Brady [1987])

• Describe functional concepts geometrically.• Generalize !

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Understanding Functional Reasoning ([Di Manzo, Ricci et al 1989])

• Knowledge representation Semantic Networks

• Objects 3D octree models (Synthetic).• Try to account for real-world noise

• Functional Elements: Support, Grasp, Hang, Cut, Equilibrium, Enter, Contain, Pierce, Stop

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Understanding Functional Reasoning ([Di Manzo, Ricci et al 1989])

More Attempts ([Stark et al. 1991-

1994])

• Concept of Knowledge Primitives• Dimensions (length or area of surface)• Relative orientation: between surfaces• Proximity: between surfaces/faces• Clearance : Lack of obstacles in a defined area• Stability: being in rest in certain orientation.

• Pre-define Categories and Sub-Categories

• CAD and Range-sensor data.

([Stark et al. 1991-1994])

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

([Stark et al. 1991-1994])

([Stark et al. 1991-1994])

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

• Categories Considered• Chairs, Tables, Bench, Bookshelf, Bed, Not Known

A Part based approach ([Rivlin et al

1995])

• Extract 4 parts Reason about their relative configuration• Sticks, Blobs, Plate, Strip

Criticism• Highlight the importance of Knowledge representation

• Hard-Coded definitions

• Almost no testing on real world data

• Instead of trying to recognize surfaces for sitting, sleeping, keeping objects ended up recognizing chairs, beds and tables !• Pseudo-functional Space

Using Affordance Cues for Object Detection

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Continued..

• 2 Oranges, 1 Apple, Can, Tissue Packs, Beaker, Bottle• Coarse features generalize the most.• Active Stereo, Multiple Viewpoints

Use of Coarse Features ([Dillman et al.

ICRA 2011])

Learning by ActionsAffordance Learning

Human Actions and Object Context (Moore et al. [ICCV 1999])

• Jointly Model actions and Image features

• Pre-defined object model

• Shape: Pixel area, size of bounding box, L2-distance from known classes

• Action: HMM based hand pose estimation

Results• Click to edit Master text styles

• Second level• Third level

• Fourth level• Fifth level

Interaction Signatures ([Venkatesh et al

ICCV-05])

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Interaction Signatures ([Venkatesh et al

ICCV-05])

Consider only printer, chair, keyboard and paper !

Observing Humans ([Veloso et al. ICRA

2005])

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Objects in Action ([Gupta et al CVPR

2007])

• HOG Initial guess on probability of object in a window.• Reach (Mr)• Manipulation (Mm)• Reaction (Or)

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Objects in Action ([Gupta et al CVPR

2007])

Interactive Learning (Leonardis et al

2009])

• Object Shape Ellipses (Curvature, area, etc)

• Action features Color and Edge histograms

• SVM object features to clustered action features.

Object-Action Recognition ([Kragic

2011])

• Consider• Book, Magazine, Hammer, Pitcher, Box,Cup• Hammering, opening, pouring

• Video Data

• Object Recognition HOG

• Hand pose (velocity, angle b/w joints, orientation) SVM

• Learn a joint model using Factorial CRF.

Learning by Simulation Affordance Learning

Learning Spatial Relations Using Functional Simulation (Sjoo et al [IROS 2011])

• Learn relation between 2 objects• Support• Protection• Constraint• Move Together

• Features• Pose, closest seperation,

area, distance, contact patch area etc.

• Predict Relation given feature.

What makes a chair chair ?• Discussed !

Indoor Scenes• Highly Structured !

• Surface Orientations: Mainly Vertical and Horizontal

• Components• Boundaries

• Walls, Floors, Doors• Furniture

• Tables, Chairs, Beds, Shelves, Cabinets• Actions

• Cups, Bottles, Glasses, Books, Pens, Kitchen Appliances etc.

• Current Proposition Discover the first 2 categories of scene components

Scene Interpretation

Most Relevant Work (Rusu et al [2010])

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Framework

• Kitchen Environment• Co-Register 16 scans Laser and TOF Cameras.

• Bottom and Topmost regions Floor and Roof

• Determine X and Y axes

• Use heuristics on remaining vertical surfaces to get walls.

• Label other vertical surfaces as furniture.

Segmentation

• Click to edit Master text styles• Second level

• Third level• Fourth level

• Fifth level

Furniture Labelling

Moving Ahead (Replacing Heuristics)• Horizontal L-1 features

• Z-Coordinate, Length and Width

• Vertical L-1 features • Height, Floor Distance, Roof Distance, Width

• L2 features • Height, Width• Num Handles, Knobs

• Learn using CRF.

Some Results• Click to edit Master text styles

• Second level• Third level

• Fourth level• Fifth level

• Legend• Horizontal Planes: Floors, Tables, Ceilings• Vertical Planes: Walls, Furniture Candidates• Furniture: Cupboards, Drawers, Kitchen

Appliances

Leftover Objects• Like cups, bottles etc.• Application: Grasping, Manipulation

Geometrical Primitives• Planes, Spheres, Cylinders, Cones, Tori, Edges and Corners

• Use local point features for primitive labeling using CRF.

• Further using point labels, an SVM modeling capturing shape is used for identifying class of object. (4 object classes).

Proposition

Pipeline

Point Cloud

Identify Horizontal and Vertical Surfaces

Surface Normal

Identify floor, roof, Z axis Compute

Features

Walls, X and Y Axes

Segmentation Normal

Edges

Observe Clusters

Clustering

Features• Defined for each horizontal/vertical surface

• Orientation of Surface• Area of Surface• Volume of Object• Distance from floor• Distance from walls

• At a 2nd level• Relation with other surfaces in the object

• Metrics• Human Height• Width• Arm Span• Area

Hopeful Objects• Identification

• Walls, Floors, Navigable Spaces

• Emerge by un-supervised clustering (pure geometrical features)• Tables/Desks • Chairs• Beds• Shelves• Almirahs• Doors• Cabinets• Windows• Dustbins

Further Extensions• Poselet driven affordance learning:

• Human moving around in an environment.• Vision system Tracks humans, associates poses and objects.• Supplement object detection by using poses.

• Eg: Recognizing bean bags for sitting.• Predict the affordance pose given the object.

• fMRI Study:• Learn a model by showing common tools.• Use say a screwdriver for hammering would be interesting to

see if it is predicted as hammer or screw-driver.