3d Pose Detection• Used by Kinect• Accurate when the pose closely matches a stored pose• Inaccurate when novel poses are made• Can often produce shaky movement due to pose snapping

3d Pose Tracking

• Calculate poses based on previous poses and current data• No datasets required• Has issues with local minima

Hybrid

• Store a collection of preprocessed poses• Start session with a detected pose• Track until a failure is hit• Detect that frame, and continue

tracking from there

Pose Tracking in Detail

• Given the current set of pixels, and the calculated poses of the past few frames, calculate the most probable current pose• Repeat the following until error is reduced• Calculate a hypothesized 3d representation (rendered image) using the 3d

camera image (observed data)• Calculate quite a lot about the rendered image:

• Edepth, Eextra, Esilhouette, Eprior

• Adjust the hypothesized pose accordingly• If that fails significantly, “detect” the current pose and try again

Optimizing the Position

• Solve energy minimization problems:• Edepth: Depth data, excluding any rendered foreground pixels that match

background observed pixels• Eextra: The above depth term, but with added calculations where rendered

pixels don’t match with observed pixels• Esilhouette: The difference in silhouettes (pixel is in foreground of one image but

the background of the other)• Eprior: Evaluation of the rendered pose based on the previous rendered poses

More Pose Detecting

• Grab a random subset of pixels and match them to known poses• Throw out uncertain data• 20k poses total

Setup

• User must stand in a specific pose for calibration• Select random subset of pixels to determine widths of limbs

Results

Results

• Roughly 1 in 200 frames reinitialized by the detector• Very fast movement often

causes local minima• On average, 1 failure every

33 seconds• High performance running in

parallel on the GPU

http://www.youtube.com/watch?v=ICFKEOk3SyA&feature=youtu.be

Further Work

• Use skinned mesh models• Increase pose set• Use color data in the

algorithm as well