EEC-492/592EEC-492/592Kinect Application Kinect Application
DevelopmentDevelopmentLecture 10Lecture 10
Wenbing ZhaoWenbing [email protected]@ieee.org
Skeleton Smoothing Skeleton data can be smoothed by registering a set of
smooth parameters As a parameter to SkeletonStream.Enable(….);
// create the smooth parametersvar smoothParameters = new TransformSmoothParameters{ Correction = 0.1f, JitterRadius = 0.05f, MaxDeviationRadius = 0.05f, Prediction = 0.1f, Smoothing = 1.0f};// Enable the skeleton stream with smooth parametersthis.sensor.SkeletonStream.Enable(smoothParameters);
Smoothing Parameters Correction: specifies the amount of correction
needed for the raw data. Range [0.0, 1.0] The value must be within the range of 0 to 1.0 and the
default value is 0.5. With lower values more correction is applied, the raw data is
corrected, and the data looks smoother Smoothing: determines the amount of smoothing
applied while processing. Range [0.0, 1.0] The value must be within the range of 0 to 1.0 and the
default value is 0.5. The larger value, the smoother skeleton data, however, it
increases the latency Zero value => you will get the raw skeleton data
Smoothing Parameters JitterRadius: limit the radius value for jittery data, >=0
Measured in meters and the default value is 0.5. If the position of a jitter is outside the set radius, it is corrected to be
positioned at the radius MaxDeviationRadius: max limit of the deviation that is
allowed to be considered for determining a jitter, >=0 points outside of the MaxDeviationRadius range are not considered
as jitter default value is 0.04 meters
Prediction: number of frames predicted into the future, >=0 default value: 0.0 A value greater than 0.5 will likely lead to overshoot when the data
changes quickly
Kinect SDK Smoothing Algorithm Holt double exponential smoothing algorithm is used to reduce the jitters from skeletal joint data The smoothing algorithm applies to each set of data and
calculates a moving average based on the previous set of data
During the calculation of moving average, it uses the values passed by the smoothing parameter.
Getting Data Frames Together We have covered three (color, depth, skeleton) types of data
streams that are returned by the sensor For a real application, we often need all three types of data
streams We can use a single event AllFramesReady, which will do the
job for all three of them The AllFramesReady event fires when new frames are available
for the color, depth, and skeleton streams
this.sensor.AllFramesReady+=sensor_AllFramesReady;
void sensor_AllFramesReady(object sender, AllFramesReadyEventArgs e){}
Floor Determination Each skeleton frame contains a floor-clipping-plane
vector, which contains the coefficients of an estimated floor-plane equation The equation is normalized so that the physical
interpretation of D is the height of the camera from the floor, in meters
The skeleton tracking system updates this estimate for each frame and uses it as a clipping plane for removing the background and segmenting players
Ax + By + Cz + D = 0
How to Use FloorClipPlane In Your App FloorClipPlane can be used to calculate the height
of each joint with respect to the floor Can be used for validation study, as well as fall
detection How to get FloorClipPlane parameters:
How to calculate the height of a joint:
float A = frame.FloorClipPlane.Item1;float B = frame.FloorClipPlane.Item2;float C = frame.FloorClipPlane.Item3;float D = frame.FloorClipPlane.Item4;
float height = A*joint.Position.X+B*joint.Position.Y+C*joint.Position.Z+D;
Joint Orientation The bone orientation is provided in two forms:
A hierarchical rotation based on a bone relationship defined on the skeleton joint structure
An absolute orientation in Kinect camera coordinates The orientation information is provided in form of quaternions
and rotation matrices for use in different animation scenarios BoneOrientation Class, public properties:
StartJoint: Gets the skeleton joint where the bone starts EndJoint: Gets the skeleton joint where the bone ends HierarchicalRotation: Gets or sets the rotation of a bone relative to its
parent bone, of type BoneRotation AbsoluteRotation: Gets or sets the rotation of the bone relative to
camera coordinates, of type BoneRotation
BoneRotation Class BoneRotation has two properties
Matrix: Gets or sets a matrix representation of the bone rotation Quaternion: Gets or sets a quaternion representation of the bone
rotation
Vector4 struct: has four properties W, X, Y, Z
Matrix4 struct: has 17 properties Identity, of type Matrix4 M11, M12, M13, M14, M21, M22, M23, M24, M31, M32, M33,
M34, M41, M42, M43, M44
public Matrix4 Matrix { get; set; } public Vector4 Quaternion { get; set; }
Access Joint Orientationprivate void DrawSkeletonsWithOrientations() { foreach (Skeleton skeleton in this.skeletonData) { if (skeleton.TrackingState == SkeletonTrackingState.Tracked) { foreach (BoneOrientation orientation in skeleton.BoneOrientations) { // Display bone with Rotation using quaternion
DrawBonewithRotation(orientation.StartJoint, orientation.EndJoint,
orientation.AbsoluteRotation.Quaternion);
// Display hierarchical rotation using matrix DrawHierarchicalRotation(orientation.StartJoint, orientation.HierarchicalRotation.Matrix) } } } }
Build ShapeGame App Create a new C# WPF project with name ShapeGame
It is a much simplified app from the Kinect ShapeGame Only one shape, i.e., a ball is used You are limited to use your righthand to hit the ball Ball only drops from a designated spot down, i.e., no X velocity
Add Microsoft.Kinect reference Design GUI Added WindowLoaded() method in xaml file Adding code
Adding Code Add member variables:
WindowLoaded(): Enable both ColorImageStream and SkeletonStream Register event handler for both ColorFrameReady and SkeletonFrameReady events Initialize the shape (i.e., ball)
KinectSensor sensor;Skeleton[] totalSkeleton = new Skeleton[6];WriteableBitmap colorBitmap;byte[] colorPixels;Skeleton skeleton;Thing thing = new Thing(); // a struct for balldouble gravity = 0.017;
Adding Code The Thing Struct. Make it a private struct inside the MainWindow class
private struct Thing { public System.Windows.Point Center; public double YVelocity; public double XVelocity; public Ellipse Shape; public bool Hit(System.Windows.Point joint) { double minDxSquared = this.Shape.RenderSize.Width; minDxSquared *= minDxSquared; double dist = SquaredDistance(Center.X, Center.Y, joint.X, joint.Y); if (dist<= minDxSquared) { return true; } else return false; }}
Adding Codeprivate static double SquaredDistance(double x1, double y1, double x2, double y2){ return ((x2 - x1) * (x2 - x1)) + ((y2 - y1) * (y2 - y1));}
private void WindowLoaded(object sender, RoutedEventArgs e){ // same as before …. // new code for ball initialization thing.Shape = new Ellipse(); thing.Shape.Width = 30; thing.Shape.Height = 30; thing.Shape.Fill = new SolidColorBrush(Color.FromRgb(0, 255, 255)); thing.Center.X = 300; thing.Center.Y = 0; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width); canvas1.Children.Add(thing.Shape);}
Modify Event Handler for Skeleton Framesvoid skeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e){ canvas1.Children.Clear(); advanceThingPosition(); canvas1.Children.Add(thing.Shape);
// remaining code same as before …..
Point handPt = ScalePosition(skeleton.Joints[JointType.HandRight].Position); if(thing.Hit(handPt)) { this.thing.YVelocity = -1.0*this.thing.YVelocity; }}
Shape Animation Code void advanceThingPosition() { thing.Center.Offset(thing.XVelocity, thing.YVelocity); thing.YVelocity += this.gravity; thing.Shape.SetValue(Canvas.LeftProperty, thing.Center.X - thing.Shape.Width); thing.Shape.SetValue(Canvas.TopProperty, thing.Center.Y - thing.Shape.Width);
// if goes out of bound, reset position, as well as velocity if (thing.Center.Y >= canvas1.Height) { thing.Center.Y = 0; thing.XVelocity = 0; thing.YVelocity = 0; }}
Challenge Task For advanced students, improve the shape game in
the following ways: Keep the hit count, count should reset if ball falls to the
bottom Display the hit count at the ball Consider the angle of the hit to the ball, and adjust the x
velocity accordingly Allow other joints to hit the ball too, such as left hand,
shoulder, etc. Add smoothing to skeleton data
05/06/23EEC492/693/793 - iPhone Application
Development 21