Date post: | 30-Dec-2015 |
Category: |
Documents |
Upload: | ashlyn-cain |
View: | 216 times |
Download: | 0 times |
Coding4Fun: Build Fun, Cool, Commercial Applications Using the Kinect for Windows SDK Dan FernandezDirectorMicrosoft Corporation
Rick BarrazaSenior Technical Evangelist
Microsoft Corporation
DEV330
AGENDA
Kinect OverviewManaging SensorsKinect Data – Color, Depth, SkeletonKinect StudioFace TrackingAudio Programming
KINECT IS TWO THINGS
HardwareKinect for Xbox 360Kinect for Windows
SoftwareKinect for Windows Runtime 1.5
For deploying Kinect applications
Kinect for Windows SDK 1.5API’s for Color, Depth, Skeletal tracking, and Audio
Developer Toolkit 1.5.1Face Tracking, Kinect Studio, Kinect Controls, Samples, etc.
KINECT SENSOR
IR Emitter Color SensorIR Depth Sensor
Tilt Motor
Microphone Array
KINECT CAMERA ANGLES
Horizontal
Vertica
l
KINECT API BASICS
Manage Kinect state ConnectedEnable Color, Depth, Skeleton, AudioStart Kinect Streams
Get DataEvents – AllFramesReadyPolling – OpenNextFrame
IMAGE PROCESSING 101
BGR32 Format – Every Pixel (0,0 | 0,1 | 0,2) has blue, green, red, empty
B G R Empty
255 255 255 0
Kinect ImageSizes: 80x60, 320x240, 640x480DPI: 96Stride: # of bytes per single line:
Width (320 or 640) x 4 bytes (B,G,R,E)
B G R Empty
255 255 255 0
320x240
640x480
80x60
CAMERA FUNDAMENTALS
DEMO
DEPTH DATA
Returns the distance and player index for every pixelEx: 320 x 240 = 76,800 pixels
DistanceDistance in mm from Kinect ex: 2,000mm (6.56 feet)
PlayerIdentify which player index is shown (1-6 or 0 for no player)
Depth is a 16-bit short D D D D D D D D D D D D D P P P
15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
Depth Player Index
DEFAULT/NEAR MODE RANGE
Near Default
0.4m/1.3 ft 0.8m/2.6 ft
DISTANCES – KINECT BETA
1.3’ 2.6’ 9.8’ 13.1’ 26.2’
.4 .8 3 4 8
DefaultMode
Near Mode
Feet
Meters
Unknown Normal
DISTANCES – KINECT V1+
1.3’ 2.6’ 9.8’ 13.1’ 26.2’
.4 .8 3 4 8
DefaultMode
Near Mode
Feet
Meters
Unknown Too Near Normal Too Far
DEPTH FORMULASDepth datashort[] allData = new short[depthFrame.PixelDataLength];
Get depth for a point (X:160, Y:120, Width: 320)int index = (width * y) + x; // 38,560
Get X, Y for an index (Index: 5,000, Width:320)int x = index % width; // 200int y = index / width; //15
Distance Formulashort depthPoint = allData[index];int depth = depthPoint >> DepthImageFrame.PlayerIndexBitmaskWidth; // 2,000
Player Formulaint player = depthPoint & DepthImageFrame.PlayerIndexBitmask; // 1
DEPTH DATA
DEMO
SKELETAL TRACKING
Max of two skeletons per Kinect6 Player Proposals
HandLeft
WristLeft
ShoulderLeft
Head
ElbowLeft
ShoulderRight
HandRight
WristRight
ElbowRight
ShoulderCenter
FootLeft
AnkleLeft
KneeLeft
HipLeft
HipCenter
HipRight
FootRight
AnkleRight
KneeRight
Spine
Seated (10)
Default (20)
COORDINATES
Z AxisX Axis
Y AxisJoints in meters from camera
Negative
Positive
Positive Negative
Positive
JOINT TRACKING & SMOOTHING
Each joint has associated tracking stateTracked, Not tracked, or InferredInferred - Occluded, clipped, or low confidence joints
Use TransformSmoothParameters to smooth joint data to reduce jitter
Kinect transform smoothing uses Holt Double Exponential Smoothing - http://en.wikipedia.org/wiki/Holt-Winters
Skeleton.ClippedEdges = FrameEdges.Top;
PHYSICAL INTERACTION ZONE
Fatigue Kills Gestures Use Coding4Fun ScaleTo extension method
SKELETAL TRACKING & KINECT STUDIO
DEMO
KINECT AS ANOFFICE MONITOR.
KINECT AS ANOFFICE MONITOR.
KINECT FOR MONITORING
DEMO
KINECT SERVICE – PROJECT DETROIT
http://download.codeplex.com/Download?ProjectName=kinectservice&DownloadId=336928
Kinect data over the network using socketsColor, Depth, Skeleton, and AudioClient libraries for WPF, Windows Phone & WinRT
FACE TRACKING
Real-time tracking of multiple faces2D/3D Mesh and Points3D Head PoseAnimation Units
Z Axis
3D HEAD POSE
Pitch
+20
-20
Yaw
-45 +45
Roll
-90 +90
FACE TRACKING
DEMO
SOUND POSITION
Sound Source Angle – The angle and confidence level of where audio is coming fromBeam Angle – The angle used to record audio that you can set as a “directional microphone”
SPEECH RECOGNITION GRAMMAR
<!-- Confirmation_Yes._value: string ["Yes"] --><rule id="Confirmation_Yes" scope="public"> <example> yes </example> <example> yes please </example> <one-of> <item> yes </item> <item> yeah </item> <item> yep </item> <item> ok </item> </one-of> <item repeat="0-1"> please </item> <tag> out._value = "Yes";</tag>
var grammar = new Choices();grammar.Add(“yes please");grammar.Add(“yes");grammar.Add(“yeah");grammar.Add(“yep");grammar.Add(“ok");
Ensure AutomaticGainControl = falseSample : C:\Program Files\Microsoft SDKs\Speech\v11.0\Samples\Sample Grammars
Speech Recognition: English/Great Britain, English/Ireland, English/Australia, English/New Zealand, English/Canada, French/France, French/Canada, Italian/Italy, Japanese/Japan, Spanish/Spain and Spanish/Mexico
VOICE-ENABLED TIC TAC TOE
DEMO
JOINT ORIENTATION
Rotation in camera space, or amount of rotation from parent bonePlayer orientation stored in HipCenter, or ShoulderCenter when seatedMatrix or Quaternion format for animations
BOXINGBOTS
DEMO
Related Content
BoxingBots – In Expo Hall
Resources
Connect. Share. Discuss.
http://europe.msteched.com
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Resources for Developers
http://microsoft.com/msdn
Evaluations
http://europe.msteched.com/sessions
Submit your evals online
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to
be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS
PRESENTATION.