Date post: | 21-Dec-2015 |
Category: |
Documents |
Upload: | debra-vivien-allen |
View: | 217 times |
Download: | 3 times |
Kinect with People and the Real WorldAndreas Erben
WIN-B210
Session Objectives And TakeawaysSession Objective(s):- Introduce Kinect for Windows V2- Understand capabilities of K4W V2
Key takeaways:- Have an understanding what Kinect for
Windows V2 can do for you- Get pointers to get started
4
• daenet was established in 1998• Gold Certified Partner• WPC Partner of the Year 2005, 2010• Regular “Inner Circle” / “Core Partner”
with Microsoft • Featured in joint Microsoft-Gartner
newsletter and in Gartner Magic Quadrant
Andreas Erben, daenet
• More than 15 years experience in the “Integration” and architecture space
• V-TSP Integration• Consulting, Coaching,
Implementation, Outsourcing• Scrum Master, Trainer, Coach• Passionate about Innovation• Community, User Group,
Presentations
• http://HackAndi.com• [email protected]• [email protected]
Geos:Western Europe (Germany-HQ)Americas (USA)South-eastern Europe
http://www.daenet.com
Overview
Welcome to Kinect• A new kind of Camera, that can see in
3D• A traditional video camera• A 4-element microphone array• A powerful SDK which can:• Recognize that a person is in front of the sensor, and what position their body is in
• A software controllable directional microphone, and speech recognition technology
• And lots more
Wider field of view (horizontally and vertically) for depth and color3x depth fidelity1080p, HD color camera (30fps)Lighting independent IR (30 fps)Improved microphones (zero balanced)25 skeletal joints for six people -> Bio correct skeletal joints (particularly hips/shoulders/spine) plus more points and rotationsThumb tracking, end of hand tracking, Open/ClosedImproved range -> .5M near, 4.5M far (extended depth to 8M)
Kinect for Windows v2 Sensor Features
New sensor shipping now http://aka.ms/k4wv2purchase
New v2 SDK (public now) http://aka.ms/k4wv2sdk
Single set of cross-platform Kinect APIs (Desktop, Store, Xbox)
Create & publish Kinect apps to the Windows Store Still have robust support for WPF & Windows desktop applications
Kinect Unity plugin for both desktop & store apps
What’s New in K4W v2
Usage ScenariosTRAININGRETAIL HEALTHCARETHERAPY EDUCATION
Sensor Components
Power Light
RGB Camera
IR Emitters
Microphone Array
https://www.ifixit.com/Teardown/Xbox+One+Kinect+Teardown/19725
Kinect for Windows V2
Color (1920x1080) at 30 FPS (15 FPS low light)
Infrared (512x424) @30 FPS
Depth (512x424) @30 FPS
Body-index pixel-mask at 512x424 @30 FPSPerson detection and
joint tracking, up to 6 people @30 FPS
Floor plane detection
Transformation between coordinate systems
Event driven and polling-based APIs
J. Sell and P. O’Connor, “The Xbox One System on a Chip and Kinect Sensor,” IEEE Micro, 2014
Kinect Data Sources
Audio
Color Infrared Depth
BodyIndex
Body
High-level architecture
Kinect Drivers
Kinect Runtime
.NetAPI
.NetApps
Native API
Native Apps
WinRT API
Windows Store AppsC#, VB, JS, C++/CX
Physical Kinect Sensor
CPU -> i7 (recommended)RAM -> 4GB (recommended)GPU -> DirectX 11 (required)USB 3.0 -> Intel or Renesas chipsets (required)OS -> Windows 8.0/8.1 (required)
Recommended Hardware
Minimize per-frame allocationsLow latency – high throughputExpose low level data and high level interpretations
Design Principles
Kinect Data Sources
• Color Sensor• 1920 x 1080 • 30 or 15 fps, based on lighting
conditions
• IR Sensor• 512 x 424• 30 fps
Two Physical “Image” Sensors
• 1920 x 1080 array of color pixels• 30 or 15 fps, based on lighting
conditions
• Mirrored image to match other formats
ColorFrameSource
• Pixel Data• 16-bit IR intensity value
• Ambient light removed
InfraredFrameSource
• Range: 0.5 - 8 meters• Pixel Data
• 16-bit distance in millimeters from the sensor’s focal plane
DepthFrameSource
• Pixel Data • 0 to 5: Index of the corresponding
body, as tracked by the body source• > 5: No tracked body at that pixel
BodyIndexFrameSource
Range is 0.5-4.5 metersFrame data is a collection of Body objects each with 25 joints
Each joint has position in 3D space and an orientation
Up to six simultaneous bodies30fpsHand State on 2 bodiesLean
BodyFrameSource
Data is audio samples captured over a specific interval of timeAudio data is associated with an “audio beam”
A steerable “cone” of focus for audioMay be automatically or manually aimed
AudioFrameSource
Demo
Kinect SDK and Kinect Evolution (DataSources)
Interactions and Speech
Hand pointer gestures
Engagement Targeting
Press
Panning/ZoomAvailable to desktop or store apps as an Opt-In input mechanism with Kinect for
Windows 2.0
Speech Engagement for Global
Commands
Available to desktop apps (ONLY) as an Opt-In input mechanism with Kinect for
Windows 2.0
“Xbox”
Engagement forApp Commands
“Xbox Select”
Xbox Onew/ Kinect
Windows 8.0 or 8.1
w/ Kinect
N/A“Kinect”
or …
“In Experience” Commands
App-defined
grammar
App-defined
grammar
Demo
Interactions
Kinect Studio and Gesture Recognition
Recordable Data SourcesInfrared13 MB/s
Depth13 MB/s
BodyFrame
BodyIndex
Color120 MB/s
Audio32 KB/s
Legend
Record/Play
Record Only
V2.0Now can record AudioPlayback works without sensor attachedAPI access to playback/record
Kinect Studio, only uses these public APIs!
Kinect Studio – Major Improvements
Heuristic
Gesture is a coding problemQuick to do simple gestures/poses (hand over head)ML can also be useful to find good signals for Heuristic approach
Machine Learning (ML) with G.B.
Gesture is a data problemSignals which may not be easily human understandable (progress in a baseball swing)Large investment for productionDanger of over-fitting, causes you to be too specific – eliminating recognition of generic cases
Gesture Recognition
Posture recognition – heuristic/algorithmic
Floor normal
Upper legvector
Angle between Floornormal and bone
Common sense assumptions about posture:• Spine less than 45˚ from
floor normal• Upper legs less than 45˚
from floor normal
Standing
• Spine less than 45˚ from floor normal
• Upper legs more than 45˚ from floor normal
Sitting
• Spine more than 45˚ from floor normal
• Upper legs more than 45˚ from floor normal
Lying
Gesture Builder
New tool, shipping with v2 SDKOrganize data using projects and solutionsGive meaning to data by tagging gesturesBuild gestures using machine learning technology
Adaptive Boosting (AdaBoost) TriggerDetermines if player is performing gesture
Random Forest Regression (RFR) ProgressDetermines the progress of the gesture performed by player
Analyze / test the results of gesture detectionLive preview of results
! Machine learning innovations for your apps for free !
Visual Gesture BuilderRinse andrepeat
Your Application
Record example
gestures using Kinect Studio
Tag gestures using Gesture
Builder
Build and analyze
gestures using Gesture Builder
Preview gestures in
Gesture Builder
Call gesture detector for tracked bodies
Demo
Gesture Recognition
Kinect Fusion and Face
FusionProvides a rich mesh reconstruction of environment and objectsSurface reconstruction done using voxels
Fusion and Face SummaryFace
Offers a great entry-point into easy to use face attributesOperations and outputs are returned in 2D space
HD FaceProvides a high-definition mesh of a faceReconstruction is done by deforming “shape units”Operations and outputs are returned in 3D space! PAY TO PLAY !
Demo
Fusion (3D Reconstruction), and Face
Third Party Support
Third party supportIt’s easy to use Kinect APIs in your Unity3D app or gameKinect Common Bridge is the way to get to other frameworks (e.g. Cinder)Download KCB here -> http://aka.ms/kinectcommonbridgeCinder & openFrameworks -> http://aka.ms/k4wccsetup
Example: Motion Metrics
Experimental Implementation - VisualizeMotionMetricsIntegrate functionalities in controlling application
Visualize work per joint for linear and angular motion
Display velocity and acceleration vectors
Display energy expenditure rate and total energy expenditure
Display activity level
Display fall-detection state
Display detected posture
Center of mass
Enable/Disable “Jitter” filters
Enables logging
Demo
MotionMetrics
Evaluation: Energy Expenditure
Sensors of thearmband:
• Accelerometer• Heat flux• Near-Body Ambient temperature• Galvanic Skin Response Sensor• Skin Temperature
Perform physical exercise with 2 different models of a BodyMedia SenseWear armband worn at the same time
Test subject:• Male• 43 yrs• 178 cm• 72.6 kg• Non-smoker• right handed
Idea: Compare Kinect-based EE estimation with third-party EE estimation method
Evaluation: Validation of adjustments
-1 1 3 5 7 9 11 13 15 170
2
4
6
8
10
12
14
BodyMedia Kinect EE estimation comparison after corrective factors
Armband1 Armband2 KinectLin KinectAng KinectAvg
Runin place Pause
IntenseSquatsAndJumps
Lighter butmixed intensityrunningjumps/squats
Warmuprunningin place
Cooldown
Talk about Kinect for your organizationGet Sensor/Adapter: http://aka.ms/k4wv2purchaseDownload the SDK: http://aka.ms/k4wv2sdkTry samples and demos in the SDKTry apps and demos in Windows StoreBe creative, Brainstorm, Organize a hackathonCheck for announcements, events: http://kinectforwindows.com
Call to action
Questions & Contact
Andreas Erben
+1 (206) 349 2367+49 (179) [email protected]@daenet.comHackAndi.com
WIN-B210 Kinect with People and the Real WorldWednesday, October 29 5:00 PM - 6:15 PM Room: Hall 8.1 Room H
WIN-B308 The New Cocktail: 1 Tablet + 1 PC + 1 Phone + 1 Kinect + 1 Wall, Served Up on a CloudFriday, October 31 10:15 AM - 11:30 AM Room: Hall 8.1 Room K
WIN-B309 Kinect in Your Apps - Build to Amaze! Friday, October 31 8:30 AM - 9:45 AM Room: Hall 8.0 Room F4
Find me later: [email protected]
Related content
General Info & Blog -> http://kinectforwindows.com
Track resources
Purchase Sensor -> http://aka.ms/k4wv2purchasev2 Preview SDK -> http://aka.ms/k4wv2sdk
Developer Forums -> http://aka.ms/k4wv2forum
Speaker’s blog -> http://hackandi.com
Windows 10http://aka.ms/trywin10
Stop by the Windows Booth to sign up for the Windows Insider Program to get a FREE Windows 10 T-shirt, whiles supplies last!
Windows Springboardwindows.com/itpro
Windows Enterprisewindows.com/enterprise
Windows ResourcesMicrosoft Desktop Optimization Package (MDOP)microsoft.com/mdop
Desktop Virtualization (DV)microsoft.com/dv
Windows To Gomicrosoft.com/windows/wtg
Internet Explorer TechNet http://technet.microsoft.com/ie
Resources
Learning
Microsoft Certification & Training Resources
www.microsoft.com/learning
Developer Network
http://developer.microsoft.com
TechNet
Resources for IT Professionals
http://microsoft.com/technet
Sessions on Demand
http://channel9.msdn.com/Events/TechEd
Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC
TechEd Mobile appPhone or Tablet
QR code
Evaluate this session
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
BACKUP
Fusion
2Create Reconstruction• Pass in a set of parameters defining the environment
and processing type• Outputs the reconstruction
3Other considerations• Can enable color to be lifted from the scan• Experimental pose finders (6DoF trackers) are also
included to play with
1Process Function• Provides “six degrees of freedom” transform• Can be rendered and use to begin the Fusion
Fusion | Face | HD Face
Face
Detection
Alignment
Orientation
Expressions
• Outputs a bounding box around the face
• Can be visualized in color or IR
• Identifies 5 facial landmarks on the face
• Operation performed in color or IR
• Returns quaternion of the head joint with respect to the sensor
• Quaternion prevents gimbal lock
• Provides classifiers for happy, left / right eye open, engagement, mouth open and mouth moving
Alignment and detection performs the operation in IR, but are converted to color using the coordinate mapper for output
Fusion | Face | HD Face
HD Face
Face Model Builder• Once of cluster of
classes that implement the capture interaction
• Interactive API
• Provides collections status and evaluation of frames you’ve collected
1
Face Model• Set of 94 shape units,
scale, hair color, and skin color
• All adjustments are set against an “average face” mesh that is deformed by the shape units acquired during the face capture
2
Use the Mesh• The mesh is analogous
to most animation meshes for easy application to other rigs
• Mesh topology is the same for all faces represented in a standard way (numTriangles, numVertices, etc.)
3
UX is a non-trivial consideration in a good implementation of the Face Model Builder.
Fusion | Face | HD Face
Fall detection
“Violent” fallInterval
to fall down
•Sum of max downwards force per joint over threshold
•Important joint (spine, shoulders, hips) close to floor
Interval to
recover
•Important joint locations above threshold height
“Slow” FallInterval to fall down
•Head position moves downward more than threshold value•Base of spine and head close to floor
Interval to recover
•Head and spine not in recovery threshold distance
Physical formulas
Body model, parameters
(WPF) rendering engine
(.NET)EventHandlers
Kinect for Windows SDK
APIs
Velocity, Acceleration,
Energy...
Activity & Fall detect.
Rule evaluation, fall detection state
machine
Activity Level,Fall detection
rules,...
K4W Runtime
Kinect V2 Camera
Kinect Drivers
KinectService
Posture recognition
Posture classification
Posture templates or
rules