1

Microso(KinectDepthSensing

2

PeterSperlC#developeratAnylineStudentatTUWien(MedicalInformaCcs)GamedeveloperhobbyistEmail:peter@anyline.io

3

Agenda– Microso(Kinect

•  About•  Timeline•  Components•  SpecificaCons

–  Depthsensing•  V1

–  Codedlight–  TriangulaCon

•  V2–  TimeofFlight

–  LiveDemo

4

•  MoConsensinginputdevice•  Naturaluserinterfacethroughvoiceandgestures•  InnovaCveuse-cases

–  Gaming,HealthCare,Fitness,Medicine,VR,AR,RoboCcs,etc.

•  Sensor&SDK:ForWindowsPCandXbox–Codewri[eninC#,C++/CLIorVB.NET

5

Timeline

–  1stgeneraCon:•  2010:ReleaseforXbox360•  2011:KinectforWindowsSDK1.x•  2012:KinectforWindowsV1

–  2ndgeneraCon:•  2013:ReleaseforXboxOne•  2014:KinectforWindowsV2+SDK2.0•  2015:

–  KinectforWindowsV2productwasdisconCnued,BUT:–  WindowsuserssimplyuseKinectforXboxOnesensor+Kinectadapter

6

Components•  Colorsensor:

•  commoncamerasensortoacquireRGBimages

•  Depthcamera:•  IRemi[er+IRsensoruseinfraredlightto

acquiredepthimages•  V1:StructuredLight/CodedLight•  V2:TimeofFlight

•  Audiosensor:•  Microphonearraytocapturesoundsignal

andsoundposiCon

7

KinectSpecificaCons

Feature V1 V2

Colorstream 640x480at30FPS 1920x1280at30FPS

Depthstream 320x240 512x424

Infraredstream --- 512x424

Depthdistance 0.4m(nearmode)–4m 0.4m–4.5m

Horizontalfieldofview 57° 70°

VerCcalfieldofview 43° 60°

#definedskeletonjoints 20 25

#bodiestracked 2(+4recognized) 6

OperaCngsystem Windows7/8 Windows8

8

KinectSpecificaCons

V1 V2

9

DepthSensing(V1) 1.  IREmi[er(E)projectsaknowninfraredlightpa[ern

(“codedlight”)intothescene2.  IRSensor(S)retrievesreflectedlightfromthe

environment3.  DepthmapiscomputedusingtriangulaConandlight

codingtechnology(byPrimeSense)4.  DepthdataavailableviaUSBtotheAPI

10

CodedLight(alsocalledstructuredlight)

•  Emi[erprojectspseudo-randompa[ernoflight(“Speckles”)

•  Pa[ernis“coded”:–  hastobeuniqueper

posiConsoeachpointcanberecognizedinthepa[ern

11

CodedLight•  Lightisdistorted

onthesurface•  Pa[ernisunique

ineveryposiCononthescene

•  AllowstocomputedepthinformaConthroughtriangulaCon

12

TriangulaCon

•  UsedinstereovisiontoreconstructapointPin3Dspace,giventwoormoreimages

•  MulCpleimageprojecConsofthesamesceneyielddepthinformaCon

•  PosiCons,orientaCons,andopCcalproperCesoftheimagesourcesmustbeknown

13

TriangulaCon

•  Problem:– TriangulaConrequirestwoormoreimagesources,butKinecthasonlyonedepthsensor

14

TriangulaCon

•  Trick:Kinectactuallyusestwoimages:–  TheimagecapturedfromtheIRsensor–  The“invisible”hardwiredreferencepa[ern

•  TriangulaConapplicablebecausethereisadistancebetweenIRsensorandIRprojector

15

TriangulaCon

•  Simplifiedmodel:–  CamerasareparallelalongbaselineB–  Cameracentersareatthesameheight–  Camerashavesamefocallengthf

•  Goal:calculatedisparityforeachpixel

B

16

TriangulaCon

D=x–x’=(B*f)/z

z=(B*f)/(x–x’)

•  DisparityD:DifferenceinimagelocaConofthesamepointXwhenprojectedfromdifferentperspecCves

•  DisparityisinverselyproporConaltodepth

f Focallength

D Measureddisparity

B Baselengthbetweenemi[erandcamera

z DistancetoX

17

TriangulaCon

•  IRPa[erniscorrelatedagainstthereferencepa[ern•  Projectedspeckles:

–  ifdistancetothesensorissmallerorlargerthanonthereferenceplane:shi(edindirecConofthebaselinebetweenprojectorandperspecCvecenteroftheIRcamera

–  x-direcConshi(saremeasuredforeachspeckle

•  Distancefromeachpixeltothesensorisretrievedfromcorrespondingdisparity

xk,yk Coordinatesofpointk

xo,yo Principalpointoffsets

σx,σy LensdistorConcorrecCons

Z0 DistancebetweenobjectandSensoronreferenceplane

Zk DistancebetweenpointKandSensorinobjectspace

17

TriangulaCon

•  IRSensorandIRCameraareepipolarrec6fiedØ  BothhavematchingFieldofViewØ  Correspondencesearchreducedto1DproblemØ  IfreflecCngsurfacegetsfartheraway,lightfromprojectorgets“larger”sinceeach

dotisaconeoflightØ  Lightexpandingrateandcamerapixelexpansionarethesame

•  Whenobjectdistancechanges,dotsappeartobeshi(edonlyinx-direcCon

•  Trackingadot’sx-coordinateyieldsthepixelsobserveddepth

18

DepthSensing(V2)

•  2ndgeneraConofKinectusesTimeofFlighttomeasuredepth

•  IndirectmeasurementofCmeittakesforlaserpulsestoasurfaceandbacktothesensor

Ø  Howisthispossible?

19

TimeofFlight •  Speedoflight=299,792,458m/s•  1GHzCPU=1*109Hz=1,000,000,000Hz->Pulseoflighttravels~29cmwithin1CPUcycle

•  IRSensor:•  divides1pixelinhalf•  Halfpixelisturnedonandoffextremelyfast

•  On:Absorbphotonsoflaserlight•  Off:Rejectphotonsoflaserlight

•  Otherhalfofthepixeldoesthesame,but180°phaseshi(ed

20

TimeofFlight •  LaserIlluminator(atthesameCme):

•  Pulselightsourceinphaseofthefirstpixel-half

•  Only“on”whenfirstpixelhalfis“on”

•  Distanceat0cm:•  Lightwillbeabsorbedinthe1sthalf,

nothingwillhappeninthesecondhalf

•  Let’smovelightsourceawayforabit:•  TimeofFlightwillslightlyincrease,photons

willreturninthe2ndhalfandberejected•  Totalamountoflightabsorbedwill

decreaseslightly,totalamountoflightrejectedwillincreaseslightly

21

TimeofFlight •  Movingsourceevenfartheraway:

•  Absorbedamountdecreasesfurther,rejectedamountincreasesfurther

•  A(erseveralmillisecondsexposure:•  Twototalamountsofphotonsabsorbedbythetwofirsthalvesarecompared

tothetwototalamountsofphotonsrejectedbythetwosecondhalves•  RaCoisimportant,notthetotalamount–surfacecanabsorblight,butit

affectsbothmeasurementsequally

•  Depthismeasuredbytheround-tripdistance

22

TimeofFlight •  Movingevenfurtheraway:

•  Photonsmightovershoot1stand2ndhalfandarriveinthenextcycle•  ->Ambiguity

•  Ambiguityeasilyresolvedbyincreasingpixel-halfCmes,butthensmalldepthchangescan’tbemeasured

•  SoluCon:•  OnelowresoluConmeasurementasesCmate•  Onehighprecisionmeasurement,usingesCmatetoeliminateambiguiCes

24

LiveDemo

25

Thankyou!

AndthanksPRIPforlendingmetheKinectforXboxOnesensorJ

Pa[ernRecogniConImageProcessingGroup@TUWien

•  Sources:–  h[ps://msdn.microso(.com/en-us/library/hh973078.aspx#Depth_Ranges–  AccuracyandResoluConofKinectDepthDataforIndoorMappingApplicaCons

h[ps://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect16.pdf–  h[p://www.slideshare.net/KouroshKhoshelham/generaCon-and-weighCng-of-3d-point-correspondences-for-improved-registraCon-of-rgbd-data–  h[ps://mirror2image.wordpress.com/2010/11/30/how-kinect-works-stereo-triangulaCon/–  h[p://people.eng.unimelb.edu.au/kkhoshelham/research_files/kinectaccuracy/isprsarchives-XXXVIII-5-W12-133-2011.pdf–  h[p://www.cs.cornell.edu/~hema/rgbd-workshop-2014/papers/Alhwarin.pdf–  h[p://www.visionrt.com/content/core-technology

•  Images:–  h[p://www.qualitymag.com/arCcles/91010--d-imaging-enters-the-machine-vision-world–  h[p://www.cnblogs.com/TracePlus/p/4138615.html–  h[ps://msdn.microso(.com/en-us/library/jj131033.aspx–  h[p://vision.ia.ac.cn/Students/gzp/kinect1.html–  h[p://2.bp.blogspot.com/-f4tcmi3LrDE/UZ0uY5f2p6I/AAAAAAAAB0Y/8FQlnBG8suA/s1600/new-kinect-internal.png

•  CodeExample:–  h[p://kinect.github.io/tutorial/