Kinect sensor

transcript

KINECT SENSOR

Group members

Bhoomit Morkar 14103180Sachin Keniya 14103170Vaishnavi Shah 14103182

Ears and Eyes of your application

Content Introduction

• Basic Construction• Basic Specifications• Kinect’s Potential and applications

Working of Kinect• Classification of Depth Measurement Techniques

Triangulation Principles of TOF Imaging Principle of structured light

• Skeletal Tracking

Applications

References

Introduction

• It was Designed by Microsoft for the Xbox 360 video game console and windows PC’s.

• While making the Kinect sensor , Microsoft’s aim was to enable interactions between users and a game without the need to touch a controller.

• Kinect is an RGB-D sensor providing synchronized color and depth images.

• It’s greatest differentials is depth sensing technology that could be extended far beyond gaming and at a much lower cost than traditional 3-D cameras.

Why Kinect is revolutionary ?

• Kinect has proven to be an incredible piece of technology.

• It is the first practical natural user interface for machines communicating with people.

• It is combining acoustical and visual sense.

• It is recognizing full body movements, even complicated ones.

• It finds application not only in games but also in the field of medical sciences and robotics.

• After selling a total of 8 million units in its first 60 days, the Kinect holds the Guinness World Record of being the "fastest selling consumer electronics device".

Basic Construction

Power Comes from the Sum

• The Kinect Sensor is the outcome of several devices(or sensors) working together.

• It is comprised of a color camera (also known as RGB), a depth sensor, a 4-microphone array, and a motorized tilt.

• The depth sensor, in itself, consists of an infrared (IR) projector and an IR camera.

• Due to that Kinect being a motion sensing input device, able to capture, track, and interpret full-body movements(3D), and recognize gestures and voices.

• The depth sensor is a monochrome complementary metaloxide semiconductor (CMOS) sensor.

• One of Kinect’s greatest differentials is its relatively low cost depth sensor.

• The depth-sensing technology is licensed from the Israeli company PrimeSense. Although the exact technology is not disclosed, it is based on the structured light principle.

• The IR projector is an IR laser that passes through a diffraction grating and turns into a set of IR dots.

• The sensor has an RGB (red-green-blue) camera for color video.

• It has four microphones that enable your application to respond to verbal input, in addition to responding to movement.

Basic Specifications

Hardware

• Depth resolution : 640x480 px • RGB resolution : 1600x1200

px• FrameRate : 60 FP

Software• Depth;

30 FPS: 80x60 , 320x240 , 640x480

• Color;12 FPS : 1280x980 RGB15 FPS : Raw YUV 640x48020 FPS : 640x480

Kinect’s Potential and applications

Depth sensor Skeletal tracking Microphone array

Background removalObject recognition

Multi-user Easy Gesture Recognition

Sound source detection Speech recognition

How it works ?

Audio input

Microphone array

Sound threshold

How does Kinect hear ?

Track me !

Speak louder .

what does it see?

Near mode depth ranges

Default mode depth ranges

Angle of vision(RGB-D)

Area under Observation

Classification of Depth Measurement Techniques

Triangulation

• In trigonometry and geometry, triangulation is the process of determining the location of a point by forming triangles to it from known points.

• Here rather than measuring distances to the point directly as in trilateration; we use of both angles and distance

Basic Mathematical model for Kinect

Principles of TOF Imaging

• Time-of-Flight (TOF) Imaging refers to the process of measuring the depth of a scene by quantifying the changes that an emitted light signal encounters when it bounces back from objects in a scene.

Pulsed ModulationContinuous Wave Modulation

Principle of structured light

• Project a known light pattern into the 3D scene, viewed by cameras.

• Distortion of light pattern allows computing the 3D structure.

• This pattern has to be unique per position in order to recognize each point in the pattern.

• Kinect uses pseudo random pattern.

speckles

Based on distance and orientation

First Region

High accurate depth.

(0.8 – 1.2 m)

Second Region

Medium accurate depth.

(1.2 – 2.0 m)

Third Region

Low accurate depth.

(2.0 – 3.0 m)

Depthobserved patternCalibration

IR camera observes the scene

known pattern

projected

3D map of the

beginning frame

New 3D map of the

Renew 3D map of the

Between projector

and camera&

x-direction speckle shift

Precision

Spatial x/y resolution

3mm @ 2m distance

Depth Z resolution

1cm @ 2m distance 10 cm @ 4m distance

operation range

0.8m ~ 4m | 0.5m ~ 3m

Skeletal Tracking

In the beginning, it was justhands;

• Head and hands,• Added shoulder and elbows.

Today!• It can track 20 joints of our body and

rans fast.

Process

Applications

• The Kinect fusion project allows a user to use an off-the-shelf Kinect camera as a 3D scanner for producing high-quality 3D scans of small or large objects.

• Researchers in China, in collaboration with Microsoft Research, developed the Kinect

Sign Language Translator, a system that can translate sign language into spoken and written language in near real time.

• The Holoflector is an augmented reality mirror. Behind the mirror is a large LCD monitor and right above the mirror is the Kinect.

• Combining a Kinect, a few hacking tools and a 3D environment, Kinectasploit allows a

user to break into a computer’s security system using body gestures. The system works almost like a video game. In one demo, a hacker ‘shoots’ at Wi-Fi targets to break into them.

• UBI Interactive developed their software UBI which combines your Windows 8 computer to Kinect and projector to create an interactive touchscreen off any surface.

• In another case of copying from the movies, many users have hacked their Kinect to allow users to control robots via body movement.

• We can pick out cloths and try them all out virtually with the help of Fitnect, an interactive dressing room.

References http://mms.ecs.soton.ac.uk/2014/papers/2.pdf

http://lshao.staff.shef.ac.uk/pub/KinectReview_TC2013.pdf

https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Microsoft20Kinect20Sensor20and20Its20Effect20-20IEEE20MM202012.pdf

http://mms.ecs.soton.ac.uk/2014/papers/2.pdf

http://campar.in.tum.de/twiki/pub/Chair/TeachingSs11Kinect/2011-DSensors_LabCourse_Kinect.pdf

http://www.slideshare.net/MatteoValoriani/introduction-to-kinect-update-v-18?qid=e70a04cc-282d-4c11-a0d0-cff7f58c2435&v=&b=&from_search=9

http://www.slideshare.net/rojizofrio/xbox-one-kinect?qid=e70a04cc-282d-4c11-a0d0-cff7f58c2435&v=&b=&from_search=15

Videos https://youtu.be/uq9SEJxZiUg

http://youtu.be/bRgEdqDiOuQ?t=10m16s

http://www.google.com/patents/US20090

http://campar.in.tum.de/twiki/pub/Cha... (p. 33)

http://en.wikipedia.org/wiki/Range_im... (Stereo triangulation)

https://www.youtube.com/watch?v=-RELJD0LBg8&spfreload=5

https://youtu.be/uq9SEJxZiUg

http://www.hongkiat.com/blog/innovative-uses-kinect/

Kinect sensor

Technology