Leveraging the Kinect SDK to Control a Remote Device

Akhil Acharya and Sean FreemermanSummer Ventures in Science and Mathematics

Appalachian State University

Computing Today

http://www.youtube.com/watch?v=FZ-ZB5OkFgE

Objective

● Original research on novel interaction techniques

● Develop new methodologies to interface the Kinect Software Development Kit with an Infrared (IR) transmitter

● Investigate the technology used by the Kinect to accurately track body parts

● Determine the viability of the Kinect platform

Tools

● Kinect○ Tracks movement

● IR Toy○ Sends IR signals

● WinLIRC software package○ "Middle man" between Kinect Application and IR toy

● 2 AirSwimmer Remote Controlled Balloons○ System scalable to any IR device

Kinect SDK Application

WinLIRC (Always Running)

Position Data

Pass command to WinLIRC

Serial information over USB

IR signals

Microsoft Kinect

● Announced: 2009● Released: 2010● Full body motion controller for Xbox 360● Scatters multiple IR beams

○ Readings at discrete points○ Saves on processing power

● Tracks depth by measuring depth of focus ● Kinect SDK allows developers to create

applications using Kinect

Why Kinect?

● Novel form of human-computer interaction● Relatively cheap

○ $150 to get started● Hands-Free● 3D Capabilities

Kinect SDK Application

WinLIRC (Always Running)

Position Data

Pass command to WinLIRC

Serial information over USB

IR signals

Application Design

● Built using example application "Skeleton-Basics WPF"

● Communicates with the Kinect

Control System (Kinect)

● Relative distance measured○ Distance between right shoulder and right hand○ User doesn't need to stand in the center of the

Kinect's FOV● Radius of 0.2 units - "null space"

○ Better differentiate commands○ Space to rest hand without performing action

SystemKinect SDK Application

WinLIRC (Always Running)

Position Data

Pass command to WinLIRC

Serial information over USB

IR signals

Control System (WinLIRC)

● If X and Y values exceed "null space" boundaries, command is sent to WinLIRC○ Done every 30 Frames (1 second)

■ Prevents WinLIRC from being overloaded○ Commands

■ Left/Right (X Values)■ Up/Down (Y Values)

● All commands defined as bytes in AirSwimmers.cfg file.

AirSwimmer

● Two models○ Shark ("Bruce")○ Clownfish ("Nemo")

● Lightweight control system○ Microcontroller with IR receiver○ Weighted ballast○ Servo○ IR remote

Challenges

● Translating information● Debugging● Documentation

Results

● It works! ● Movement occurs in near real time● Caveats:

○ Balloon requires line of sight■ Limited movement capability

○ Remotes avoid this by having higher power LEDs

Next Steps

● Higher emission IR transmitter● Full-on voice control

○ Partially implemented already● Replicate with other motion control devices

○ Second Generation Kinect (Late 2013)○ Leap Motion

● Untapped potential○ Controlling non-IR devices○ Potential to control any device

Kinect SDK Application

WinLIRC (Always Running)

Position Data

Pass command to WinLIRC

Serial information over USB

IR signals

Kinect SDK Application

WinLIRC (Always Running)

Position Data

Pass command to WinLIRC

Serial information over USB

IR signals

Kinect SDK Application

Position Data

Lessons

● Value of documentation● Powerful Kinect API● Potential of Gesture-based computing

Acknowledgements

● Mentors: Dr. Rahman Tashakkori, Mr. Luke Rice, Ms. Bahar Akhtar

● Appalachian State University● AirSwimmer and IR Toy donated by Dan

Thyer.● UNC-Charlotte and the State of North

Carolina Undergraduate Research Symposium

Thank You