Human computer interaction advancement by usage of smart phones for motion tracking and remote operationJ. ANISH DEVCOLLEGE OF ENGINEERING, GUINDY, ANNA UNIVERSITY

Introduction To HCI (Human Computer Interaction)

Various means of HCI - so far? Standard systems:

Standard Mouse and Keyboard Setup Touch screen Setup

Custom systems: Gesture based interaction Custom hardware based interaction:• Nintendo Wii• Microsoft Kinect• Infrared based interaction systems◦ Hybrid systems involving Custom H/W and image processing

Introduction To HCI A deeper look into custom HCI systems and their components Essential components:

Custom hardware to facilitate user input or feedback

Introduction To HCI A deeper look into custom HCI systems and their components Essential components:

Computer Vision based software for Visual data processing

Analysis of current HCI systemsA PRECURSOR TO DEVELOPMENT OF THE SAME

Analysis of current HCI What are the common requirements in these systems?1) A device-linking network:

Facilitated by WiFi, Bluetooth or any other mean of networking Links the Custom hardware to the computer or target device

Its implications? Requirement of a preset registration of a user’s input device to the system Possible hassles of setting up an ever present network Constraints of being wired and therefore physically tethered within sometimes inconvenient operating

distances Difficulty of control of distributing control to multiple users

Analysis of current HCI What are the common requirements in these systems?2) Custom Hardware:

Devices which are built exclusively for a singular or preplanned set of generic uses

Its implications? Construction of devices may require fabrication of basic components which may prove to be expensive Hardware once built and deployed for a specific purpose or set of generic capabilities may require to be

fundamentally modified to match that of the target computer’s Computer Vision software. Constraints of shape, feature or functionality of devices due to usage of unmodifiable commercially

available components

Analysis of current HCI What are the common requirements in these systems?3) Inter reliance of software and hardware:

Computer vision software is written exclusive for each device

Its implications? Generally, software written for a target product cannot be used by another, even if only minutely

different by design. Software advancement depends on hardware specifications or component specific improvement.

Why work towards liberation from these requirements? EXPLORING THE POSSIBILITY OF ADVANTAGES

Custom Hardware

Linking Network

Co relation of H/W and S/w

Typical HCI

Custom Hardware

Linking Network

Co relation of H/W and S/w

Typical HCI

Standard Hardware

Linking Network

Unbound co relation

of H/w and S/w

Proof of Concept

HCI

IFFAT for HCIINTEGRATED FACULA FILTERING AND TRACKING FOR HUMAN COMPUTER INTERACTION

IFFAT for HCI System setup overview:

Average home machine with a built in or separate webcam

Smart or Feature phone Simple Computer Vision Softwarefeaturing OpenCV

IFFAT for HCI

Operation Overview

Pre-Operational ProceduresSYSTEM INITIALIZATION

IFFAT for HCI Pre-Operational Procedures:

1) Manual Calibration: Brightness and Contrast Improves accuracy for individual operating environments

2) Specification of Region of Interest: Rectangular extraction of visual data from stream Used to avoid light sources Adds to convenience of operation

IFFAT for HCI Pre-Operational Procedures:

3) Specification of colour codes to client: Range of colours Specified in RGB values Each colour code is assigned a computer function Colour can correspond to motion tracking or event triggers

RGB range values can be auto assigned for preset operating conditions No restrictions on number of colours

Track Mouse

Left Click

Right Click

Open

Operational ProceduresSYSTEM’S WORKING

IFFAT for HCI Operational Procedures:

1) Reception and cropping of live feed: Application of R.O.I specifications

2) Application of manual calibration Brightness specification Contrast specification cvSetCaptureProperty() function in OpenCV when possible. Or cvConvertScale() during post processing

IFFAT for HCI Operational Procedures:

3) Visual feed processing: Input frame is filtered in accordance with

RGB ranges specified for each colour that the client expects to find. cvInRangeS() is used as a filtering function The outcome is a set of n single channelled

frames, each of which are filtered with respectto the n number of colours that the clientexpects to find.

IFFAT for HCI Operational Procedures:

4) Gaussian blur: Applied to each segmented frame Reduces noise and helps improves accuracy

5) Blob Detection Connected-component labelling is performed

on these segmented frames by using the cvLabel()* cvBlob function.* "A linear-time component-labeling algorithm using contour tracing technique“ - Fu Chang, Chun-Jen Chen and Chi-Jen Lu

IFFAT for HCI Operational Procedures:

6) Blob Removal: Removal of blobs outside the maximum and minimum areas

of expected beacon size, if specified, by usage of cvFilterByArea(). Only one of the segmented frames is expected to contain a blob.

7) Blob Tracking: Tracking of blob movement is done by usage of

cvBlob function cvUpdateTracks()*

* “high level tracking” A. Senior, A. Hampapur, Y-L Tian, L. Brown, S. Pankanti "Appearance Models for Occlusion Handling". R. Bolle

IFFAT for HCI Operational Procedures:

8) Tracking or event detection: If the resultant blob’s colour corresponds to the colour for

motion tracking, the centroid of the blob is retrieved and its equivalent position on a computer screen is calculated by computing a new position in terms of the movement of the centroid against the bounding box of the webcam’s vision or the R.O.I

• Only one of the segmented frames is expected to contain a blob.

• In case of a colour corresponding to a pre-programmed event, the corresponding event is triggered by the client

Results and Performance measurements

IFFAT for HCIThe proposed approach was tested on a standard laptop machine, dubbed C1, with specification: CPUs Intel I3-3210M @ 2.5 GHz, having a built in HP TrueVision HD webcam supporting a resolution of upto 1280x800.

This machine was chosen to resemble a general home machine whose capabilities do not necessarily stand out in terms of processing power of the CPU or feature or resolution of the webcam

Test machine Specifications

Means of Performance Measurement:

1) Stress exhibited due to addition of beacon colour messages2) Latency exhibited during various experimenting scenarios 3) Latency exhibited to detect change in beacon colour4) Rate of success in different lighting conditions 5) Accuracy and ease of motion tracking.

1) Stress exhibited due to addition of beacon colour messages

IFFAT for HCI2) Latency exhibited during various

experimenting scenarios

Latencies observed were highly minimal and did not cause any hindrance to the operation of the system except with the exception of minor latency spikes seen during sudden exposure or increase in general ambient light

3) Latency exhibited to detect change in beacon colour

Multiple messages, namely, right-click, click-hold and click-drop were detected without a noticeable latency. Detection and firing of events corresponding to beacon messages were smooth and error free.

Results and efficiency measurements

4) Rate of success in different lighting conditions

Rates of success were measured by observing the number of events in which the client failed to detect the beacon. This was then compared to the total number of attempts performed in each respective lighting condition.

IFFAT for HCI

5) Accuracy and ease of motion tracking

All tests performed indicated an error free translation of cursor as long as the beacon signal was being tracked. Ease of use was initially restricted as maximum working efficiency was obtained only when the beacon was held in a position that caused it to maximally face the webcam during motion towards the ends of the webcam’s sight. Once this was learnt by the user, operation did not prove to be difficult

Results and efficiency measurements

Issues and possible fixes.LIMITS AND CONSTRAINTS OF USAGE

IFFAT for HCI

Operating environment lighting dependence?

Indoor lighting• Well lit• Directly facing light sources• Spot lights over operation areas

Issues, limits and constraints of usage

Light From computer

screen

Light reflected to

webcam Total darkness

• Paradoxically less successful.• Minimized by Anti-reflective screens

Beacon Colour

appears mixed

IFFAT for HCI

Operating style dependence? Angle towards screen in large degrees Beacon colours diminish and fail detection in

these conditions

Issues, limits and constraints of usage

45°45°

Possible ApplicationsEXTENSIONS AND FURTHER WORK

IFFAT for HCI

Applications? A Non-touch central system usable by multiple users without involving a network User interaction in situations where networks cannot be suitable deployed Alternate means of easy HCI for commercial and non commercial purposes.

Issues, limits and constraints of usage

Extensions? Can be extended to multi user compatibilty by beacon frequency recognition Auto calibration Auto colour range specification

Questions?