Whole-Home Gesture Recognition Using Wireless Signals

——MobiCom’13

Author: Qifan Pu et al. University of WashingtonPresenter: Yanyuan Qin & Zhitong Fei

•Why interactive technique?Control music volume with gesture when showering or cooking.Turn on the thermostat with gesture while in bed.The disabled need it more.

•Options?Kinect at each room? Expensive…

Wearable Devices? Wear a device in a shower?…

Background

• Can we use ？

Can we use it to recognize gestures?

How to detect gestures using Wi-Fi?

• Wi-Fi Doppler shifts• Humans reflect Wi-Fi signals, thus can be treated as signal sources• Human motion introduce Wifi Doppler shifts

• Different gestures exhibit different patterns.

Two Challenges• Challenge 1: How to detect small Doppler shifts within

WiFi’s bandwidth (typically 20MHz)

2 / ,f fv c 5

0.5 /

_

f GHz

v m s

c light speed

17f Hz

• Challenge 2: How to determine the relationship between Doppler shifts and gestures

In WiSee the target human performs a repetitive gesture, which we use as that person’s preamble.

WiSee• WiSee is a wireless system that enable whole-home gesture reconigtion.

• Three main questions: First, how does WiSee extract Doppler shifts from conventional wireless signals like Wi-Fi?

Second, how does it map the Doppler shifts to the gestures performed by the user?

Third, how does it enable gesture recognition in the presence of other humans in the environment?

Extracting Doppler shifts from Wireless Signals

Doppler shift is the change in the observed frequency as the transmitter and the receiver move relative to each other.

• Move towards crests arrive at a faster rate

• Move from crests arrive at a slower rate

• Where c is the speed of light in the medium and f is the transmitter’s center frequency.

Extracting Doppler shifts from Wireless SignalsHow does WiSee deal with frequency offsets?

Mapping Doppler Shifts to Gestures

WiSee extracts the Doppler information by computing the frequency-time Doppler profile of the narrowband signal.

Segmentation

Gestures

WiSee leverages the stucture of the Doppler profiles. Set of segments that have positive and negative Doppler shifts.

Gestures Classification

The receiver can classify gestures by matching the pattern.

Positive “1”

Negative “-1”

Both “2”

Challenge 3: Interference from other people in the environment• WiSee uses MIMO in 802.11n to improve accuracy • But MIMO requires a known preamble• In WiSee, a repeated gesture acts as preamble to specify certain user

User pushes hand toward

and away

WiSee detect target user

WiSee iteratively finds MIMO channel that maximizes Doppler energy

Multi-path EffectsThe reflections usually arrive at the receiver along multiple paths.

A gesture towards receiver can create both positive and negative Doppler shifts at the receiver.

WiSee automatically finds a proper MIMO direction

because of iteration algorithm

Strong reflectors (metal) can flip the positive and negative Doppler shifts.

The receiver can calibrate the sign of the subsequent Doppler shifts based on preamble gestures.

Experiments

•ScenariosOffice buildingTwo-bed apt

•Many conditionsLine-of-sight, non-line-of-sight, through-the-wall,

through-the-corridor, through-the-room

TX: transmitter RX: Wisee receiver

Wall

Feasibility of Wireless Gesture Detection

Feasibility of Wireless Gesture Detection

Gesture Recognition in Two-bed apt

Gesture Recognition Accuracy

Confusion matrix

Gestures in the Presence of Other Humans (No target user, only other 12 people)

False detection rate for a whole day

Influence from the number of interfering user (Target user and other people)

Influence from the distance of interfering user

Summary

•Design and evaluate, WiSee, a gesture recognition system that leverages wireless signals to enable whole-home sensing and recognition of human gestures

•WiSee can extract a rich set of gesture information from wireless signals and enable whole-home gesture recognition using only two wireless sources placed in the living room

Disadvantages

Whole-home gesture recognition with few wireless sources. Without sensing device on human body or many devices in

environmentHigh accuracy: 94%(on average) for 9 gestures.

Advantages

Heavily influenced by the number of other people.The number of gestures are limited (9 gestures) and hard to

extend.Using scale is smaller than wearable devices.

Comments on WiSee

Future Works

•Security? •Information fusion: Add other information to make the system more robust, like sound•Explore whether the shape of body will affect the performance.