Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones Dr. Ramviyas Parasuraman Motivation Challenges Previous work Objectives Proposed solution Experiments Demo Discussion Questions References Learning On-Air Hand Gestures From Wi-Fi Signals on Smartphones Deep Learning Seminar Dr. Ramviyas Parasuraman Assistant Professor, UGA Computer Science Work: Master’s thesis (2017) of Mohamed Haseeb at KTH Sweden Slides courtesy: Mohamed from his thesis presentation. November 8, 2019 1 / 77
Transcript
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Learning On-Air Hand Gestures From Wi-FiSignals on Smartphones
Deep Learning SeminarDr. Ramviyas Parasuraman
Assistant Professor, UGA Computer ScienceWork: Master’s thesis (2017) of Mohamed Haseeb at KTH Sweden
Slides courtesy: Mohamed from his thesis presentation.
November 8, 2019
1 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Contents
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
2 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Smartphones are becoming more and more essential tohumans.
I As of 2016, 3.9 billion smartphone subscriptions(expected to reach 6.8 billion in 2022) out of 7.5 billionmobile phone subscriptions in the world [1].
I Yet, interaction with smartphones is largely bound totheir screens (limited by screen size, battery power andcomputation capability).
3 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Smartphones are becoming more and more essential tohumans.
I As of 2016, 3.9 billion smartphone subscriptions(expected to reach 6.8 billion in 2022) out of 7.5 billionmobile phone subscriptions in the world [1].
I Yet, interaction with smartphones is largely bound totheir screens (limited by screen size, battery power andcomputation capability).
3 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:
I Camera based systems [2]. Limited camera field ofview, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
The problem
I Quest for intuitive ways to interact with smartphones;examples: speech recognition, gesture recognition.
I Based on the sensing mechanism, gesture recognitionsystems can be grouped into:I Camera based systems [2]. Limited camera field of
view, sensitive to lighting conditions and consume highpower.
I Inertia based systems [3]. Sensors (e.g.accelerometers, gyroscopes) have to be carried by users.
I Radio Frequency (RF) based systems.
4 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Some approaches tries to introduce new HW into thesmartphones:
I Google’s Soli project [4].
I Specialized gesture recognition radar chip.
Other approaches leverage the existing phone capabilities:
I Sense activity and gestures using FM,GSM/WCDM/LTE or Wi-Fi signals [5], [6], [7] and [8].
5 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Some approaches tries to introduce new HW into thesmartphones:
I Google’s Soli project [4].
I Specialized gesture recognition radar chip.
Other approaches leverage the existing phone capabilities:
I Sense activity and gestures using FM,GSM/WCDM/LTE or Wi-Fi signals [5], [6], [7] and [8].
5 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
RF based gesture recognition
Advantages:
I Require no line of sight between the gesture subject andthe smartphone.
I Consume less power.
I Ubiquitous.
6 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Radio wave propagation
Static or moving objects (e.g. a human hand) impact thesignal power at the receiving end.
I Hand gestures and other background activities (e.g.walking) have closely similar impacts on the Wi-Fisignal.
I Wi-Fi RSSI stream is bursty (occurs in short non regularepisodes).
29 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Challenges
I Hand gestures and other background activities (e.g.walking) have closely similar impacts on the Wi-Fisignal.
I Wi-Fi RSSI stream is bursty (occurs in short non regularepisodes).
29 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI stream is bursty
Wi-Fi frames received by a smartphone while browsingFacebook.
30 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Wi-Fi RSSI stream is bursty
Wi-Fi frames received by a smartphone while playing aYoutube video.
31 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
32 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Previous work
I Wi-Fi RSSI was used to recognize activities (e.g.walking) on smartphones [9].
I It was also used to recognize moving hand gestures onsmartphones [8], [10] and [11].
I But, to gain access to enough RSSI samples:I The Wi-Fi interface has to operate on the monitor
mode (which prevents other applications from usingthe Wi-Fi interface).
I A rooted Android OS was needed, to install a specialWi-Fi firmware.
I Supported by a limited subset of the Wi-Fi devices.
33 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Objectives
34 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Project objectives
Demonstrate the possibility to recognize dynamic handgestures on smartphones from the Wi-Fi RSSI stream,without modification, in a passive online setting.
I dynamic: involves hand movement.
I passive: leverages existing Wi-Fi sources.
I online: in realtime on the smartphone.
I without modification: without requiring additionalHW, or core SW modification.
35 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
36 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:
I Suitable for sequential inputs (e.g. audio and videosignals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Proposed solution
Core ideas:
I Induce Wi-Fi traffic between the AP and thesmartphone to make enough RSSI measurements.
I Use an LSTM RNN model:I Suitable for sequential inputs (e.g. audio and video
signals).
I Suitable preprocessing of the input Wi-Fi RSSI stream.
37 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
38 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
39 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Traffic induction
40 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
LSTM RNN model
I N = 200 neurons/LSTM cell
I T = 50 (RNN time steps)
41 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Recognition system diagram
42 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Experiments
43 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Performed hand gestures:
44 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Swipe samples collection session:
45 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Swipe gestures:
46 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Push gestures:
47 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Sample Pull gestures:
48 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
Spatial setup:
49 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Collected dataset
I Summary of the collected dataset:
Dataset 1 2 3 4
Location room room room two roomsInduction
√ √ √
Internet√
Size 440 432 434 337
I An Android app was developed to collect the dataset.
50 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Training and Evaluation of the LSTM RNN model wasconducted on a Laptop (hence offline), using the collecteddataset.
51 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Traffic induction impact on prediction accuracy.
Dataset 1 2 3 4
Location room room room two roomsInduction
√ √ √
Internet√
LSTM accuracy 91% 83% 78% 87%
52 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Swipe and Push gestures are hardly distinguishable wheninduction is OFF.
Left: Swipe gesture with induction ON. Middle and Right:Swipe and Push gestures respectively performed while notraffic is OFF.
53 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracy as a function of the number of samplesper prediction window.
10 20 30 40 50 80 100
85
888889
91
89
83
samples per window
accu
racy
54 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracy as a function of the number of thehidden (LSTM) layers.
1 2 3
9192
90
number of layers
accu
racy
55 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
The LSTM RNN model accuracy when trained with fractionsof (Dataset1 + Dataset2 + Dataset4).
0.1 0.25 0.5 1
83
88
9294
dataset fraction
accu
racy
56 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
Evaluating the full solution implementation (an Androidapp) on a smartphone.
57 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
Spatial setup
58 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
I Accuracy of Line-of-sight (LOS) experiments is ∼81%.
I Accuracy of no Line-of-sight (No LOS) experiments is∼74%.
I The Overall accuracy is ∼78%.
59 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Online experiments
When no hand gesture is performed over a period of thirtyminutes, the false positivie rate was ∼8%.
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Conclusion
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Conclusion
I We demonstrated its possible to predict hand gestureson unmodified smartphones from Wi-Fi RSSI.
I The recognition accuracy can be improved by collectingmore data, and increasing the model size.
I The recognition accuracy can be improved by samplingRSSI at higher frequency.
63 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Limitations
I Vulnerability to interference from background activities.
I High CPU usage (25%).
64 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Limitations
I Vulnerability to interference from background activities.
I High CPU usage (25%).
64 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Publication
Published in IEEE Sensors (2019).Citation: Haseeb MA, Parasuraman R. Wisture:Touch-Less Hand Gesture Classification in UnmodifiedSmartphones Using Wi-Fi Signals. IEEE SensorsJournal. 2018 Oct 16;19(1):257-67.
65 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Publication
We open-sourced most of the codes and dataset (datacollection and Wisture recognition).https://github.com/mohaseeb/wisturehttps://www.ieee-dataport.org/documents/wi-fi-signal-strength-measurements-smartphone-various-hand-gestures
hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise
and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Introducing a preamble gesture detection mode:
I Preamble gesture needs to be: hard to confuse withnoise and require small power to detect.
I Push gesture is a candidate; easy to recognize withoutinduction.
Benefits:
I Increased robustness against interference.
I Reduced power consumption.
67 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Future work
Native support on Wi-Fi devices for a cheap high frequencysampling of Wi-Fi RSS.
I By inducing traffic at the Wi-Fi device level, the OS isbypassed, which results in a higher throughput at areduced power level.
I Reliable recognition capability at lower cost, comparedto, for example, introducing a completely new HW likeGoogle’s Soli [4].
68 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Questions
69 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References I
Ericsson mobility report.https://www.ericsson.com/en/mobility-report/latest-mobile-statistics.
[Online; accessed 22-Oct-2019].
Jie Song, Gabor Soros, Fabrizio Pece, Sean RyanFanello, Shahram Izadi, Cem Keskin, and OtmarHilliges.In-air gestures around unmodified mobile devices.In Proceedings of the 27th Annual ACM Symposium onUser Interface Software and Technology, UIST ’14,pages 319–329, New York, NY, USA, 2014. ACM.
70 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References II
Gabe Cohn, Daniel Morris, Shwetak Patel, and DesneyTan.Humantenna: Using the body as an antenna forreal-time whole-body interaction.In Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems, CHI ’12, pages1901–1910, New York, NY, USA, 2012. ACM.
Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi,Shwetak Patel, and Matthew S. Reynolds.Sideswipe: Detecting in-air gestures around mobiledevices using actual gsm signal.In Proceedings of the 27th Annual ACM Symposium onUser Interface Software and Technology, UIST ’14,pages 527–534, New York, NY, USA, 2014. ACM.
Stephan Sigg, Shuyu Shi, Felix Buesching, Yusheng Ji,and Lars Wolf.Leveraging rf-channel fluctuation for activity recognition:Active and passive systems, continuous and rssi-basedsignal features.In Proceedings of International Conference on Advancesin Mobile Computing & Multimedia, MoMM ’13,pages 43:43–43:52, New York, NY, USA, 2013. ACM.
S. Sigg, U. Blanke, and G. Troster.The telepathic phone: Frictionless activity recognitionfrom wifi-rssi.In Pervasive Computing and Communications (PerCom),2014 IEEE International Conference on, pages 148–155,2014.
73 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References V
Stephan Sigg, Mario Hock, Markus Scholz, GerhardTroster, Lars Wolf, Yusheng Ji, and Michael Beigl.Mobile and Ubiquitous Systems: Computing,Networking, and Services: 10th InternationalConference, MOBIQUITOUS 2013, Tokyo, Japan,December 2-4, 2013, Revised Selected Papers, chapterPassive, Device-Free Recognition on Your Mobile Phone:Tools, Features and a Case Study, pages 435–446.Springer International Publishing, Cham, 2014.
Christoph Rauterberg, Mathias Velten, Stephan Sigg,and Xiaoming Fu.Simply use the force - implementation of rf-basedgesture interaction on an android phone.In IEEE/KuVS NetSys 2015 adjunct proceedings, 2015.
74 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
References VI
C. Rauterberg and X. Fu.Demo abstract: Use the force, luke: Implementation ofrf-based gesture interaction on an android phone.In Pervasive Computing and Communication Workshops(PerCom Workshops), 2015 IEEE InternationalConference on, pages 190–192, March 2015.
Anthony Bagnall, Aaron Bostrom, James Large, andJason Lines.The great time series classification bake off: Anexperimental evaluation of recently proposed algorithms.extended version.CoRR, abs/1602.01711, 2016.
75 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Supporting slides
76 / 77
Learning On-AirHand GesturesFrom Wi-FiSignals on
Smartphones
Dr. RamviyasParasuraman
Motivation
Challenges
Previous work
Objectives
Proposed solution
Experiments
Demo
Discussion
Questions
References
Offline experiments
Prediction accuracies, training and prediction times fordifferent algorithms evaluated using Dataset1.
![Massachusetts Institute of Technology5 Mobility is HUGE… • 3.3 Billion mobile phone subscriptions (11/29/07) [Informa] – 1.3 Billion internet users (900 Million PC users) •](https://static.documents.pub/doc/80x56/6022051c1d14370b9e744cab/massachusetts-institute-of-technology-5-mobility-is-huge-a-33-billion-mobile.jpg)
