;
University of Rome ”Tor Vergata”Department of Electronics Engineering
Physical analytics: Internet of Things e Data Analytics
Passive Device-Free Radio Frequency-based Human Sensing
Dr. Simone Di Domenico
ISCOM - 22/06/2018
Introduction Problem and Motivation Background RF sensing applications and experimental results
Presentation outline
• Introduction;
• Problem and Motivation;
• Background;
• RF sensing applications and experimental results:
◦ Activity recognition and people counting based on the passive use ofWiFi signals.
◦ Through-The-Wall presence detection using WiFi signals.
◦ Algorithms to reduce the need of training for activity recognition andpeople counting.
◦ People counting through the opportunistic use of LTE signals.
Dr. Simone Di Domenico 2 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Human Activity Recognition
Activity recognition is the process which aims to recognize the activityperformed by one or more users in a given area using the data collectedfrom one or multiple sensors.
Dr. Simone Di Domenico 3 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Activity Recognition Approaches
• Device-based: user wears on-body sensors and is cooperative.
• Device-free: user does not carry any sensors on his body.
◦ Active: transmitter is under control of AR system.
◦ Passive: RF signals already present in the environment are exploited.
Dr. Simone Di Domenico 4 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Passive Device-Free Sensing
Pros
3 No dedicated transmitter;
3 No concern of privacy;
3 Through the wall and NLOS sensing;
3 360◦ field of view.
Cons
7 Lower accuracy;
7 Need of dedicated training for each propagation environment.
Objectives
• Develop new models to improve recognition accuracy.
• Develop new models to reduce the need of training.
Dr. Simone Di Domenico 5 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Physical Principle
Each human activity changes the multipath propagation betweentransmitter and receiver imprinting a specific pattern on the CFR.
Dr. Simone Di Domenico 6 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Validation of the Approach through Ray-TracingSimulator (1)
Figure: Amplitude of the CIR for the ”empty” and ”1 static person” scenarios.
Dr. Simone Di Domenico 7 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Validation of the Approach through Ray-TracingSimulator (2)
Figure: CIRs overlapped for 3 consecutive instants of time for ”1 moving person”.
Dr. Simone Di Domenico 8 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Sources of data
• Signals of opportunity:◦ WiFi signal emitted by an AP already present in the environment;◦ LTE signal emitted by an eNodeB ”close” to the target area.
• Signal measurements:◦ Received Raw Signal Sample (RRSS);◦ Channel State Information (CSI);◦ Reference Signal Received Power (RSRP);◦ Received Signal Strength Indicator (RSSI).
• SDR receivers:◦ IEEE 802.11b Beacon receiver (Matlab/Simulink);◦ IEEE 802.11n Receiver (Intel NIC 5300);◦ LTE receiver (C++).
Dr. Simone Di Domenico 9 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Activity recognition and people counting based on thepassive use of WiFi signals.
Dr. Simone Di Domenico 10 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Architecture & Setup
RRSS CSI
# TX/RX 1/1 2/3Carrier [GHz] 2.4 2.4BW [MHz] 20 20Rate [pkt/s] ≈ 10 ≈ 100Window [s] 10 10# Envs 3 3
ActivitiesEmpty, Sit,
Stand, WalkEmpty, Sit,
Stand, WalkCrowd size up to 5 up to 7
Dr. Simone Di Domenico 11 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Processing
CSI RRSS
ActivityRecognition
Doppler-based approach
(i) Preprocessing(ii) Doppler spectrum estimation
over a sliding window(iii) Shape extraction(iv) Feature Selection(v) Classification
Shape-based approach
(i) Spectral estimation(ii) Shape extraction (MPEG-7)(iii) Statistical analysis over
sliding window(iv) Feature Selection(v) Classification
PeopleCounting
Doppler-based approach
(i) Preprocessing(ii) Doppler spectrum estimation
over a sliding window(iii) Shape extraction(iv) Feature Selection(v) Classification
Distance-based approach
(i) Spectral estimation(ii) Distance between spectra(iii) Statistical analysis
over a sliding window(iv) Feature Selection(v) Classification
Dr. Simone Di Domenico 12 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Scatter Plots
(a) Activity Recognition - CSI Doppler-based features. (b) Activity Recognition - RRSS shape-based features.
(c) People Counting - CSI Doppler-based features. (d) People Counting - RRSS distance-based features.
Dr. Simone Di Domenico 13 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Classification Results
CSI RRSS
ActivityRecognition
• Naıve Bayes classifier(50% train, 50% test)• Classes:{empty,sit,stand,walking}• Room A(30m2) avg acc. 77%• Room B(30m2) avg acc. 92%• Room C(45m2) avg acc. 87%
• Naıve Bayes classifier(50% train, 50% test)• Classes:{empty,sit,stand,walking}• Room A(30m2) avg acc. 93%• Room C(45m2) avg acc. 88%• Room E(200m2) avg acc. 78%
PeopleCounting
• Naıve Bayes classifier(50% train, 50% test)• Classes:{0p,1p,2p,(3,4)p,(5,6,7)p}• Room A (30m2) avg acc. 92%• Room C (45m2) avg acc. 89%• Room D (75m2) avg acc. 73%
• Naıve Bayes classifier(50% train, 50% test)• Classes:{0p,1p,2p,(3p,4p,5p)}• Room A (30m2) avg acc. 78%• Room B (45m2) avg acc. 87%
Dr. Simone Di Domenico 14 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Through-The-Wall presence detection using WiFisignals.
Dr. Simone Di Domenico 15 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Architecture & Setup
CSI
Carrier [GHz] 2.4BW [MHz] 20# TX/RX 2/3Rate [pkt/s] ≈ 100Window [s] 10# Setups 3
ActivitiesEmpty,Static,
Dynamic
Dr. Simone Di Domenico 16 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Processing
Dr. Simone Di Domenico 17 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Scatter Plot
Figure: Scatter plot of the selected features for the Setup C - Double TTW scenario.Dr. Simone Di Domenico 18 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Classification ResultsPrediction
EmptyRoom
Presence(Static)
Presence(Dynamic)
EmptyRoom
1 0 0
Tru
th
Presence(Static)
0 1 0
Presence(Dynamic)
0 0 1
1
Table: Confusion matrix for Setup A.
PredictionEmptyRoom
Presence(Static)
Presence(Dynamic)
EmptyRoom
0.98 0.02 0
Tru
th
Presence(Static)
0 0.97 0.03
Presence(Dynamic)
0 0 1
0.98
Table: Confusion matrix for Setup B.
PredictionEmptyRoom
Presence(Static)
Presence(Dynamic)
EmptyRoom
0.98 0.02 0
Tru
th
Presence(Static)
0.02 0.98 0
Presence(Dynamic)
0 0 1
0.99
Table: Confusion matrix for Setup C.
• Effectiveness of the method inrecognizing the presence of aperson in a TTW scenario.
• Detection of stationary humansis challenging also in not TTWscenarios.
• Presence detection is possibleeven in double TTW scenario.
Dr. Simone Di Domenico 19 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Algorithms to reduce the need of training for activityrecognition and people counting.
Dr. Simone Di Domenico 20 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Training options
Three different types of training options:
• Trained : recognition system is trained and tested in the sameenvironment and conditions.
• Trained-once: recognition system is trained in one environmentunder certain conditions, and is then tested in differentenvironments and/or conditions.
• Training-free: recognition system is based on anenvironment-independent model and requires lighter forms oftraining.◦ ”Empirically-modeled”◦ ”Fully-modeled”
Dr. Simone Di Domenico 21 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Key Idea
Problem: Find features that are more sensitive to the change of thepropagation environment and less sensitive to the characteristics of the
environment in static conditions.
Solution: Each person leads to a temporal variation of the CFR.
The higher the number of moving people/intensity of movement, the higher thenumber of intercepted multipath components and hence, the higher the temporal
variation of the CFR.
(i) CFR variation is correlated to the number of people/human activity.(ii) CFR variation is poorly correlated to the environment.
A trained-once model can be developed by exploiting the CFR variation.
Doppler spectrum is used to measure the CFR variation.
Dr. Simone Di Domenico 22 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Doppler Spectrum
Figure: Mean Doppler Spectrum for different activities.
Dr. Simone Di Domenico 23 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Doppler Spectrum
Figure: Mean Doppler Spectrum for different activities.
Dr. Simone Di Domenico 24 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Doppler approach
Dr. Simone Di Domenico 25 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Scatter Plots
Figure: Spectral Skewness and Kurtosis for Empty and 1 person moving scenarios fordifferent rooms.
Dr. Simone Di Domenico 26 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Trained-once - Classification Results
Training Env.Testing Env.
Room A Room B Room C
Room A - 75% 71%Room B 72% - 79%Room C 71% 88% -
Classes = {Empty-Sit, Stand, Walking}
Table: Activity Recognition performances using the Trained-once approach.
Training Env.Testing Env.
Room A Room C Room D
Room A - 72% 70%Room C 81% - 68%Room D 73% 58% -
Classes = {Empty, 1 person, (2,3,4) people, (5,6,7) people}
Table: People Counting performances using the Trained-once approach.
Dr. Simone Di Domenico 27 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
”Empirically-modeled” - Key Idea
Problem: Find ”physical features” that are independent to the environmentand are a monotone function of the degree of motion, making possible the use
of a threshold-based classifier.
Solution: Doppler spectrum tends to broaden as the intensity ofmovements/number of people increases.
Spectral spreading can be used as a metric to build the training-free model.
RMS Doppler Bandwidth and Spectral Spread measure the spectral broadeningand have been selected to build the model.
There are no mathematical models that relate these quantities to the humanactivities/numbers of people, then a calibration to find out the threshold for the
classifier is needed.
Dr. Simone Di Domenico 28 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
”Empirically-modeled”-Scatter Plots
(a) Human activity-Room A. (b) Human activity-Room B. (c) Human activity-Room C.
(d) People counting-Room A.(e) People counting-Room C.(f) People counting-Room D.
Dr. Simone Di Domenico 29 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
”Empirically-modeled”- ResultsEmpty Sit/Stand Walk
Empty 1 0 0Sit/Stand 0.37 0.63 0
Walk 0 0.02 0.980.87
Table: Human activity - Room A.
Empty Sit/Stand WalkEmpty 1 0 0
Sit/Stand 0.45 0.55 0Walk 0 0 1
0.85
Table: Human activity - Room B.
Empty Sit/Stand WalkEmpty 1 0 0
Sit/Stand 0.28 0.71 0.01Walk 0 0 1
0.72
Table: Human activity - Room C.
Empty 1 p >1 pEmpty 1 0 0
1 p 0 0.92 0.08>1 p 0 0 1
0.97
Table: People counting - Room A.
Empty 1 p >1 pEmpty 1 0 0
1 p 0 0.76 0.24>1 p 0 0 1
0.92
Table: People counting, Room C.
Empty 1p >1 pEmpty 1 0 0
1 p 0 1 0>1 p 0 0.20 0.80
0.93
Table: People counting, Room D.Dr. Simone Di Domenico 30 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
”Fully-modeled” - Key Idea
Problem: Find a mathematical model that relates one or more physicalquantities to each activity/number of people.
Solution: Such a model does not exist yet, but Doppler energy spectrum describeshow speedily scatterers move and, hence, its shape can reveal the presence of
moving scatterers.
(i) Empty scenario: Doppler energy is concentrated around 0 Hz since there areno movements.
(ii) 1 or more persons: Doppler energy is spread out over a wider range offrequencies.
Spectral Rolloff (SR) is a physical quantity which measures in Hz the Dopplerspreading and hence, can be used to build our simple model without the need of
calibration or training:
Predicted scenario =
{”Empty” if SR = 0
”1 or more persons” if SR! = 0
Dr. Simone Di Domenico 31 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
”Fully-modeled” - Scatter Plot
Figure: Spectral Rolloff for ”empty” and ”1 moving” person scenarios.
Dr. Simone Di Domenico 32 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
People counting through the opportunistic use of LTEsignals.
Dr. Simone Di Domenico 33 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Introduction & Motivation
WiFi signal has been widely used as source of opportunity fordeveloping RF activity recognition system, but its availability is not
always guaranteed.
LTE signal is an excellent candidate as a signal of opportunity thanksto its wide availability and good penetration in indoor environments.
Dr. Simone Di Domenico 34 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Key Idea
Amplitude variation of the LTE signal is strongly correlated to what happens inproximity to the transmitter and/or receiver, therefore the LTE received signal can
be used to estimate the number of people nearby the receiver.
LTE uses the Cell specific Reference Signal (CRS) to compute the CSI, which isthen used to compute the RSRP as follows:
RSRP c(n) =1
Nsub
Nsub∑j=1
|hcj(n)|2 (2)
where n is the discrete time index and c stands for the c-th CRS channel.
Since the RSRP is computed on the CSI, its variations depends only on thechanges along the propagation environment.
RSRP variations can be evaluated by dispersion-based sample moments toestimate the number of people inside the room.
Dr. Simone Di Domenico 35 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
RSRP variations
Figure: RSRP histograms for different numbers of people.Dr. Simone Di Domenico 36 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Architecture & Setup
• Very low cost SDR receiver.
• No SIM card needed.
• People were free to move.
• eNodeB with the highest SNR.
• Room size was 5m x 9m.
# Carrier [MHz] 796
# BW [MHz] 1.4
# TX/RX 2/1
Rate [RSRP/s] 2000
Window [s] 15
# Envs 1
# Setups 4
Crowd size Up to 5
Dr. Simone Di Domenico 37 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Scatter Plot
Figure: Standard deviation and Fano factor of the RSRP for different numbers of people.
Dr. Simone Di Domenico 38 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering
Introduction Problem and Motivation Background RF sensing applications and experimental results
Classification Results
PredictionEmpty 1p 2p {3,4,5}p
Empty 1 0 0 0
Tru
th 1p 0 0.38 0.05 0.572p 0 0.26 0.62 0.13
{3,4,5}p 0 0 0.27 0.730.72
Table: Confusion Matrix for Position 1.
PredictionEmpty 1p 2p {3,4,5}p
Empty 1 0 0 0
Tru
th 1p 0 0.65 0 0.352p 0 0.05 0.91 0.04
{3,4,5}p 0 0 0.08 0.920.87
Table: Confusion Matrix for Position 2.
PredictionEmpty 1p 2p {3,4,5}p
Empty 1 0 0 0
Tru
th 1p 0 0.87 0.13 02p 0 0.02 0.98 0
{3,4,5}p 0 0 0.05 0.950.95
Table: Confusion Matrix for Position 3.
PredictionEmpty 1p {2,3,4,5}p
Empty 1 0 0
Tru
th 1p 0 0.40 0.60{2,3,4,5}p 0 0.13 0.87
0.75
Table: Confusion Matrix for TTWscenario.
Achieved average accuracy ranges from 72% to 95%, including themore challenging TTW scenario.
Dr. Simone Di Domenico 39 / 39
University of Rome ”Tor Vergata” Department of Electronics Engineering