Navigation and Instrumentation Research Group
Mostafa ElhoushiPh.D. Candidate, Queen’s University, Kingston, Canada
Department of Electrical & Computer Engineering
PhD Thesis Defense
Advanced Motion Mode Recognition for Portable Navigation
05/02/2023
NavINST Group - Queen's University 2
• This work was done under collaboration with Trusted Positioning Inc. (TPI) - later acquired by InvenSense Inc. - and Royal Military College (RMC) of Canada through funds from Mitacs, NSERC CRD grant, and TPI.
05/02/2023
NavINST Group - Queen's University 3
Contents
• Introduction– Background– Motivation– Problem Statement– Objective– Prior Work– Contributions
• Proposed Methodology• Experimental Results• Summary
05/02/2023
NavINST Group - Queen's University 4
Introduction
Background• Navigation:
– the techniques of determining the position, velocity, and attitude of a moving body
• Portable Navigation:– the modified navigation techniques for a person
moving using a portable navigation device• Applications of Portable Navigation:
– cell phone navigation for trips– cell phone localization for advertising– dismounted soldier and first responder
localization– handheld portable surveying systems in urban
environments
05/02/2023
NavINST Group - Queen's University 5
Introduction
Portable Navigation• GNSS is very accurate BUT:
– Not available indoors and urban canyons– Power consuming
• Other absolute positioning methods are less accurate and need infrastructure
• Other relative positioning methods use sensors which are erroneous
• Solution: Sensor Fusion
WiFiRFID Gyroscopes
Accelerometers
Magnetometer
GNSS
Barometer
Cell towers
Relative MeasurementsAbsolute Measurements
05/02/2023
NavINST Group - Queen's University 6
Introduction
Motivation• Portable navigation is becoming increasingly popular• Accurate, reliable, cost effective personal positioning
is needed• There is a need for seamless outdoor/indoor portable
navigation• A variety of commercial and military applications
need such technology
05/02/2023
NavINST Group - Queen's University 7
Introduction
Problem Statement• The portable device (e.g., cell phone)
is untethered– able to move freely without constraints
within another moving platform (person or vehicle)
• The system has to be environment independent, i.e., work seamlessly outdoor/indoor/urban
• The system has to work autonomously in different modes of transit (e.g., walking, driving, train, elevator, …)
05/02/2023
NavINST Group - Queen's University 8
Introduction
Objective• Automatically detect the following
Motion Modes of a portable device on a user/platform:– for various device types– utilizing low-cost MEMS sensors– in real-time– for arbitrary Device Usage– for arbitrary orientation of the device– for an arbitrary user/platform
05/02/2023
NavINST Group - Queen's University 9
Introduction
Need for Motion Mode Recognition• Each motion mode can have its own
optimized navigation algorithm/constraints
Detect Motion Mode
Walking PDR with Walking Parameters
Running PDR with Running Parameters
Driving Driving Algorithms
Stationary Apply Zero-Update Velocity
ElevatorFix 2D Position
If Map Matching Used -> Correct 2D Position
Escalator Correct Along Track Distance/Position
… …
Motion ModeAlgorithm / Constraints /
Optimization
05/02/2023
NavINST Group - Queen's University 10
Introduction
Prior Work• Most of prior work had limited
robustness– portable navigation device tethered in
much of the work– covered limited device usages– covered limited orientations
• Most of prior work covered a small number of motion modes
• Some depended on:– GNSS signal availability → don’t work
indoors– Wi-Fi positioning → need special
infrastructure
05/02/2023
NavINST Group - Queen's University 11
Introduction
Contributions• Detected large number of motion
modes• Detected for the first time new
motion modes:
• Recognition is robust:– independent to device usage– independent to device orientation
• Implemented in real-time on consumer devices
• Only mandatory inputs are signals from self-contained low-cost sensors
05/02/2023
NavINST Group - Queen's University 12
Methodology
• Pattern recognition is used to detect motion modes.
• The figure shows the steps of the pattern recognition process.
• The next slides explain what is performed in each step in the pattern recognition process.
05/02/2023
NavINST Group - Queen's University 13
Methodology
• Data Inputs
05/02/2023
NavINST Group - Queen's University 14
Methodology
• Pre-Processing– Raw sensor readings have little meaning– Need to process them to come up with
more meaningful variables• Levelled Vertical Acceleration: • Magnitude of Levelled Horizontal Plane
Acceleration: • Compensated Norm of Angular Rotation
Components: • Vertical Velocity: differentiation of smoothed
altitude
05/02/2023
NavINST Group - Queen's University 15
• Feature Extraction – Statistical Features
• mean, median, mode, variance, standard deviation, 75th percentile, inter-quartile range, average absolute difference, and binned distribution, skewness, kurtosis
– Energy, Power, and Magnitude Features• energy, sub-band energies, sub-band energy ratios, and signal
magnitude area– Time-Domain Features
• zero-crossing rate and number of maximum peaks– Frequency-Domain Features
• absolute values of short-time Fourier transform, FOS spectral analysis, power spectral centroid, frequency domain entropy, average of continuous wavelet transform
– Other• cross-correlation between leveled vertical and horizontal
acceleration components• ratio between vertical velocity and number of peaks in levelled
vertical acceleration
Methodology
05/02/2023
NavINST Group - Queen's University 16
• Classification – Classification method used is Decision
Tree – Optimized using pruning
Methodology
Land-Based VesselBicycle
Land-Based Vessel F5>Thr5
F2>Thr2 F3>Thr3
F1>Thr1
F6>Thr6
Walking
Running
Bicycle
Walking F4>Thr4Bicycle Running
05/02/2023
NavINST Group - Queen's University 17
Methodology
• Classification (cont.)– Separate classifiers for various groups of
motion modes– Required classifier invoked based on
certain conditions
05/02/2023
NavINST Group - Queen's University 18
Methodology
• Post-Classification Refining– In the full solution, further enhancements
are used after machine learning, such as: • Majority Selection• Context Exclusion• Map Information• GNSS Velocity
05/02/2023
NavINST Group - Queen's University 19
Experimental Results
• Data Collection
05/02/2023
NavINST Group - Queen's University 20
Experimental Results
• Data Collection (cont.)
Data Collected>2300 trajectories, >225 hours
divided into:
Training Data Evaluation DataFeatures extracted from it are used to train classifier to generate classifier model
Features extracted from it are fed into generated classifier model to evaluate its performance
05/02/2023
NavINST Group - Queen's University 21
Experimental Results
• Data Collection (cont.)– Various users with various genders,
heights, weights, motion dynamics, speeds, and gaits
– Various device usages, and orientations– Different Elevators/Escalators/Moving
Walkways in different places and cities– Land-based vessels included:
• Different types of: Car, Truck, Bus, Train, Light-rail Train
• Sitting, Standing, and placing device On Platform
• Moving in busy areas, quiet neighborhoods, and highways in different cities
05/02/2023
NavINST Group - Queen's University 22
Experimental Results
• Data Collection (cont.)– Device Usages covered:
Handheld
Hand Still by Side
Pocket/Thigh
Ear
Belt Holder
Dangling
Arm Band
Chest
Leg
Wrist / Smartwatch
Backpack
Goggle/Smart glasses
Purse
Laptop Bag
Bicycle Handle
Bicycle Holder
Car Dashboard
Car Drawer
Car Box bet. Seats
Car Holder
On Seat
ü ü ü ü ü ü ü ü ü ü ü ü ü
ü ü ü ü ü ü ü ü ü ü ü ü
ü ü ü ü ü ü ü ü ü ü ü
ü ü ü ü ü ü ü ü ü ü
ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü ü
05/02/2023
NavINST Group - Queen's University 23
Experimental Results
• Results shown are of Classification only– i.e., without Post-Classification Refining
• Performance Measure is Confusion Matrices of evaluating classifiers on Evaluation Data (not Training Data)– Another evaluation was made on
trajectories after inserting GNSS outage
05/02/2023
NavINST Group - Queen's University 24
Experimental Results
Actual Motion Mode
Predicted Motion Mode
90.5% 9.5%
2.1% 97.9%
Average Recall Rate: 94.2%
Actual Motion Mode
Predicted Motion Mode
90.6% 9.4%
1.3% 98.7%
Average Recall Rate: 94.65%
GNSS-Outaged Trajectories
05/02/2023
NavINST Group - Queen's University 25
Experimental Results
Actual Motion Mode
Predicted Motion Mode
96.5% 2.2% 1.1% 0.3%
0.4% 99.4% 0.2% 0.0%
1.4% 1.9% 92.0% 4.8%
0.3% 0.0% 8.4% 91.2%
Average Recall Rate: 94.77%
Actual Motion Mode
Predicted Motion Mode
95.2% 0.7% 4.0% 0.2%
0.1% 98.3% 1.6% 0.0%
2.6% 1.3% 91.4% 4.8%
1.7% 0.1% 7.8% 90.4%
Average Recall Rate: 93.825%
GNSS-Outaged Trajectories
05/02/2023
NavINST Group - Queen's University 26
Experimental Results
Actual Motion Mode
Predicted Motion Mode
97.2% 2.8%
15.8% 84.2%
Average Recall Rate: 90.7%
Actual Motion Mode
Predicted Motion Mode
90.2% 9.8%
26.9% 73.1%
Average Recall Rate: 81.65%
05/02/2023
NavINST Group - Queen's University 27
Experimental Results
Actual Motion Mode
Predicted Motion Mode
96.2% 3.8%
5.9% 94.1%
Average Recall Rate: 95.15%
Actual Motion Mode
Predicted Motion Mode
90.2% 9.8%
21.7% 78.3%
Average Recall Rate: 84.25%
05/02/2023
NavINST Group - Queen's University 28
Experimental Results
Actual Motion Mode
Predicted Motion Mode
87.07% 12.93%
30.94% 69.06%
Average Recall Rate: 78.06%
Actual Motion Mode
Predicted Motion Mode
90.22% 9.78%
30.28% 69.72%
Average Recall Rate: 79.97%
GNSS-Outaged Trajectories
05/02/2023
NavINST Group - Queen's University 29
Summary
• Conclusion– Robust detection of wide range of motion modes
is possible with self-contained low-cost sensors• GNSS unavailability had minor – if not positive - effect
– Decision trees best choice for real-time:• high accuracy and low computation requirements
– Solution commercialized and implemented in consumer devices in the market
• Future Work– Improve detection of:
– Add new motion modes• E.g., Crawling, Airplane, Marine-Based Vessel
05/02/2023
NavINST Group - Queen's University 30
Publications
Patents:[1] Inventors: M. Elhoushi, J. Georgy, and A. Noureldin, Assignee: Invensense Inc., “Method and System for Estimating Multiple Modes of Motion”, U.S. Serial No. 14/528,868, filing date: 30 October 2014.Accepted Journal Publications:[2] M. Elhoushi, J. Georgy, A. Noureldin, and M. Korenberg, “Motion Mode Recognition for Indoor Pedestrian Navigation using Portable Devices,” in IEEE Transactions on Instrumentation and Measurement
Published Conference Publications:[3] M. Elhoushi, J. Georgy, A. Wahdan, M. Korenberg, and A. Noureldin, “Using Portable Device Sensors to Recognize Height Changing Modes of Motion,” in 2014 IEEE International Instrumentation and Measurement Technology Conference (I2MTC), 2014, pp. 477 – 481.[4] M. Elhoushi, J. Georgy, M. Korenberg, and A. Noureldin, “Robust Motion Mode Recognition for Portable Navigation Independent on Device Usage,” in 2014 IEEE/ION Position, Location and Navigation Symposium - PLANS, 2014, pp. 158–163.[5] M. Elhoushi, J. Georgy, M. Korenberg, and A. Noureldin, “Broad Motion Mode Recognition for Portable Navigation,” in Proceedings of the 27th International Technical Meeting of The Satellite Division of the Institute of Navigation (ION GNSS+ 2014), 2014, pp. 1768–1773.Submitted Journal Publications:[6] M. Elhoushi, J. Georgy, A. Noureldin, and M. Korenberg, “Online Motion Mode Recognition for Portable Navigation using Low-Cost Sensors,” in NAVIGATION[7] M. Elhoushi, J. Georgy, A. Noureldin, and M. Korenberg, “A Survey on Approaches of Motion Mode Recognition Using Sensors,” in IEEE Transactions on Intelligent Transportation Systems
05/02/2023
NavINST Group - Queen's University 31