Gabriel: Towards
Wearable Cognitive Assistance
Kiryong Ha, Zhuo Chen, Wenlu Hu, Wolfgang Richter,
Padmanabhan Pillai, and Mahadev Satyanarayanan
Faces Daily Routine
Cognitive Decline
20,000,000+ Americans affected
Ron traumatic brain injury
Alzheimer’s disease
mild cognitive impairment survivors of stroke
One-month delay in nursing home
admissions saves $1,000,000,000+/year
10/24/2014 2
Text
Can Wearable Technology Help?
10/24/2014 3
Continuously capture, interpret, give guidance and
System Architecture
10/24/2014 4
“Please stop and check traffic”
“Barack is saying hello to you”
“Your dog wants to go out for a walk”
Why Today? Advances in 3 Independent Arenas
10/24/2014 5
Algorithms
Offloading Technology
Wearable Hardware
Challenges – Architecture
1. Crisp Interactive Response
2. Graceful Degradation of Services
3. Coarse-grain Parallelism
10/24/2014 6
C1. Crisp Interactive Response
Goal: Latency of infrastructure = tens of millisecond
10/24/2014 7
Face exists?
<700ms
Whose face?
~500ms
Voice Phrases?
~400ms
Humans are fast and sensitive
Satellite Phone?
ANNOYING!
S1. Crisp Interactive Response
Choice 1: standalone apps
Choice 2: offload to cloud
Choice 3: offload to cloudlets
10/24/2014 8
Distant Resources in Cloud
Nearby Cloudlet
Wi-Fi WAN
LAN
~74ms
<10ms
Glass Device
Exp. – Cloudlet Shortens Latency
10/24/2014 9
0
0.25
0.5
0.75
1
0 200 400 600
Response Time (ms)
Cloudlet
Cloud
Cloudlet vs. Cloud (Augmented Reality)
Offloading vs. Standalone (OCR)
Metric Standalone With Offload
Per-image speed (s) 10.49 1.28
Per-image energy(J) 12.84 1.14
Cloudlet shortens
response time
Offloading saves
latency and energy
C2. Graceful Degradation of Services
10/24/2014 10
What if offloading impossible?
Distant Resources in Cloud
Wi-Fi WAN
LAN
Situation 1: No cloudlet
Situation 2: No network
? ?
Nearby Cloudlet
Goal: still work during failures – with performance drop
Glass Device
S2. Graceful Degradation of Services
10/24/2014 11
Use fallback resources
No cloudlet – offload to cloud
No network – offload to carry-on device
Application-specific fidelity vs. Crispness & battery life
Distant Resources in Cloud
Glass Device
Nearby Cloudlet
Wi-Fi WAN
LAN
? ?
Wi-Fi or Bluetooth
On-body Offload Device
C3. Coarse-grain Parallelism
• Programming languages are different
• Runtime systems are different (different OSes, closed-
source, etc.)
10/24/2014 12
Don’t reinvent the wheel
Goal: reuse existing work, but…
Face recognition
Object detection
Activity inference
OCR ……
S3. Coarse-grain Parallelism
10/24/2014 13
VM Ensemble and PubSub Backbone
Core Algorithm
Logic
Virtual Machine
Dataset
Proxy
Config
Glass Device
S3. Coarse-grain Parallelism
10/24/2014 14
Cognitive Engine 1 e.g. Face recognition
Cognitive VMs
Video/Acc/
sensor streams
Device
Comm
UPnP
PubSub
Control VM
Cloudlet VM boundary
Sensor flows
Cognitive flows
User Guidance VM
VM Ensemble and PubSub Backbone
Cognitive Engine 2
Cognitive Engine 3
Cognitive Engine n
...
Wi-Fi
S3. Coarse-grain Parallelism
10/24/2014 15
Cognitive Engine 1 e.g. Face recognition
Cognitive VMs
Device
Comm
UPnP
PubSub Video/Acc/
sensor streams
Control VM
Cloudlet
User Guidance VM
Cognitive Engine 2
Cognitive Engine 3
Cognitive Engine n
...
Glass Device
Wi-Fi
Context Inference Wi-Fi
VM boundary
Sensor flows
Cognitive flows
VM Ensemble and PubSub Backbone
Exp. – Gabriel Overhead
10/24/2014 16
0
0.25
0.5
0.75
1
0 20 40 60 80 100
Response Time (ms)
Gabriel
Ideal
Echos every image
Dummy Cognitive Engine
No VM, echos from host
Gabriel Latency: ~50ms
Gabriel Overhead: ~3ms
10/24/2014 17
GDK Preview
Ice pack to cool down Glass
TCP Connection
Speech Guidance
Back-end Server
Prototype
10/24/2014 18
Cloudlet: 4 advanced desktop machines Running OpenStack – Virtualized Cloud Computing Platform
Back-end Server
Prototype
10/24/2014 19
Face Recognition
Object Recognition (1. MOPED 2. STF )
OCR (1. Tesseract 2. VeryPDF )
Motion Classifier
Augmented Reality
Activity Detection
Commercial Product
Based on Accelerometer
Prototype Cognitive Engines
Exp. – Full System Performance
10/24/2014 20
Gabriel architecture allows easy upgrade.
Cognitive Engines are slower
Cognitive Engine FPS Response time (ms)
Glass Life 1% 10% 50% 90% 99%
Face Recognition 4.4 196 389 659 929 1175
~1 hour
Object (MOPED) 1.6 877 962 1207 1647 2118
Object (STF) 0.4 4202 4371 4609 5055 5684
OCR (Open) 14.4 29 41 87 147 511
OCR (Comm) 2.3 394 435 522 653 1021
Motion Classifier 14.0 126 152 199 260 649
Augmented Reality 14.1 48 72 126 192 498
Exp. – Full System Performance
10/24/2014 21
Gabriel uses two-level token-based flow control
Cognitive Engines require different FPS
Cognitive Engine FPS Response time (ms)
Glass Life 1% 10% 50% 90% 99%
Face Recognition 4.4 196 389 659 929 1175
~1 hour
Object (MOPED) 1.6 877 962 1207 1647 2118
Object (STF) 0.4 4202 4371 4609 5055 5684
OCR (Open) 14.4 29 41 87 147 511
OCR (Comm) 2.3 394 435 522 653 1021
Motion Classifier 14.0 126 152 199 260 649
Augmented Reality 14.1 48 72 126 192 498
10/24/2014 22
drop
M A X ••
•
drop
drop
• •
•
Glass Device
Control VM
Cognitive VM
Cognitive VM
sensor
sensor data counts
yes no
a > b?
sum
a b
Processed Items Count
Drop Count
tokens
More in the Paper
1. Token-based flow control improves response
time a lot
2. Gabriel supports multi-VM parallelism
3. Tradeoff between fidelity reduction and crisp
user interaction
10/24/2014 23
10/24/2014 24
Conclusion & Future Work
Algorithms
Offloading Technology
Wearable Hardware
Gabriel: low-latency, flexible architecture
Speed improvement
needed
Cloudlets are helpful,
need good biz. model
Longer battery, better
thermal dissipation
10/24/2014 25
Cognitive Assistance Wearable Towards
Gabriel:
Contact: Kiryong Ha ([email protected]) Zhuo Chen([email protected])
Backup
10/24/2014 26
S3. Coarse-grain Parallelism
10/24/2014 27
Cognitive Engine 1 e.g. Face recognition
Cognitive VMs
Device
Comm
UPnP
PubSub Video/Acc/
sensor streams
Control VM
Cloudlet
User Guidance VM
Cognitive Engine 2
Cognitive Engine 3
Cognitive Engine n
...
Glass Device
Wi-Fi
Context Inference
VM boundary
Sensor flows
Cognitive flows
VM Ensemble and PubSub Backbone
10/24/2014 28
Prototype Mitigating Queueing Delay
Queues hurt latency:
Node A
Node B
Two-level token-based flow control
10/24/2014 29
Queues hurt latency:
Node A
Node B
Two-level token-based flow control
Throughput vs. Latency
= 1 0 ACK
Prototype Mitigating Queueing Delay
Exp. – Queuing Delay Mitigation
10/24/2014 30
0
10
20
30
40
0 100 200 300 400
Res
po
nse
del
ay
(s)
Request time (s)
token=1 token=2 TCP
0
200
400
600
token = 1 token = 2 tcp
nu
mb
er o
f fr
am
es
On-time Late
Dropped at Backend Dropped at Glass
Fidelity vs. Response Time
10/24/2014 31
S1. Crisp Interactive Response
10/24/2014 32
Justification of offloading
Table 1: Evolution of hardware performance
Table 2: Experiment result with OCR
Scale-Out of Cognitive Engines
• Cognitive Engine: Motion Classifier
– Slaves run in separate VMs
• Feature extraction
10/24/2014
• Classification
– A subset of classes for
each slave
• Waving
• Clapping
• …
33
Scale-Out of Cognitive Engines
# of slaves 1 2 4
Frame Rate / fps 9.8 15.9 19.0
10/24/2014 34