1
Context-Aware ApplicationsWorkflows, Challenges, and Opportunities
Simon GuestDistinguished Engineer
Neudesic, [email protected]
Image Attribution: http://www.flickr.com/photos/48600098314@N01/2612632787
2
Workflow for Determining Context
3
1. Determine SensorsSensors Critical to Determine Context
• Hard Sensors – GPS, Accelerometer, Gyro, Temperature, Humidity, etc.
• Soft Sensors – Calendar, Facebook Friends, Profile, Likes/Dislikes, History, etc.
• Mobile Devices – Arrays of sensors, with new ones being introduced all the time (e.g. fingerprint democratized with Android)
• Sensor Co-processors – Sensors moving to co-processors means lower power consumption (which means that sensors can be on all of the time)
• Bluetooth LE – Greatly extends the the range and types of sensors (e.g. indoor proximity and awareness)
4
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
etc.
5
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
etc.
2. Sensor AggregationWe Don’t Need *All* of the Raw Sensor Data to Determine Context
• Augmented Sensors – Algorithm that augments or abstracts existing sensors (e.g. “Shaking” using the accelerometer)
• Extended to Motion APIs – Google introducing standing, walking, cycling, driving “sensors” in Android 4.3
• Vendor Opportunities – Multiple vendors building SDKs for sensor aggregation – inc. Semusi (Male/Female “sensor” from accelerometer and gyro)
6
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
etc.
7
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
etc.
etc.
etc.
8
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
etc.
etc.
etc.
3. Send to Non-Relational Data StoreGoal is to capture aggregated sensor data quickly and efficiently
• Schema-less– No need to care about schema or structure• Fire and Forget – No need to send confirmation back to client, or
handle missed messages or retries• Scale and Partitions – Data store should be a single entry point that
automatically handles scale and partitioning
9
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
etc.
etc.
etc.
10
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
etc.
etc.
etc.
11
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
etc.
etc.
etc.
4. Determine Perceived ContextAnalyze data store to determine one or more perceived contexts
• Simple Matches– Single sensor data can often reveal simple matches (e.g. movement or in/out geo-fenced location)
• Combined Matches – Multiple parallel or serialized sensors to reveal more complex matches (e.g. movement with external GPS, combined with loss of signal and walking activity might indicate parked in underground parking structure).
• Learned Matches – Multiple sensor data used to reveal matches through patterns (e.g. driving detected at same time every day might indicate commuting, which is turn can infer place of work)
• Timing – Context often derived from “time blocks” of sensor data – e.g. what is happening now vs. what has happened over the last 30 minutes
12
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
etc.
etc.
etc.
13
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
14
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
5. Triggers on Context ChangeTriggers used to monitor change in context
• Change in Context– Move from one perceived context state to another (e.g. from in office to commuting)
• External Data Source– Triggers often rely on external data source (which can be interpreted as another soft sensor). Could include environmental data, move listings, weather forecasts, etc.
• Machine Learning– Often benefits from some kind of ML to help determine change in context across multiple sensors, especially in recommendations space
15
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
16
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Better movie
Startedcommute
House too cold
17
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Better movie
Startedcommute
House too cold
6. Invoke FunctionsCertain Triggers Used to Invoke Functions
• Take Action– Functions take action based on triggers. Invoke an action that may or may not involve some user input.
18
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Better movie
Startedcommute
House too cold
19
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Turn on heating
Make recommendation
Better movie
Startedcommute
House too cold
Adaptive application
Proactive application
Functions
20
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Turn on heating
Make recommendation
Better movie
Startedcommute
House too cold
Adaptive application
Proactive application
Functions
7. User Interaction and Feedback LoopInteract with user to notify them of any action, and/or provide feedback
• User Notification– Notify the user of the result from a function• User Interaction/Confirmation – Get confirmation from the user that
a function should really happen, or provide a recommendation.• Feedback Loop – Provide option for user to submit feedback based
on the function (like, dislike, rating, etc.)
21
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Turn on heating
Make recommendation
Better movie
Startedcommute
House too cold
Adaptive application
Proactive application
Functions
22
GPS
Hard Sensors
Accel.
Temp
Noise
Step
Profile
Soft Sensors
Schedule
History
Likes/Dis.
Social
Collections
Geo
Events
New Like
Motion
Environ.
Friend
Non-RelationalStore
PerceivedContext
At Work
Commuting
In Traffic
At Home
Watching TV
etc.
etc.
etc.
Asleep
etc.
Make determination
Query
Context Change(Triggers)
ExternalData
Flippingchannels
TimingLoop
Turn on heating
Make recommendation
Better movie
Startedcommute
House too cold
Adaptive application
Proactive application
User Interaction
Decision Making Feedback
Functions
23
Types of Context-Aware Applications
24
Types of Context-Aware ApplicationsAdaptive
• Act on behalf of the user• Try to adapt to the user’s
context• Often based using single-trigger
• Turn on heating when leave for home
• Phone to silent in meetings• Offer music library when get on
bus
• Need to be careful if building adaptive UI, especially if hiding/showing features
Proactive
• Involving the user• Often tied to a recommendation
engine• Often requires user-interaction• Often uses multiple triggers
• Recommend movie to watch when watching TV
• Display offer based on history when walking through grocery store
• UI needs to be unobtrusive, and easy to dismiss
25
Challenges When Developing Context-Aware Applications
26
ChallengesContext Mismatch
• Sometimes the perceived context is just wrong…• “You really think I want to do this now?!?!”• Multiple contexts help accuracy, and feedback mechanism is critical
Being Creepy
• Sharing everything that you know about the context/trigger can appear creepy
• “Your Mom’s birthday is tomorrow, and you’ve missed the last three years in a row. Do you want to pull into the 7-Eleven ahead? They have flowers on sale…”
• Reveal only enough to let the user deduce the same context, and feedback again is critical
Being Annoying
• Interruptions should follow normal human behaviors• “I wanted to let you know that I’ve turned the heating on” – “Not
now, I’m driving!”• Knowing the context of the recipient during the UI loop is also
important
27
Opportunities
28
Examples of Opportunities
Retail
• Opportunistic product offers while shopping
• Shopping recommendations based on current context
Real Estate
• Contextual awareness of prospective buyers searching for properties
Gaming
• Using context to enhance the gaming experience of patrons
Field Employees
• Context-aware applications that helps field-based employees become more productive
Home Automation
• Context-aware applications that interact with other systems in the home
Travel Applications
• More intelligent travel applications through context
29
References
30
References• Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile
Computing Systems and Applications • http://www.interaction-design.org/references/conferences/
proceedings_of_the_workshop_on_mobile_computing_systems_and_applications.html
• Swati A. Sonawane: Context-Aware Computing• http://www.slideshare.net/swatibaiger/context-aware-computing-
14084995• Albrecht Schmidt (2013): Context-Awareness, Context-Aware User
Interfaces, and Implicit Interaction • http://www.interaction-design.org/encyclopedia/context-
aware_computing.html