Creating Context-Aware Applications

transcript

Context-Aware ApplicationsWorkflows, Challenges, and Opportunities

Simon GuestDistinguished Engineer

Neudesic, LLCsimon.guest@neudesic.com

Image Attribution: http://www.flickr.com/photos/48600098314@N01/2612632787

Workflow for Determining Context

1. Determine SensorsSensors Critical to Determine Context

• Hard Sensors – GPS, Accelerometer, Gyro, Temperature, Humidity, etc.

• Soft Sensors – Calendar, Facebook Friends, Profile, Likes/Dislikes, History, etc.

• Mobile Devices – Arrays of sensors, with new ones being introduced all the time (e.g. fingerprint democratized with Android)

• Sensor Co-processors – Sensors moving to co-processors means lower power consumption (which means that sensors can be on all of the time)

• Bluetooth LE – Greatly extends the the range and types of sensors (e.g. indoor proximity and awareness)

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

2. Sensor AggregationWe Don’t Need *All* of the Raw Sensor Data to Determine Context

• Augmented Sensors – Algorithm that augments or abstracts existing sensors (e.g. “Shaking” using the accelerometer)

• Extended to Motion APIs – Google introducing standing, walking, cycling, driving “sensors” in Android 4.3

• Vendor Opportunities – Multiple vendors building SDKs for sensor aggregation – inc. Semusi (Male/Female “sensor” from accelerometer and gyro)

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

3. Send to Non-Relational Data StoreGoal is to capture aggregated sensor data quickly and efficiently

• Schema-less– No need to care about schema or structure• Fire and Forget – No need to send confirmation back to client, or

handle missed messages or retries• Scale and Partitions – Data store should be a single entry point that

automatically handles scale and partitioning

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

4. Determine Perceived ContextAnalyze data store to determine one or more perceived contexts

• Simple Matches– Single sensor data can often reveal simple matches (e.g. movement or in/out geo-fenced location)

• Combined Matches – Multiple parallel or serialized sensors to reveal more complex matches (e.g. movement with external GPS, combined with loss of signal and walking activity might indicate parked in underground parking structure).

• Learned Matches – Multiple sensor data used to reveal matches through patterns (e.g. driving detected at same time every day might indicate commuting, which is turn can infer place of work)

• Timing – Context often derived from “time blocks” of sensor data – e.g. what is happening now vs. what has happened over the last 30 minutes

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

5. Triggers on Context ChangeTriggers used to monitor change in context

• Change in Context– Move from one perceived context state to another (e.g. from in office to commuting)

• External Data Source– Triggers often rely on external data source (which can be interpreted as another soft sensor). Could include environmental data, move listings, weather forecasts, etc.

• Machine Learning– Often benefits from some kind of ML to help determine change in context across multiple sensors, especially in recommendations space

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

Context Change(Triggers)

ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

6. Invoke FunctionsCertain Triggers Used to Invoke Functions

• Take Action– Functions take action based on triggers. Invoke an action that may or may not involve some user input.

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Better movie

Startedcommute

House too cold

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold

Adaptive application

Proactive application

Functions

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold

Functions

7. User Interaction and Feedback LoopInteract with user to notify them of any action, and/or provide feedback

• User Notification– Notify the user of the result from a function• User Interaction/Confirmation – Get confirmation from the user that

a function should really happen, or provide a recommendation.• Feedback Loop – Provide option for user to submit feedback based

on the function (like, dislike, rating, etc.)

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold

Functions

Hard Sensors

Accel.

Profile

Soft Sensors

Schedule

History

Likes/Dis.

Social

Collections

Events

New Like

Motion

Environ.

Friend

Non-RelationalStore

PerceivedContext

At Work

Commuting

In Traffic

At Home

Watching TV

Asleep

Make determination

ExternalData

Flippingchannels

TimingLoop

Turn on heating

Make recommendation

Better movie

Startedcommute

House too cold

User Interaction

Decision Making Feedback

Functions

Types of Context-Aware Applications

Types of Context-Aware ApplicationsAdaptive

• Act on behalf of the user• Try to adapt to the user’s

context• Often based using single-trigger

• Turn on heating when leave for home

• Phone to silent in meetings• Offer music library when get on

• Need to be careful if building adaptive UI, especially if hiding/showing features

Proactive

• Involving the user• Often tied to a recommendation

engine• Often requires user-interaction• Often uses multiple triggers

• Recommend movie to watch when watching TV

• Display offer based on history when walking through grocery store

• UI needs to be unobtrusive, and easy to dismiss

Challenges When Developing Context-Aware Applications

ChallengesContext Mismatch

• Sometimes the perceived context is just wrong…• “You really think I want to do this now?!?!”• Multiple contexts help accuracy, and feedback mechanism is critical

Being Creepy

• Sharing everything that you know about the context/trigger can appear creepy

• “Your Mom’s birthday is tomorrow, and you’ve missed the last three years in a row. Do you want to pull into the 7-Eleven ahead? They have flowers on sale…”

• Reveal only enough to let the user deduce the same context, and feedback again is critical

Being Annoying

• Interruptions should follow normal human behaviors• “I wanted to let you know that I’ve turned the heating on” – “Not

now, I’m driving!”• Knowing the context of the recipient during the UI loop is also

important

Opportunities

Examples of Opportunities

Retail

• Opportunistic product offers while shopping

• Shopping recommendations based on current context

Real Estate

• Contextual awareness of prospective buyers searching for properties

Gaming

• Using context to enhance the gaming experience of patrons

Field Employees

• Context-aware applications that helps field-based employees become more productive

Home Automation

• Context-aware applications that interact with other systems in the home

Travel Applications

• More intelligent travel applications through context

References

References• Shilt, Adams, Want (1994): Proceedings of the Workshop on Mobile

Computing Systems and Applications • http://www.interaction-design.org/references/conferences/

proceedings_of_the_workshop_on_mobile_computing_systems_and_applications.html

• Swati A. Sonawane: Context-Aware Computing• http://www.slideshare.net/swatibaiger/context-aware-computing-

14084995• Albrecht Schmidt (2013): Context-Awareness, Context-Aware User

Interfaces, and Implicit Interaction • http://www.interaction-design.org/encyclopedia/context-

aware_computing.html

Thank You

Simon GuestDistinguished Engineer

Neudesic, LLC

simon.guest@neudesic.com@simonguest

Creating Context-Aware Applications

Technology