Object Recognition-based Mnemonics Mobile App for Senior Adults

Communication

Centre of Social Media Innovations for Communities (COSMIC)Nanyang Technological University, Singapore

Agenda

• Introduction• Related Work• UbiCuts System Requirements

– Usage Scenarios– Roles and Events

• UbiCuts System Design– Image Recognition and Data Storage Technology– System Architecture– User Interface

• Current Status of Work

Introduction of Target Domain

Background• One of the main aims of ICT is in creating and sustaining an all-

inclusive environment for co-existence of different age-groups [1]

• Smartphones revolutionized communication capabilities by integrating traditional telephony services with broadband internet services

Problem Area • There will be more than two billion senior adults by 2050 [2]

• Most of the smartphone apps are not meant for senior adults usage

Related WorkSmartphone Apps for Senior Adults Usage

• Protégé [6] -> Android app for facilating effective communuication between caregivers and seniors Features:

Fall alert, No-activity alert Medicine alert Activity report request

• eCAALYX [7] -> Healthcare-based android app for seniors Input data from patient-wearable garments with sensors and GPS (Global

Positioning System)

Specifically designed for older patients with multiple chronic problems such as diabetes mellitus and cardiovascular diseases

What is UbiCuts (Ubiquitous Shortcuts)?

Aim: To provide a simple communication solution for older adults and disabled users with motor impairments and technological illiteracy which prohibit conventional use of smartphones.

Enables a series of commands/actions (e.g. making a phone call, sending out an SMS, posting a tweet…) to be executed by just taking pictures of objects in the user's environment.

Usage of UbiCuts in its Intended Final Form

System Requirements – Usage Scenarios

Scenario 1: Twitter Update• Elderly person uses the app to indicate his current location to his family

members• Points the app at his usual seat at the local library

Scenario 2: Making Phone Call• Elderly person uses the app to make a phone call by pointing it at a photo

Scenario 3: Sending SMS• Elderly person uses the app to inform family people or caregivers for

completing some task (e.g. buy new packet of rice)• Points the app at the rice container

Events1. Sending SMS Messages

2. Making Phone Call

3. Posting updates to Twitter

System Requirements – Roles and Events

Senior Adults Family People

Caregivers

System Design• Qualcomm’s Vuforia Augmented Reality (AR) service used for

image recognition• SQLite used for data storage in smartphones

UbiCuts System Architecture

User Interface• Use-case: Create New Ubicut

User Interface

• Use-case: Scan Target Objects

User Interface• Use-case: Edit or Delete Existing Ubicuts

Design Limitations• Plain objects such as walls, bottles/containers with no stickers or

any object are currently not recorded due to lack of adequate features

• Unnecessary objects in the background when an image of the target object is captured, can potentially hamper the target scanning process

• The features present in the particular view of a target object will only be captured

• If there is a lack of natural lighting, objects will not be recognized during target scanning.

Current Status of Work An improved Android app A workable prototype in the

shape of a bracelet

User testing in progress with 20 elderly smartphone users Field study to find out older adults’ perceptions and preferences of

wearables

• Rissanen, M.J., Fernando, O.N.N., Iroshan, H., Vu, S., Pang, N. & Foo, S. (2013) Ubiquitous shortcuts: mnemonics by just taking photos. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA), Paris, France, April 27-May 2, 2013. New York, NY: ACM.

• Rissanen, M. J., Vu,  S.,  Fernando, O. N. N., Pang, N., and Foo, S. (2013).  Subtle, natural and socially acceptable interaction techniques for ringterfaces ─finger-ring shaped user interfaces. In: Proceedings of Distributed, Ambient and Pervasive Interactions (DAPI), July 21-26, 2013. Las Vegas, NV, Springer.

• Foo, S., Pang, N. and Zhang, X. (2014). UbiCuts: Wearable communication and surveillance aids for older adults and the disabled. Gerontechnology, 13(2), 196.

• Pang, N., Zhang, X., Foo, S., and Vu, S. (2014). Smartphones Use by older adults in Singapore. Gerontechnology, 13(2), 270.

• Pang, N., Vu, S., Zhang, X., & Foo, S. (2015). Older adults and the appropriation and disappropriation of smartphones. To be published in the Proceedings of the 17th International Conference on Human-Computer Interaction HCII 2015, under Theme: Human Aspects of IT for the Aged Population.

• Pang, N., Foo, S., Raamkumar, AS., Zhang, X., & Vu, S. (2015, May). UbiCUTS: Ubiquitous Shortcuts for easy communication. Demo presented at The Seventh International Conference on Information and Communication Technologies and Development (ICTD 2015), Singapore.

Related Publications

THANK YOU