Computer/Human Interaction Spring 2013
Northeastern University 1
HelpMeSpeak!
An email access system for dysarthric speakers.
Ganesh Arumugam CCIS MS 2014Stephen Flaherty PHI-CCIS PhD candidateMansoor Pervaiz PHI-CCIS PhD candidateZhichun Ye CCIS-MS 2014
Computer/Human Interaction Spring 2013
Northeastern University 2
The problem• People with various Neurological Speech
Motor disorders have impairments that limit their ability to communicate.
Computer/Human Interaction Spring 2013
Northeastern University 3
The problem• These people suffer from Dysarthria - a
motor speech disorder resulting from neurological injury that makes if difficult to pronounce words. (wikipedia)
Computer/Human Interaction Spring 2013
Northeastern University 4
The problem• Coupled with their
inability to use physical input devices, dysarthric speakers are not able to use standard computer interfaces and can struggle with assistive devices (AACs)(Best &Butler 2012).
Computer/Human Interaction Spring 2013
Northeastern University 5
Our user population
• Adult residents of The Boston Home-a residence and care center for adults with progressive neurological diseases.
www.thebostonhome.org
Computer/Human Interaction Spring 2013
Northeastern University 6
Motivation• Residents at Boston Home find it difficult
to communicate with their– caregivers (in one to one communication)– family members (through email and skype)
Computer/Human Interaction Spring 2013
Northeastern University 7
Motivation• Currently available systems are
cumbersome and difficult for people with speech and cognitive impairment to use (Hux, et al. 2000).
• Enable dysarthric speakers to create, send and view emails using only voice commands.
Computer/Human Interaction Spring 2013
Northeastern University 8
Motivation• Provide a customizable system to
recognize their individual vocal characteristics and fit their needs.
• Greatly reduce the time and effort needed for our target users to communicate with their friends and family.
Computer/Human Interaction Spring 2013
Northeastern University 9
Surprises
• In our ethnographic studies, we discovered that our target user population was very interested in using computers to interact with friends and family or surf the web.
Computer/Human Interaction Spring 2013
Northeastern University 10
Surprises
• Our target users do not want devices that speak for them.
• They want to continue to use their own voice as long as they can.
Computer/Human Interaction Spring 2013
Northeastern University 11
The best today• AAC devices do not fit our users needs
well.– Require physical touch or eye tracking– Slow– Expensive
Dynavox Maestro, www.dynavoxtech.com
Computer/Human Interaction Spring 2013
Northeastern University 12
Standard Speech Recognition• Dragon Naturally Speaking (www.nuance.com)
• Windows Speech Recognition
Computer/Human Interaction Spring 2013
Northeastern University 13
Standard Speech Recognition• Windows Speech Recognition
(www.windows.microsoft.com)
– Difficult to set up-requires physical interaction– Tedious and tiring to use
Computer/Human Interaction Spring 2013
Northeastern University 14
Standard Speech Recognition• Grid Systems
– Slow and awkward to use.
www.windows.microsoft.com
Computer/Human Interaction Spring 2013
Northeastern University 15
Innovation• We propose a very simple email system
that: – Uses pattern recognition instead to regular
speech recognition. – Sends an attached voice message instead of
text in the email.
Computer/Human Interaction Spring 2013
Northeastern University 16
Innovation• It was taking our target population at
least 40 minutes to write a short email.• Our users were keen to use their own
voice and they are understandable to their family and friends.
Computer/Human Interaction Spring 2013
Northeastern University 17
Demo
• HelpMeSpeak http://www.ccs.neu.edu/home/zhichun/team/T8/Home.html
Computer/Human Interaction Spring 2013
Northeastern University 18
User Testing Video
Computer/Human Interaction Spring 2013
Northeastern University 19
A UI challenge • Original design was only a few screens
that provided access to many functions, the idea to reduce navigation and put all the commands in front of the user.
Computer/Human Interaction Spring 2013
Northeastern University 20
A UI challenge • However, during prototyping the
abundance of options proved confusion to our users.
• Users only wanted the most critical functions.
Computer/Human Interaction Spring 2013
Northeastern University 21
A UI challenge • We had to scale back on features and
increase the number of different pages in the application.
• The result was a much sparser layout with larger buttons.
Computer/Human Interaction Spring 2013
Northeastern University 22
Tricky programming problem • Provide a popup message when the user
gave a command that was not recognized by the system.
• This proved difficult to implement – Expect the system to be always on in the
subject’s room – Could trigger off of any conversations in the
room.
Computer/Human Interaction Spring 2013
Northeastern University 23
Tricky programming problem • Require ability to distinguish specific
commands from continuous speech and take the correct action.
Computer/Human Interaction Spring 2013
Northeastern University 24
Tricky programming problem • Our system is based on pattern
recognition of recorded commands for each user, not live speech recognition and interpretation.
Computer/Human Interaction Spring 2013
Northeastern University 25
Evaluation method• Paper prototype testing with 3 subjects
at Boston Home.• Medium fidelity prototype testing at
Boston Home• Methods
– Standard performance tasks– Observation– Interviews
Computer/Human Interaction Spring 2013
Northeastern University 26
Redesign and evaluation• Initial application scope scaled down
based on user feedback and prototyping.
• Limited visual cognitive ability of user a huge factor.
• User testing with 3 subjects at Boston Home
Computer/Human Interaction Spring 2013
Northeastern University 27
Redesign and evaluation• Minimalist design with clear
functionality.• Reduce cognitive load on user• Seek high visibility of features.• Audio cues• Feedback from Team 6 helped identify
some consistency issues and broken components.
Computer/Human Interaction Spring 2013
Northeastern University 28
The future• Design limitations
– Minimal design look and feel could be improved with more development time.
– Functionality and features can be refined with more user testing.
– Improve user voice control options for recording and navigation
– User access was our biggest challenge
Computer/Human Interaction Spring 2013
Northeastern University 29
Our future process• Spend more time field testing
prototypes with our user group.• Understand that the system is based
around what they want and not what we want.
Computer/Human Interaction Spring 2013
Northeastern University 30
Our future process• Implement a more functional prototype
over the summer• If successful prepare a CHI paper for fall
submission
Computer/Human Interaction Spring 2013
Northeastern University 31
Acknowledgements• Images:
– www.nuance.com– www.microsoft.com
• The Boston Home www.thebostonhome.org
• Thanks to Team 6 for design feedback!• Coding resources:
– www.w3schools.com
Computer/Human Interaction Spring 2013
Northeastern University 32
References• Best, K., & Butler, S. (2012). Disability and Communication: A
Consideration of Cross-disability Communication and Technology. Disability Studies Quarterly,32(4).
• Hux, K., Rankin-Erickson, J., Manasse, N., & Lauritzen, E. (2000). Accuracy of three speech recognition systems: Case study of dysarthric speech.Augmentative and Alternative Communication, 16(3), 186-196.
Computer/Human Interaction Spring 2013
Northeastern University 33
HelpMeSpeak!• Web-based email system for dysarthric
speakers• Simple design for physically and visually
impaired users• Customizable for the user’s voice and
other needs