Post on 17-Apr-2018
transcript
Design of Hands-Free Interface for Object Relocation
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
GDMS Sr Engineer Mike DeMichele
User Robotic Aid Proposed design eliminates need for operator to perform
physical motion
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 2
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
2
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 3
PHYSICALLY DISABLED STATISTICS
3
• 5,596,000 people in the US, reported
some form of paralysis [1]
• 2,900,000 severely paralyzed persons in
America [1]
• Physical movement required by control
interfaces.
• Limiting factor in the system usability
and user workload.
[1] 2012 study performed by the Christopher and Donna Reeve Foundation
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 4
COMPLEXITY OF HUMAN MACHINE INTERACTION
4
Challenges of using human machine
interaction in a practical setting
• Systems requiring a huge amount
of user input and focus
• Time sensitive systems
• Systems requiring user to perform
small, precise movements
• Systems requiring frequent
operator turnover
[3] Robot FRIEND, Institute of Automation (IAT)
of University of Bremen
• Process driven by input from human user
• Relies on interface to continue data exchange
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 5
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
5
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 6
PROBLEM STATEMENT
6
The 2.9 million severely paralyzed individuals in the US require a technology
enable them to relocate objects without requiring motion. This design has the
potential to assist users in other daily activities that ordinarily rely on physical
input, such as navigating a robot to perform fetch and deliver tasks.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 7
STAKEHOLDER ANALYSIS
Stakeholder Significance Tensions
Control Technology
Companies
- Lead on current development work for
alternatives
- Potential to modify control technologies to
best fit need
- Competition between companies
- Tension with government approvers
Severely Paralyzed - Benefit from Hands-Free Control System - Learning time
- May initially distrust system
Insurance Companies - Create new policies to include HFCS - Cost of insurance
- Additional costs to insurance
companies
HHS and FDA - Under Medical Device Act, aid would require
approval
- Tension if design is not accepted
In-home Care
Organizations
- Leading care for severely paralyzed
- Potential to integrate new robotic care device
- Concerns of being replaced/ losing
market dominance
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 8
STAKEHOLDER INTERACTIONS
8
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 9
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
9
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 10
ALTERNATIVES
• Eye Tracking Software
• Head Movement (Gyroscope)
• Voice Control
• Muscle Contraction Detection
• Brain-Computer Interface
10
TRADE SPACE
[4]RoboNurse, Computers Making Decisions- Standford
[5] Accompany Care-O-Bot, Fraunhofer Institute
Methods to Perform Hands-Free
Control
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 11
SELECTING ALTERNATIVES
• EEG will be evaluated as the BCI Alternative as it is the most
researched and cost effective BCI
• Muscle Contraction Detection and Gyroscope will not be evaluated as
these alternatives require physical input or movement
Alternatives to Evaluate:
- EEG
- Eye Tracking
- Voice Commands
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 12
VOICE COMMAND SUMMARY
• Unlimited Number of Commands
• Ready to Use Technology
• Study by School of Information Technology in China set up
a wheelchair to be driven via voice command • 92% average success rate
• 2.2s average delay
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 13
ELECTROENCEPHALOGRAPHY (EEG) SUMMARY
• Non-invasive BCI for detecting brain activity
• Uses electrodes to measure ERP pulses along the scalp
• Event related potentials (ERP) - Brain’s electrical response to a sensory, cognitive, or
motor event
• Key to utilizing EEG as a control method is to detect desired ERP from other ERPs (noise)
and map to a command
• Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
13
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 14
EYE TRACKING SUMMARY
• Subject to false positives
• Calibration is key to success
• 2014 study published by Max Planck Institute for Informatics
in Germany • 40ms slower than using a mouse
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 15
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
15
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 16
GAP ANALYSIS
• The necessity of physical movement of the control interface is a highly limiting factor
is the system usability and user workload.
• For the 2,900,000 severely paralyzed persons in America, the necessity of physical
motion to operate a device interface renders the system unusable.
Win-Win - Create a hands-free control system interface which can match the
quality and cost of a traditional interface.
16
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 17
SCOPE
Before a new control interface can be selected for the robotic aid, we must examine
the maneuvers which the robot will need to perform.
This project will perform a simulation to collect and analyze the data crucial to
creating a hands-free control interface.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 18
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
18
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 19
CONOPS
A paralyzed person cannot perform basic motor tasks without
assistance. A robot, controlled by the person, can cross the room and
retrieve an item (prepackaged meal, medicine, cell phone).
19
User with HFCS YouBot by Kuka [9] Youbot, Vertically-Integrated Projects, GT
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 20
Functional Diagram
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 21
USE CASE
21
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 22 22
MISSION REQUIREMENTS
R.0 The HFCS shall be operable without tactile user input.
R.1 The HFCS shall not harm the user in any way.
R.2 The HFCS shall provide flexibility for use for a variety of functions.
R.3 The HFCS shall operate in real-time.
R.4 *The HFCS shall be capable of differentiating between [X] commands.
R.5 *The HFCS shall be capable of executing [Y] commands.
* where [X] and [Y] to come from simulation
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 23
COMMANDS FOR CONOPS
23
Loop 1: Correct direct
1. Stop motion
2. Pivot to correct direction
3. Resume forward motion
Assumption 1:
Hand contacting object will result in grasp.
(No slipping, etc)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 24
VIRTUAL ROBOTIC SIMULATOR
• V-REP as a virtual robot simulator • Physics based collisions
• Large number of ready made models
• Free for educational use
• Matlab
• TRS and RTB - binds VREP and
Matlab
• YouBot will pick up an object and
move it to another location • Limited preprogrammed movement
YouBot, a popular multi-function robot
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 25
YouBot VIRTUAL ROBOT[9]
• Platform
• Omnidirectional wheels
• Zero turn radius pivoting
• Surface for placing multiple objects
• Arm
• 5 joints for 5 DoF movement (Arm is not capable
of full pitch motion)
• Each joint has a limited range of movement
• Gripper
• Open and close
• To simplify the YouBot movement, all the joint movement can be automated according to
the x-y-z position of the gripper in order to reduce user complexity
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 26
YouBot Dimensions
Platform
• Raise distance from ground .7m
• L x W x D: .58m x .38m x 14m
• Payload 20 kg
• Max Velocity .8m/s
Arm (Scale factor x1.4)
• Total reach .917m
• Payload .7kg
Gripper (Scale factor x1.4)
• Range .007m
Work Envelope .7182 m^2
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 27
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
27
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 28
Simulation Objectives
• For object relocation be domestic robot: • How many & what turn commands are needed
• How many & what travel commands are needed
• How many & what arm commands are needed
• How many & what gripper commands are needed
• What tasks are infeasible for robot?
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 29
Simulation Requirements
• The simulation shall model the path of a robot fetching objects in a domestic setting.
• The simulation shall model the arm movement of a robot fetching objects in a domestic
setting.
• The simulation shall model the gripper of a robot fetching objects in a domestic setting.
• The simulation shall collect platform travel statistics.
• The simulation shall collect platform turn statistics.
• The simulation shall collect arm movement statistics.
• The simulation shall collect gripper statistics.
• The simulation shall define range of robot.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 30
SIMULATION OVERVIEW Fetch item - an item the virtual robot is instructed to pick up
Fetch point - the location of the fetch item, random variable
Task - a randomly generated trip from a start point, to the fetch point, picking up the fetch item and carrying it to an assigned finish point
Start location - initial (x,y) coordinates of the center of the robot’s platform, random variable
Finish location - final location of the center of the robot’s platform, random variable
Start Points (Green), End Point (Red) Fetch Points (Blue)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 31
SIMULATION STATISTICS
Statistics to Record per Task
1. Start location
2. Fetch item location
3. End location
4. Total time on task
5. Time spent on fetch trip
6. Time spent on return trip
7. Time spent picking up object (frame
between fetch trip and return trip)
8. Total distance traveled
9. # of platform rotations
10. degrees rotated for each rotation
11. # lateral arm movements
12. Arm displacement per movement
(cm)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 32
2D- Simulation Inputs
Room Map - Red marks obstacles
Start and fetch location
coordinates
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 33
2D- Simulation Modules
Name Description Output
GeneratePath.m Select start and fetch coordinates
then uses D* navigation algorithm
from Robotics Toolbox
Array of coordinates stepping through
shortest path to fetch location.
StatisticsScript.m Locates turns along shortest path.
Calculates angle of turn and
distance traveled on straight-aways
Matrix
Column 1,2 - (x,y)
Column 3 - angle of turn @ pivot point
Column 4 - distance traveled in previous
straight away
GraphicalData Displays frequency of maneuvers Bar graphs for frequency of travel distance
and angle
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 34
2D- Simulation Output for Single Run
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 35
2D- Simulation Output for 50 runs
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 36
3D Simulation
Depending on the position of the YouBot base, the arm joint positions are different even for moving
an object on the same height.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 37
Alternative to Manual Control of Arm/Gripper
• Vision sensor to track objects to be picked up to simplify the
pick up action
• Option for user would be displayed an array of detected distinct
objects and be prompted to choose the distinct object to be
picked up (via voice command)
• Used as an aid to guide the arm movement
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 38
Vision Sensor for HFCS Display
VREP Vision sensor has an adjustable resolution, in the ranges that an object is detected, it will
return a depth value, color and intensity values for every point. From the angle of the vision
sensor, the relative coordinates can be calculated.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 39
Simulation Results
-6 platform commands - turn left 18 degrees, turn left 2
degrees, turn right 18 degrees, turn right 2 degrees, start, stop
- Vision Sensor and active select algorithm
- Automated collision avoidance
- Return to user function
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 40
Simulation Impact
Completed Mission Requirements R.4 *The HFCS shall be capable of differentiating between 9 commands.
R.5 *The HFCS shall be capable of executing 9 commands.
Additional Requirement 1. Implement a vision sensor instead of manual control once the object is in “sight”
of the robot.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 41
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
41
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 42
Decision Criteria
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 43
Weighting the Attributes
Goal
Category
Weight Measures Measure Weight Total Weight
Capabilities 0.5 Number of
Commands 0.7 0.35
Time Sensitivity 0.3 0.15
TRL 0.1 TRL 1 0.1
Usability 0.2 User Population 0.75 0.15
New User Time 0.25 0.05
Performance 0.2 Accuracy 0.4 0.08
Maintainability 0.2 0.04
Reliability 0.4 0.08
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 44
Utility Calculation
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 45
SENSITIVITY ANALYSIS
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 46
Cost vs Utility for 1 Year
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 47
AGENDA
• Motivation
• Problem Statement and Stakeholders
• Technology Introduction
• Gap Analysis
• CONOPS
• Simulation
• Trade-off Analysis
• Business Case
47
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 48
● Eliminates need for 24/7 in-home caregiver
● Fill in-home caregiver shortage [4]
● Returns level of independence to physically disabled persons
VALUE
● Unit is defined as a Robotic Aid + Hands-Free Control System
● Sell units to Home Health Care Organizations
○ Open channel for improvement in future releases
○ Allow professionals to monitor ratio of robotic care vs human interaction
● Potential to form partnership
○ Leverage knowledge of in-home care agencies to further design
● Cost per Unit = $4,500
COST
49
● TOTAL 5 years $699k
● Nonrecurring $195K
● Recurring $504k
REVENUE ● Break even at 300 sales
Sell 100 per year Sell 200 for year 1 and 2, sell 100 after that
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 50
QUESTIONS?
Hand-held controller
Robotic Aid
Proposed design eliminates need for hand-held controller User
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 51
BACKUP SLIDES
51
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 52
References
[1] "Prevalence of Paralysis." Christopher & Dana Reeve Foundation. The Reeve Foundation Paralysis Resource Center, n.d. Web. 19 Nov. 2015.
[2] "Human Factors and Medical Devices." U.S. Food and Drug Administration. U.S. Food and Drug Administration, n.d. Web. 19 Nov. 2015.
[3] "Care-Providing Robot FRIEND." Institute of Automation. University of Bremen, n.d. Web. 19 Nov. 2015. [4] "Computers and Robots: Decision-Makers in an Automated World." Computers and Robots DecisionMakers in an Automated World. University of Stanford, n.d. Web. 19 Nov. 2015.
[5] "Care-O-bot 3." Fraunhofer Institute for Manufacturing Engineering and Automation. Fraunhofer Institute, n.d. Web. 19 Nov. 2015.
[6] "Brain Cells Chat, Even Without a Synapse." Science Magazine. AAAS, n.d. Web. 19 Nov. 2015.
[7]"Parts of Central Nervous System." , Control and Coordination, Science Help. Tutorvista, n.d. Web. 19 Nov. 2015.
[8] "A Closer Look at EEG." Epilepsy Society. Epilepsy Society, n.d. Web. 19 Nov. 2015. [9] "YouBot." YouBot Store. Kuka, n.d. Web. 19 Nov. 2015. [10] "Emotiv EPOC / EPOC+." Emotiv Epoc. Emotiv, n.d. Web. 19 Nov. 2015.
[11] "Quantitative EEG and Event-Related Potentials." Neuronetrix. COGNISION , n.d. Web. 19 Nov. 2015.
[12] T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.
52
MARKET VALUE
According to a 2014 New York
Times article [4], ● 1.3 million new paid caregivers will
be needed to meet demand over
the next decade.
● By 2020 the direct care workforce
(5 million people) will become the
largest occupation in the United
States, surpassing the number of
retail salespeople.
● About 75 percent of services
provided by home care agencies
are paid by Medicaid and
Medicare.
53
Alternative Scores
EEG CONTEXT
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 56
EEG TERMINOLOGY
Signals - The electrical activity that travels through a user’s
brain
Synapses - Spaces between neurons that conduct the electrical
activity in the brain
Noise - General term for unwanted and, in general, unknown
modifications that a signal may suffer during capture,
storage, transmission, processing, or conversion
Event-Related Potential - electrophysiological response to a
stimulus
56
[6] AAAS, Brain and Behavior
User command - The instruction a user gives to perform an action (user signal to EEG)
Input Command - A collection of steps the end-device will perform in order to execute a
single user command (digital signals to end-device)
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 57
BRAIN-COMPUTER INTERFACE (BCI)
Direct communication pathway between the brain and a computing device
57
BCI Type Signal
Quality
Sensor Location Cost
Commitment
Other
1. Invasive Best In Brain Matter High Risk of scar tissue
2. Partially-
Invasive
Medium Under skull High
3. Non-invasive Poor On Scalp Low
EEG is the most studied non-invasive BCI method due to its low cost and ease of use. The data
quality is improving quickly.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 58
Status of EEG as a Control Method Projects that follow a similar approach using EEG technology:
• Non-invasive BCI Interface for Device Operation
• 96-channel EEG system
• Cursor guidance commands
• User “mastery” within 5 sessions
• Patient initial success rate of 50-70% to 80-100% accuracy
• Brain Actuated Control of a Mobile Robot
• Combined with machine-learning techniques
• Six commands combined with “mental states”
• Worse than manual control by a slim factor of 1.5
https://drive.google.com/file/d/0BzLzei3_VBoAeFVCSFg0U2ljdDQ/view
*J. del R. Millán is with the IDIAP Research Institute, CH-1920 Martigny, Switzerland, and also with the Laboratory of Computational Neuroscience
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 59
PHYSIOLOGY OF THE BRAIN
Human Brain
- Comprised of
100,000,000,000 neurons
- Neurons connect via
synapses
- When thought occurs,
neurons generate spikes
of electrical activity
59
[7] Canadian Institutes of Health Research, Institutes of Neurological, Mental Health and Addiction
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 60
ELECTROENCEPHALOGRAHY (EEG)
• Offers a non-invasive method to detect electrical activity by using electrodes to
measure the electrical pulses along the scalp to ultimately graph these impulses
• Predominantly used in medical field for sleeping/brain abnormalities and
detection, stress evaluation, and prosthetics
• Event related potentials (ERP)
• Correspond to a brain response related to a sensory, cognitive or motor event
• The key to using EEG as a control method is to detect the desired ERP from
other ERPs (noise) and map it to a command
• EEG is not widely used as a control method due to poor signal quality from
being outside of the skull. Technological advances are improving quality.
60
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 61
How the EEG Hands-Free System Works
1) When a user thinks, electrical pulses fire within synapses in lobes of the brain specific to
that “thought” - user command
2) The EEG headset reads these signals as voltage and generates a graph indicating the
intensity and pattern of the electrical signals (The same stimuli or thought should produce
roughly the same EEG patterns when repeated)
3) These graphs will then be analyzed and chosen patterns will be stored in the database with
it’s corresponding thought or stimulus
4) These selected patterns will then be mapped to varying device actions through a system
that this project will create - input command
62 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
User concentrates
on a trained user
command
System filters out
noise and converts
signals to digital
format
System sends digital
signals to library to be
matched to a command
System outputs
corresponding
command to
simulator
14
Sensors
EEG raw data stream
EEG receives
signals and captures
them in the form of
graphs Raw signals are
stored in database and
inputted into the
system
[8] A Closer Look at EEG, Epilepsy Society
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 63
PATTERN LIBRARY BLOCK DIAGRAM
63
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 64
OPERATIONAL BLOCK DIAGRAM
64
65 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 66
EEG SIGNAL PROCESSING
• The EEG signal will be composed of 14 continuous and numerical data streams, one
from each of the EEG.
• General steps to preprocess the signal:
1. High pass filter to remove low frequency noise
2. Use a window function to group a variable length interval of data such as the Hann or
Hamming window function and eliminate edge artifacts
3. Fast Fourier Transform
4. Convert the imaginary part of any complex numbers into amplitude values
• Use a machine learning approach such as neural networks to build a learning model,
then continuously test in real time to measure accuracy
66
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 67
OPTIMIZING EEG Event-Related Potentials
Goal: Determine largest limiting factor in EEG Control System reliability
• Use simulation to examining the effects of different variables on
detection rate and reliability of ERPs such as: • Optimal training procedures or increasing training time
• Use of different machine learning approaches to improve detection
• Tentative input commands to be detected: • 6 directional commands for positive and negative displacement along each
of the x, y, z axes
• 1 toggle command for switching platform control vs arm control
• May need to simplify the number of different commands to reduce user
complexity
67
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 68
SIMULATION REQUIREMENTS
S.0.1 Virtual Robot shall accept input from HFCS.
S.0.1.1 Virtual Robot model shall contain a robot with 2D movement capability
S.0.1.2 Virtual Robot model shall contain a robot capable of grasping and
relocating
S.0.1.3 Virtual Robot model shall contain an object to be relocated
S.0.2 Simulation shall accept EEG ERPs from an input device
S.0.2.1 Simulation shall respond to a detected ERP in time < 2 seconds
S.0.2.2 Simulation shall map ERPs to input commands and interface with VREP
S.0.2.3 Simulation should detect and respond to 2+ simultaneous ERP
S.0.3 Virtual robot shall employ automated collision avoidance.
68
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 69
EPOC Cognitive Suite
1)Create user account
2)Train neurological signal to a basic motion
3)Repeat thought to perform signal
4)Map signal to another vehicle motion
69 Source: Emotiv EPOC
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 70
EEG Headset Alternatives
Standard EPOC
• 14 EEG channels and two references for accurate spatial resolution
• High performance wireless device
• iOS and Android compatible
EPOC+
• 9 axis-inertial motion sensors, Bluetooth capabilities, additional applications are
enabled
Raw Data Add-on
• Includes EEG firmware that allows the raw EEG data stream and marker
events in TestBench software
70
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 71
Brainwaves
Neural oscillations are rhythmic or repetitive neural activities in the central nervous system. Neural tissue generate oscillations through
neuron interactions.
A large number of neurons activating for a particular neural computation is called a neural ensemble.
Neural oscillations can be categorized into 5 main frequencies
• Beta (14-40Hz) - Waking consciousness
• Alpha (7.5-14Hz) - Conscious relaxation
• Theta (4-7.5Hz) - Sleeping and light meditation
• Delta (0.5-4Hz) – Deep sleep
• Gamma (above 40Hz) – Insight
• Only recently discovered
71
72 DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 72
http://neuronetrix.com/technology-i-36.html
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 73
Age
• A study conducted by Thomas W. Pierce et. al suggested that
age has an effect on EEG
• Results showed that electrode groupings were higher in older
adults than younger adults
• Older adults had more electrode locations that did not load
than the younger adults
73
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 74 74
[1]T. Pierce, T. Watson, J. King, S. Kelly and K. Pribram, 'Age Differences in Factor
Analysis of EEG', Brain Topography, vol. 16, no. 1, pp. 19-27, 2003.
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 75
Gender
• A study by Corsi-Cabrera et. al suggested differences in the
brain waves between male and female
• Men showed higher beta power, women showed higher alpha
power
• The alpha waves of men decreased during analytic,spatial,
and mixed processing, while women decreased significantly
only in analytic and mixed processing
75
DESIGN OF A HANDS FREE CONTROL SYSTEM FOR DEVICE MANIPULATION
Kassidy Kenney, Angelo Huan, Kimberly Harrington, Muhammad Sungkar 76
CURRENT TECHNOLOGIES UTILIZED BY PHYSICALLY
DISABLED PERSONS
76
• Prosthesis
• Wheelchair
• Voice Recognition [13] Johns Hopkins- Modular Prosthetic Limb