HCI Challenges in Human Movement Analysis:Touchscreen interaction for older adults
Lilian Genaro Motti Ader, PhD
University College DublinSchool of Public Health,Physiotherapy and Sports Science
EU H2020 Marie Curie Career-FIT - MF2017-0204 Co-fund No: 713654
Human Movement Analysis• Detect, identify, track human body
• Technologies: motion capture, sensors (orientation, acceleration)...• Postures: body position, body segments, joint angles...• Movements: amplitudes, ranges of motion, duration, speed...
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD 2
http://resources.mpi-inf.mpg.de/biomechanics/
Qualisys
Manus VR
Fitbit
TekScan
Applications for HCI• Evaluate ergonomics• Improve performances• Design new interaction techniques
...
3
Ader et al. 2016 Bachynskyi et al. 2015 Bossavit et al. 2014
http://resources.mpi-inf.mpg.de/biomechanics/
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Current research: Assessment of gait and balance
4
Sources.Kinesis,Otago exercises
Codamotion, UCD Gait LabPressure matShimmer IMUs
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
HCI Challenges in Human Movement Analysis
5
HCI Biomechanics
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2018
PhD Thesis “Study of the interaction of older adults with touchscreen”
IRIT Lab, University of Toulouse, France
HandiBio, University of Toulon, France
Touchscreen interaction for Older Adults
7
Usability and Acceptance of technologies
Mallenius et al. 2009, Barnard et al. 2013,Caprani et al. 2015
Fortes et al. 2015
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
8
Fisk et al. 2009, Lepicard & Vigouroux 2012, Motti et al. 2013, Findlater et al. 2015
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Objective
Understand and explain the difficulties older adults find to execute the gestures of interaction on touchscreen
930/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
New approach:Human movement analysis
10
• State of the art on movements of tactile interaction Bachynskyi et al. 2015, Irwin et al. 2012, Irwin & Sesto 2011, Kim et al. 2013, Jacquier-Bret et al. 2014, Sesto et al. 2012, Pereira et al. 2013, Shin & Zhu 2011, Werth & Babski-Reeves 2014, Young et al. 2012, Young et al. 2013
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Design of research study
=> Understand differences in performances between older and younger users of touchscreen by evaluation of their movements
1130/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
HCI
- represent diversity- users- settings
- performances- time- errors
Biomechanics
- homogeneity- morphologies- gestures
- controlled environments
- iterations
12
Design of research study
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Interactive system “Puzzle Touch”
13
• Facilitate the recruitment and create social interaction• Game, Images
• Existing guidelines + iterative design• Drag-and-drop
HTML CSS + Javascript + PhPParameters (pieces, levels)
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2016
Interactive system “Puzzle Touch”
14
• Facilitate the recruitment and create social interaction• Game, Images
• Existing guidelines + iterative design• Drag-and-drop
HTML CSS + Javascript + PhPParameters (pieces, levels)
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2016
Familiarisation task with “Puzzle Touch”
1530/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Movement and performance
• 15 older adults (65-84) vs. 15 adults (18-45)• pre-test: vision, cognition, motor, practice trials
• Smartphone vs. tablet
• Finger vs. Pen
• Targets: 9 large vs. 16 small
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD 16
HandiBio Lab, Toulon, 2014Motti Ader 2016
Equipment
Reflective markers for optical motion capture(22 + 12 + 3)
17
(ISB) Wu et al. 2005
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2016
Motion capture
1830/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2016
Joint: The wrist
• Mobilisation during tactile interaction• Index of movements of upper limbs• Ageing effects on motor control and
proprioception
19
Jacquier-Bret et al. 2014, Wright et al. 2011, ISO 9241
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Postures of the wrist
20
Flexion
Extension
Ulnar deviation
Radial deviation
α < 0
α > 0
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Detection and tracking of the wrist
21
MCP2: metacarpi 2MCP5: metacarpi 5RSP: radial styloid processusUSP: ulnar styloid processus
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Motti Ader 2016
Postures of the wrist
Technical markers on the device
Anatomical markers on the user’s hand
Ulnar deviation(α < 0)
Dorsal view of the hand
Radial deviation(α > 0)
Touchscreen(frontal view)
Postures of the wrist
Touchscreen(side view)
Technical markers on the device
Anatomical markers on the user’s hand
Flexion(α < 0)
Medio-lateral view of the handExtension
(α > 0)
Data analysis
2 devices x 2 modalities x 25 targets= 100 gestures per participant, per iteration
=> 4,500 gestures from older participants (3 iterations)
=> 7,500 gestures from younger participants (5 iterations)
Measures• Performances: time and number of errors (per target)
• Movements of the wrist: angular deviations
2430/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Results: Finger interaction on tablet
• Posture: radial deviation and extended
25
Radial deviation(93% of the time)
Extension(68% of the time)
Adults
Olderadults
0° to 36° 33°
-8° to 50° 56°
-5° to 19° 23°
-6° to 33° 39°
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
26
Ulnar => Radial deviation (Younger adults)
Ulnar => Radial deviation (Older adults)
Results: Finger interaction on tablet
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
27
Flexion => Extension angles (Younger adults)
Flexion => Extension angles (Older adults)
Results: Finger interaction on tablet
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Results: performances
28
2.7 (1.3)
Time (s)
0.2 (0.3)
4.4 (1.5) 0.4 (0.9)
Number of errors
Adults
Olderadults
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Results: performances
29
2.7 (1.3)
Time (s)
0.2 (0.3)
4.4 (1.5) 0.4 (0.9)
Number of errors
Adults
Olderadults
***
** **
** **
*** strong positive correlation ** moderate positive correlation (Spearman)
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Discussion: older users
• Study of the relationship between movements and performances
• Wrist: amplitude and radial deviation• Risk of discomfort could explain longer times and
increased number of errors• Differents strategies of movements
• Priority to mobilization of distal joints (Hsiao & Cho 2012)
• Difficulty of postural stabilization (Wright et al. 2012)
3030/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Discussion: younger adults
• Joint angles close to neutral• Increased mobilisation and dexterity of fingers• Better performances may be related to comfort of use
• Future work• Stabilisation of the wrist: compensatory movements
(elbow, shoulder, trunk)• Ergonomics guidelines
• Other joints and postures• Different settings
3130/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Discussion: limitations
• Older adults: heterogeneity ⇔ variability• Diversity among users
• Discomfort• Perceived individually, further evaluations:
• Assessment of individual joint range of motion• Physiological measures (e.g. muscle activity)
• Acceptance of experiment protocol• Duration and iterations• Accessibility• Equipment
3230/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Conclusion
• Study of differences in performances through biomechanical analysis
• Contribution: demonstrate differences in movements• Increased amplitude of movements associated to risk
of discomfort and related to lower performances for older adults
• Wrist (younger vs. older)• Further studies: compensatory movements, individual
discomfort, ergonomics guidelines• Perspectives: new approach to evaluate and design
devices and interactive systems
3330/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
• Accessibility and acceptance of the protocol• Variability among participants• Data collection (synchronisation)• Defining measures and parameters• Individual skills vs. collective measures• Laboratory settings vs. real context
34
HCI Challenges in Human Movement Analysis:Touchscreen interaction for older adults
30/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
Impact
Human movement analysis:
improving ergonomics and usability of technologies for today's and future generation of users
3530/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
References
M. Bachynskyi, G. Palmas, A. Oulasvirta, J. Steimle, and T. Weinkauf, “Performance and Ergonomics of Touch Surfaces: A Comparative Study Using Biomechanical Simulation,” in Proceedings of ACM CHI’15, 2015, pp. 1817–1826.
B. Bossavit and A. Pina, “Designing educational tools, based on body interaction, for children with special needs who present different motor skills,” Proc. - 2014 Int. Conf. Interact. Technol. Games, iTAG 2014, pp. 63–70, 2014.
B. Bossavit, A. Marzo, O. Ardaiz, L. Diaz De Cerio, and A. Pina, “Design choices and their implications for 3D Mid-Air Manipulation Techniques,” Presence, vol. 23, no. 4, pp. 377–392, 2014.
M. McCarthy et al., “The REACHES Study: Lessons Learned From Conducting Remote Assessments within a Nursing Home Environment,” no. March, 2018.
B. R. Greene, S. J. Redmond, and B. Caulfield, “Fall risk assessment through automatic combination of clinical fall risk factors and body-worn sensor data,” IEEE J. Biomed. Heal. Informatics, vol. PP, no. 99, p. 1, 2016.
E. Smith, L. Walsh, J. Doyle, B. Greene, and C. Blake, “The reliability of the quantitative timed up and go test (QTUG) measured over five consecutive days under single and dual-task conditions in community dwelling older adults,” Gait Posture, vol. 43, pp. 239–244, 2016.
3630/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
References
R. Grant and N. Birch, “Otago strength and balance training exercise programme. An information guide for patients,” pp. 1–27, 2015.
S. Mallenius, M. Rossi, and V. Tuunainen, “Factors affecting the adoption and use of mobile devices and services by elderly people–results from a pilot study,” in 6th Annual Global Mobility Roundtable, 2007.
Y. Barnard, M. D. Bradley, F. Hodgson, and A. D. Lloyd, “Learning to use new technologies by older adults: Perceived difficulties, experimentation behaviour and usability,” Comput. Human Behav., vol. 29, no. 4, pp. 1715–1724, Jul. 2013.
N. Caprani, N. O’Connor, and C. Gurrin, “Touch screens for the older user,” in Assistive Technologies, InTech, 2012, no. 1, pp. 95–118.
R. P. M. Fortes, G. A. Martins, and P. C. Castro, “A Review of Senescent’s Motivation in the Use of Tactile Devices,” Procedia Comput. Sci., vol. 67, no. Dsai, pp. 376–387, 2015.
A. D. Fisk, W. a Rogers, N. Charness, S. J. Czaja, and J. Sharit, Designing for older adults: Principles and creative human factors approaches. 2009.
3730/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
References
G. Lepicard and N. Vigouroux, “Comparison between single-touch and multi-touch interaction for older people,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2012, vol. 7382 LNCS, no. PART 1, pp. 658–665.
L. G. Motti, N. Vigouroux, and P. Gorce, “Interaction Techniques for Older Adults Using Touchscreen Devices: A Literature Review from 2000 to 2013,” J. Interact. Pers., vol. (in review, p. 125:125–125:134, 2013.
L. Findlater, J. Froehlich, K. Fattal, J. O. Wobbrock, and T. Dastyar, “Age - Related Differences in Performance with Touchscreens Compared to Traditional Mouse Input,” in Proceedings of the SIGCHI Conference CHI’13, 2013, pp. 343–346.
M. E. Sesto, C. B. Irwin, K. B. Chen, A. O. Chourasia, and D. A. Wiegmann, “Effect of touch screen button size and spacing on touch characteristics of users with and without disabilities,” Hum. Factors, vol. 54, no. 3, pp. 425–436, 2012.
C. B. Irwin, T. Y. Yen, R. H. Meyer, G. C. Vanderheiden, D. P. Kelso, and M. E. Sesto, “Use of force plate instrumentation to assess kinetic variables during touch screen use,” Univers. Access Inf. Soc., vol. 10, no. 4, pp. 453–460, Jan. 2011.
C. B. Irwin and M. E. Sesto, “Performance and touch characteristics of disabled and non-disabled participants during a reciprocal tapping task using touch screen technology,” Appl. Ergon., vol. 43, no. 6, pp. 1038–43, Nov. 2012.
3830/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
References
J. H. Kim, L. Aulck, M. C. Bartha, C. a. Harper, and P. W. Johnson, “Differences in typing forces, muscle activity, comfort, and typing performance among virtual, notebook, and desktop keyboards,” Appl. Ergon., vol. 45, no. 6, pp. 1406–1413, 2014.
J. Jacquier-Bret, P. Gorce, L. G. Motti, and N. Vigouroux, “Biomechanical analysis of interaction strategies using touchscreen: preliminary study,” in Computer Methods in Biomechanics and Biomedical Engineering, 2014, vol. Vol. Suppl, pp. 86–87.
J. Jacquier-bret, P. Gorce, G. M. Lilian, and N. Vigouroux, “Biomechanical analysis of upper limb during the use of touch screen : motion strategies identification,” Ergonomics, vol. 0139, no. April, pp. 1–8, 2017.
A. Pereira, T. Miller, Y.-M. Huang, D. Odell, and D. Rempel, “Holding A Tablet Computer With One Hand,” Proc. Hum. Factors Ergon. Soc. 57th Annu. Meet., vol. 57, no. Straker 2008, pp. 1634–1638, 2013.
L. G. Motti Ader, “Study of the interaction of older adults with touchscreen,” Universtié Toulouse 3 Paul Sabatier, 2016.
L. G. Motti Ader, N. Vigouroux, and P. Gorce, “Biomechanical analysis of the user’s movements during tactile interaction: Postures of older aged users’ wrists,” in ACM International Conference Proceeding Series, 2017, vol. Part F1350.
3930/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD
References
G. Shin and X. Zhu, “User discomfort , work posture and muscle activity while using a touchscreen in a desktop PC setting,” Ergonomics, vol. 54, no. 8, pp. 733–744, 2011.
A. Werth and K. Babski-Reeves, “Effects of portable computing devices on posture, muscle activation levels and efficiency,” Appl. Ergon., vol. 45, no. 6, pp. 1603–1609, 2014.
J. G. Young, J.-H. Lin, C.-C. Chang, and R. W. McGorry, “The natural angle between the hand and handle and the effect of handle orientation on wrist radial/ulnar deviation during maximal push exertions.,” Ergonomics, vol. 56, no. March 2015, pp. 682–691, 2013.
J. G. Young, M. B. Trudeau, D. Odell, K. Marinelli, and J. T. Dennerlein, “Wrist and shoulder posture and muscle activity during touch-screen tablet use: Effects of usage configuration, tablet type, and interacting hand,” Work, vol. 45, no. 1, pp. 59–71, 2013.
L.-P. Hsiao and C.-Y. Cho, “The effect of aging on muscle activation and postural control pattern for young and older computer users,” Appl. Ergon., vol. 43, no. 5, pp. 926–932, 2012.
M. L. Wright, D. E. Adamo, and S. H. Brown, “Age-related declines in the detection of passive wrist movement,” Neurosci. Lett., vol. 500, no. 2, pp. 108–112, 2011.
4030/11/2018 lilian.mottiader[at] ucd.ie - HCI@UCD