2.9.2011

1

METHOD FOR VISUALISATION AND ANALYSIS OF HAND AND HEAD MOVEMENTS IN SIGN LANGUAGE VIDEO Tommi Jantunen University of Jyväskylä, Sign Language Centre, Finland

Matti Karppa, Markus Koskela, Jorma Laaksonen & Ville Viitaniemi Aalto University School of Science, Department of Information and Computer Science, Espoo, Finland

GESPIN 2011, Bielefeld (Germany), 6 September 2011

Demo - output of the method imported into ELAN

URL of ELAN: http://www.lat-mpi.eu/tools/elan/ (Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands); See O. CRASBORN, and H. SLOETJES, ”Enhanced ELAN functionality for sign language corpora”. In Proceedings of 3rd Workshop on the Representation and Processing of Sign Languages at 6th LREC conference, Marrakech, Morocco, May–Jun 2008, pp. 39–43.

Processing steps of the method

1. Face detection 2. Detection of skin-coloured regions

3. Skin blob detection

4. Corner point detection 5. Feature point tracking 6. Fitting of active shape models

Demo - feature point tracking & active shape models

J. SHI, and C. TOMASI, ”Good features to track”. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR ’94), Jun 1994, pp. 593–600. T. F. COOTES, D. H. COOPER, C. J. TAYLOR, and J. GRAHAM, ”Active Shape Models - Their training and application”. Computer Vision and Image Understanding, vol. 61, no. 1, pp. 38–59, Jan 1995.

'…how  the  deaf  clubs  would  func2on  without  these  posts?'  

'…but  does  this  mean  that  deaf  clubs  are  old  fashioned?'  

Types of motion information we are able to calculate per a tracked body part (both hands and the head)

Feature point tracking only •  The number of tracked motion points

Feature point tracking & active shape modeling •  The amount (& direction) of horizontal and vertical motion

•  Length of the velocity vector (i.e. speed) •  Length of the acceleration vector (currently fpt only!)

Results visualised in ELAN and used for annotation

•  Tracked data in ELAN helps the annotator to align annotations more precicely

2.9.2011

2

… linguistic analysis

•  Example question: What are the phonetic characteristics of headshakes in FinSL?

... and kinematic analysis

•  Kinematic analysis without laboratory settings and MoCap equipment

Cf. kinematic analysis of dominant hand speed with MoCap data

T. JANTUNEN, ”Signs and transitions: Do they differ phonetically and does it matter?”. Sign Language Studies, vol. 13, no. 2, 2013, (forthcoming).

Demo - comparison with MoCap data (speed)

Conclusion • The method is already a feasible tool in the annotation

and analysis of sign language and gesture

• Plans for the future •  More detailed modeling of articulators (e.g. hands and arms, and

facial gestures) •  Development of a more automated mechanism for the annotation

of sign language corpora

•  Further comparison with MoCap data

• Exact plans for sharing the software have not yet been made but we wish people to use it (when it's finished)

Thank you! • My colleagues at Aalto University School of Science • Päivi Rainò at HUMAK University of Applied Sciences • Danny De Weerdt, Birgitta Burger, Tuija Wainio at JyU

• For more information, see CoBaSiL home page at http://research.ics.tkk.fi/cbir/cobasil/

… and 3BatS home page at http://users.jyu.fi/~tojantun/3BatS/Home.html

• Please also see our paper in the proceedings for further information