A Visual Ontology-Driven Interface for a Web Sign Language Dictionary

M. Felice – EURACT. Di Mascio – University of L’Aquila

R. Gennari – Free University of Bozen

2/36

Outline

• Context of use– Italian sign language (LIS)– The e-LIS project– The e-LIS ontology

• Users and organisational requirements• Design solutions• Conclusions and future work

3/36

Italian sign language: definitionContext of use -> Italian sign language

• Visual-gestural language developed in Italiandeaf communities

• Signs are used instead of voice to conveymeaning

• A sign is realised through hand(s), facialexpressions, and mouthings

• Sign Languages (SLs) have a tempo-spatialnature, they are difficult to write

4/36

Italian sign language: transcriptionContext of use -> Italian sign language

• Several transcription systems proposed in the literature for SLs

• The Stokoe transcription system can beemployed for notating single and decontextualised signs

• LIS -> Radutzky transcription system

5/36

Radutzky transcription systemContext of use -> Italian sign language

• Four classes– Handshape– Palm orientation– Movement– Location

• Non manual components are neglected!

6/36

The e-LIS projectContext of use -> The e-LIS project

• A research project developed at the EuropeanAcademy (EURAC) of Bozen-Bolzano

• Cooperation with– ALBA from Turin, active in deaf studies– KRDB of Free University of Bozen-Bolzano– University of L’Aquila

• Main goal– Create a bilingual website (Italian and LIS), offering a

dictionary, two search engines and a forum

7/36

The dictionaryContext of use -> The e-LIS project

• It allows translations form Italian to LIS and from LIS toItalian

• Not only the translation, but also lexicographicinformation are shown

• Information is shown both in Italian and LIS (digital videos are used)

• Word search and sign search

• http://elisdiz.eurac.edu/diz

8/36

Sign searchContext of use -> The e-LIS project

9/36

Current advantages and limitsContext of use -> The e-LIS project

• Advantages– Wizard-like interaction– Iconic representations of sign components

(transcription system is hidden)– Video of signing people are shown

• Limits– Powerful undo tools are not present, thus the

decision-making process is not well supported– The interaction is too long (too many mouse clicks)– Users are not supported, thus they can make

mistakes– Users have to know the rules of sign composition

10/36

The e-LIS ontologyContext of use -> The e-LIS ontology

• Ontology: a formal representation of a specificknowledge, which encodes semantics

• The e-LIS ontology represents a sign in LIS, and it encodes the rules of sign composition (e.g.,how the Stokoe classes are linked, or how a signis realised)

• Domain ontology (+ application ontology)

11/36

Overview of the domain ontologyContext of use -> The e-LIS ontology

12/36

Taxonomy of HandshapeContext of use -> The e-LIS ontology

13/36

Taxonomy of PalmOrientationContext of use -> The e-LIS ontology

14/36

Taxonomy of LocationContext of use -> The e-LIS ontology

15/36

Taxonomy of OneHandMovementContext of use -> The e-LIS ontology

16/36

Taxonomy of RelationalMovementContext of use -> The e-LIS ontology

17/36

Revised version of the ontologyContext of use -> The e-LIS ontology

• Original ontology– Many concepts and relations– Intermediate abstract concepts (e.g. curved shapes)

• Simplified ontology– Only the five concepts related to Stokoe classes– Focus on the taxonomy of such concepts– Stokoe classes are 0-level elements– Handshapes are grouped into more intuitive concepts

(0-finger, 1-finger, etc.)– Relational movement and one hand movement are

considered both children of movement Stokoe class

18/36

Outline

• Context of use

• Users and organisational requirements– Profiling of users– Tasks– Usability goals

• Design solutions• Conclusions and future work

19/36

Profiling of usersUsers and organisational requirements -> Profiling of users

• Users– Deaf/hard-of-hearing or hearing people– LIS fluent or novice

• Deaf people– cannot be assumed to be literate [1]– problems comprehending written language– can have problems coping with abstract concepts [2, 3]– larger spatial memory than hearing non-signers– visual learners– can cope with complex visuo-spatial structure– prefer pied interfaces: different shapes, different colours, etc.

20/36

TasksUsers and organisational requirements -> Tasks

• Browsing: specify the components of the sign(e.g., handshape)

• Querying: retrieve the sign

• Result visualisation: browse the results

• Currently: users can specify the sign components by browsing the ontology -> query-oriented browsing

21/36

Usability goalsUsers and organisational requirements -> Usability goals

• Effectiveness– users are supported, thus minimising mistakes

• Efficiency– the interaction with the interface is fast and decision-

making process is well supported, thanks to efficientundo tools

• Users’ satisfaction– the interface is well organised and plain, thus

minimising cognitive effort and keeping users’attention alive

22/36

Outline

• Context of use• Users and organisational requirements

• Design solutions– Current prototype– Ontology visualisation

• Conclusions and future work

23/36

The current prototypeDesign solution -> Current prototype

• Successive refinements of the initial idea and mock-ups

• It is an information visualisation system– Visual metaphor– Number of dimenions– Space-saving strategy

24/36

Visual metaphorDesign solution -> Ontology visualisation

• We need to visualise a tree structure (the taxonomy) -> Tree visual metaphor

• TreeMap visual technique: the tree is shown in a space-constrained layout, i.e. the tree is turnedinto a planar space-filling map [4]

• Each treemap is composed of several areas, whose extents are proportional to the number of child concepts the area represents

25/36

Visualisation of ontology conceptsDesign solution -> Ontology visualisation

26/36

Dimension and space-saving strategyDesign solution -> Ontology visualisation

• Number of dimensions– 2.5 D– Simulated 3rd dimension

• Space optimisation• Idea of link between concepts

• Space-saving strategy– Focus + context

27/36

Focus + context strategyDesign solution -> Ontology visualisation

28/36

First step: 1-level conceptsDesign solution -> Ontology visualisation

29/36

Temporary selectionDesign solution -> Ontology visualisation

30/36

Second step: 2-level conceptsDesign solution -> Ontology visualisation

31/36

Expert-based evaluationDesign solution -> Ontology visualisation

• Effectiveness– Users are guided– Errors are minimised

• Efficiency– Mouse clicks are minimised– Undo tools -> decision-making (users know the next step of the

search path)

• Users’ satisfaction– Few graphic elements– Plain interface– Pied interface

32/36

Outline

• Context of use• Users and organisational requirements• Design solutions

• Conclusions and future work

33/36

ConclusionsDesign solution -> Conclusions and future work

• Advantages of the e-LIS ontology– Minimisation of errors– Creation of a dynamic interface– Non-expert users can use the dictionary

• Advantages of proposed interface– Spatial-based, thus closer to deaf people wrt the

current interface– Decision-making support (two-stage selection)– Intuitive undo tools (the miniatures)– Less obtrusive interaction (mouse over)

34/36

Future work (1)Design solution -> Conclusions and future work

• Tests to evaluate:– Treemap choice– Use of colours– Interaction strategy– Cognitive effort– Experience curve (interface usage and sign

composition)– Usability goals– Overall usability of the prototype

35/36

Future work (2)Design solution -> Conclusions and future work

• Effective intuitive icons instead of textlabels have to be designed– Icons are usually more intuitive than text (if

well designed)– Deaf people have problem reading and

comprehending written Italian

36/36

References

• [1] P. V. Paul. Literacy and Deafness: the Developmentof Reading, Writing, and Literate Thought. Allyn & Bacon. 1998

• [2] C. Chesi. Inferenze Strutturali. Analisi sull’Uso degli Elementi Funzionali nel Linguaggio Verbale dei Bambini Sordi. Master’s thesis, Siena U. 1999/2000

• [3] D. Fabbretti. L’Italiano Scritto dai Sordi: un’Indagine sulle Abilità di Scrittura dei Sordi Adulti Segnanti

• [4] B. Shneiderman. Treemaps for Space-ConstrainedVisualization of Hierarchies. http://www.cs.umd.edu/hcil/treemap-history