1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Trainable Sketch Recognizer for...

1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007

Trainable Sketch Recognizer forGraphical User Interface Design

Adrien Coyette1, Sascha Schimke2, Jean Vanderdonckt1 & Claus Vielhauer2

1Université catholique de Louvain (UCL)School of Management (IAG)Information Systems Unit (ISYS)Belgian Lab. of Computer-Human Interaction (BCHI)http://www.isys.ucl.ac.be/bchi

2Otto-von-Guericke Universität MagdeburgFakultät für InformatikInstitut für Technische und Betriebliche Informationssysteme (ITI) http://wwwiti.cs.uni-magdeburg.de/


Considerable importance of user interface in application development (50% to 70 % of the application code [Myers and Al, 2000] )

Apparition of many high fidelity graphical user interfaces editors on the market

Motivations


But designing the right User Interface (UI) the first time is very unlikely to occur

instead, UI design is eminently open iterativeiterative and incompleteincomplete

Motivations


Paper ( or whiteboard) used most of the time:familiar and unconstrained approach

fast to learn and quick to produce

focus on basic structural issues instead of unimportant details

it is very appropriate to convey ongoing, unfinished designs, and it encourages creativity,

can be performed collaboratively between designers and end-users

Motivations


Motivations: empirical findings

UI Drawing or sketching increases participatory design and subjective satisfaction with respect to UI editors

[Landay & Myers, 1995, Plimmer & Apperley, 2004, Buxton, 2005]

Equal usability problems found for a sketched UI than with an edited one[Virzi et al., 1996]

Explanatory power of a sketched UI does not decrease with respect to edited one; tool support is preferred over paper prototyping

[Walker et al., 2002 ]

A sketched UI delivers same quantity and quality of outputs than an edited one[Sefelin et al., 2006]

UI sketching fosters alternative designs and communication [Tohidi et al., 2006]

Supports throw-away prototyping due to low cost[Schneider & Petrie, 2006, Bailey & Konstan, 2003]


The idea of developing a computer-based tool for sketching UIs naturally emerged from these observations

Such tools would extend the advantages provided by sketching techniques (easily creating, deleting, updating or moving UI elements).

Apparition of hybridhybrid approachesapproaches, combining the best of the hand-sketching and computer assisted user interface design

Related Works


State of the Art:Eliciting requirements

17 different software and approaches surveyedSources: scientific literature and commercial

softwareComparative analysis relying on consistent

analysis gridTool identification (7)Install and first use (4)Tool functions (8)Shape recognition (4)Shape interpretation (7)UI editors (9)

Applied to the same case study

(eCommerce app.)


DENIM(James A. Landay, James Lin, Mark W. Newman, Jason I. Hong)

Related works


Advantages:• Support for scenario-based design• Several levels of granularity• Good documentation• Toolbox of generic widgets• Mature product, based on experimental testing • Zooming facility from local design (e.g. a web page) to a global

design (e.g., a portion of a web site or n entire web site)• Storyboarding facilities based on patterns

Shortcomings:Shortcomings:• No shape recognition and interpretation, thus loosing the effort• No code generation• No preview mode • Only dedicated to web sites

Related works


JavaSketchIt (Joaquim A. Jorge, Manuel João Fonseca, Anabela Caetano, Néri Goulart )

Motivations / Related works


Advantages:• Performance (speed and accuracy)• Multi-stroke gestures• Recognizes rotated shapes • Computationally light • Open source • Requires standard and freely available libraries

Shortcomings:• Mono-window• No scenario editor• Only generates java • Limited widget set• Shape interpretation can only take as input a construct

made of maximum two vectorial shapes



Avoidance of Effort loss

Language neutrality

Ease of use (naturalness)

Flexible processing

Robust scenario editor

Large conceptual coverage

…



SketchiXML


SketchiXML


SketchiXML: Shape Recognition

Recognition engine: Cali libraryRecognition engine: Cali library

A fast, simple and compactA fast, simple and compact approach to identify Scribbles (multi-stroke geometric shapes)

Combines temporal adjacency, Fuzzy Logic and geometric temporal adjacency, Fuzzy Logic and geometric featuresfeatures to classify scribbles.

Recognizes elementary geometric shapes, such as Triangles, Rectangles, Diamonds, Circles… and some gesture commands, such as, Delete, Cross, WavyLine, Move, Copy and Tap.

Shapes are recognized independently of changes in rotation, recognized independently of changes in rotation, size or number of individual strokessize or number of individual strokes

The recognizer has a recognition rate of 94%94% and a fast response



All widgets were based on a set of shape All widgets were based on a set of shape primitivesprimitives

Not extendableRequire a delay between each strokesLimits the number of representation



Trainable Gesture RecognizerTrainable Gesture Recognizer Handdrawn input (sketch) is

superimposed with a grid Each sampling point is mapped to

its closest grid node For adjacent nodes, a stroke

direction (out of 8 different directions) is taken

A string is generated from direction coding of input

Complex task of comparison of drawing inputs is reduced to simple calculation of string edit distances



Distance is shortest sequence of edit commands that transform s to t Simplest set of operations:

• Copy character from s over to t • Delete a character in s (cost 1) • Insert a character in t (cost 1) • Substitute one character for another (cost 1)

Levenstein Levenstein distance distance




SketchiXML: Grammar

<widget type="ListBox"> <representation id="0">

<constraint id="0" shape1="Triangle_3" shape2="Rectangle_0" condition="isInsideInUpperRightCorner" /> <constraint id="1" shape1="Triangle_4" shape2="Rectangle_0" condition="isInsideInLowerRightCorner" />

<shape id="Rectangle_0" type="Rectangle" /> <shape id="Triangle_3" type="Triangle" /> <shape id="Triangle_4" type="Triangle" /> </representation><representation id="1"><constraint id="0" shape1="Triangle_0" shape2="Rectangle_2" condition="isInsideInTop"/>

<constraint id="1" shape1="Triangle_1" shape2="Rectangle_2" condition="isInsideInBottom" /> <constraint id="2" shape1="Rectangle_2" shape2="Rectangle_3" condition="isInsideOnTheRight" />

<shape id="Triangle_0" type="Triangle" /> <shape id="Triangle_1" type="Triangle" /> <shape id="Rectangle_2" type="Rectangle" /> <shape id="Rectangle_3" type="Rectangle" /> </representation> <representation id="2"><constraint id="0" shape1="Line_0" shape2="ListBox_1" condition="isInside" /> <constraint id="1" shape1="Line_0" shape2="-" condition="isHorizontal" /> <shape id="Line_0" type="Line" /> <shape id="ListBox_1" type="ListBox" /> </representation></widget>


Conclusion / Future Works

Conclusionan innovative contribution to the domain of sketch based

design tools: a trainable multi-strokes gesture recognizer Improved the performance of SketchiXML

Future works

evaluating other feature types to describe the gesture shape, e.g. triangular grids

Improving the performances of the algorithm, when running with a large set of representations

Conduct a systematic comparative test with other shape recognition algorithms


Thank you very much for your attention

For more information and downloading,http://www.isys.ucl.ac.be/bchi

http://www.usixml.orgUser Interface eXtensible Markup Language

http://www.similar.ccEuropean network on Multimodal UIs

http://www.usixml.org/index.php?view=page&idpage=29SketchiXML home page

http://www.isys.ucl.ac.be/bchi

http://www.usixml.org/

http://www.similar.cc/

http://www.usixml.org/index.php?view=page&idpage=29

Date post:	16-Jan-2016
Category:	Documents
Upload:	andrew-baker
View:	215 times
Download:	1 times

1 Interact 2007 – Rio de Janeiro, Brazil, 14 September, 2007 Trainable Sketch Recognizer for...

Documents