3a8 picture driven computing in assistive

transcript

www.aegis-project.eu

PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY

Silvio Bonfiglio1 - George Kourousias2

1. FIMI, a Barco Company, Saronno (Italy)2. Synchrotron Radiation Facility, ELETTRA, Trieste (Italy)

1st International AEGIS Conference 7-8 October 2010, University of Seville

Domain: Improved Accessibility in ICT

It may positively affect and improve our society

PICTURE-DRIVEN COMPUTING IN ASSISTIVE TECHNOLOGY FOR ENHANCED ACCESSIBILITY1st International AEGIS Conference 7-8 October 2010, University of Seville

Two Paths to Accessibility

i.e. a product that – without add-ons – can be experienced with similar ease of use by any user independently by the disability.

DIRECT ACCESS Universal Design (UD) – Design for All

INDIRECT ACCESS Multi-modal, Personalized Accessible Design

i.e. a product that is made accessible for a specific category of users by incorporating as an add on a specific AT componentor an extra interaction modality.

Picture Driven Computing

•We aim to introduce it as an Assistive Technology

•Adding a Modality to the GUI of the computer

•Accessibility through Indirect Access

We define as Picture-driven Computing the Computing paradigm that

has as core Data Abstraction the Visual Information in a 2-

dimensional spatial domain. This Visual Information is what intends

to be visible on a graphical Output device.

Example: a Picture-Driven Search

•Using Picture objects instead of keywords

Sikuli, a Picture-Driven language

•Developed recently at MIT by T. Yeh, T.H. Chang and R.C.Miller for the search and

automation of GUI elements using screenshots

•Based on Sikuli-script , a scripting module that allows programmers to use

screenshots of GUI elements to control them programmatically without any API.

•Python

•Open Source & Multiplatform

•We use it as our main paradigm for Picture-Driven Development of AT

Sikuli: How it Works

A. a computer vision library

B. a mechanism to programmatically “dictate” Input actions

C. a mature underlying language (Python) for traditional programatic control

Sikuli: An Example

Goal: clicks all the “minimize window” buttons on the screenPicture-Object Input: target image of the button

i) then the systems computer vision (A) will locate the x,y position of that on the screen

ii) (B) the “mouse control” will “send” the mouse on x,y, and

iii) through a for or while loop (C) will click (B) for all the targets that (A) has found.

Sikuli as an Accessibility Design enabler

Automation Scripts: where Sikuli script - following a predefined set of mouse

and keyboard actions- preforms a task that may be difficult otherwise to the user with a

disability.

Event Raisers: where the system can keep monitoring the user's screen for

specific graphical objects and raise an event/alarm and consequently trigger a set of

programmatic actions.

Bridge Systems: where applications can communicate with other software even if

an API is not provided.

We Identify the following 3 classes:

+ Hybrid approaches

Future work

•Further research on Picture-driven Computing.

•Comparison of “indirect access” Accessibility Design versus the

“direct” one (Universal Design) for the computer as an AT-aid.

•Additional development of Sikuli AT Scripts.

•Evaluation of the technical issues and performance of Sikuli.

Acknowledgments

further info:

silvio.bonfiglio@barco.com

george.kourousias@elettra.trieste.it

3a8 picture driven computing in assistive

Education