Welcome

OpenInterface Project FP6-351821 sept 2006 - 33 monthsToday = M9Coordinator: Laurence Nigay, University of Grenoble 1 - Joseph Fourier

Glasgow, May 24, 2007

3

Outlines

• Overview

• OI Platform• Component based platform• Runtime Kernel• Repository of modalities• Graphical editor

Overview

5

Consortium

6

Consortium

• Complementary expertise

Signal Processing

UCLMULT

Multimodal Interaction

ALL

Hardware

UGLASWIN

Software

Engineering

UJFUGLA

FITARCPCLITXT

Application domain: Large information space FT UJF UGLA

Application domain: Game ARC FT UJF UGLA

Ergonomics

Usability

FITFT

7

OpenInterface: objectives

Scientific understanding of multimodality

Testbeds and Validators

Open source

platform

Extension of

standards

8

Multiple modalities

Multiple modalities

Interaction Continuity

The 2 action lines• Two complementary goals:

Extend the sensory-motor capabilities of mobile devices and environments

9

Modalities

+

The 2 action lines

• Two complementary goals: Extend the sensory-motor capabilities of mobile devices and environments

10

Standards

Validation

R&D

Ideas

ConceptPilo

ts

MultimodalInteraction

within a pervasive environme

nt

Multimodal

Interaction on a mobile device

Market

Concept

OpenInterface

platform for multimodality

Existing standards for multimodality

Why 2 action lines?

11

Standards

Validation

R&D

Ideas

ConceptPilo

ts

MultimodalInteraction

within a pervasive environme

nt

Multimodal

Interaction on a mobile device

Market

Concept

VALIDATORS

Why 2 action lines?

TESTBEDS

12

Testbed

• Large Information Space

13

Testbed

• Game

Increasing the intensity of the

vibration depending on how damaged the

helicopter is

Speech command for the speed

Accelerometer or speech

14

Testbeds and validators

Software prototype

Requirements

Ergonomic evaluation

Design

Software prototype

Design

Software prototype

Design

Software prototype

Ergonomic evaluation

Ergonomic evaluation

Ergonomic evaluation

TESTBEDS

OpenInterface Platform

Unit test

Component test

Acceptance test

Code

Final usable productV

AL

IDA

TO

RS

OI Platform

16

OI Platform

• Open source platform

• To make it easier to develop effective interactive systems that exploit multimodal interaction techniques in modern application settings (mobile, ubiquitous, context sensitive)

17

OI Platform

• Open source platform for developing multimodal interfaces

• that enables quick replication,• that enables iterative user-centred design,• that contains a rich and extensible set of

modalities,• that enables a focus on innovation

(new modalities or forms of multimodality),• that supports dynamic selection and

combination of modalities to fit the ongoing context of use.

18

OI Platform

- Repository of modalities (components)- Repository of Fusion/Fission Mechanisms (components)

- Runtime Kernel

Tool for the application designer/developer

Using/Assembling the components

Tool for the provider of

components

Adding new components

19

OI platform

• Who are the users?

• Three types of users

1. Programmers• Providing new components (populating the platform)

• New device• New modality• New fusion/fission mechanism

2. Multimodal Application Designers• Assembling components

to define multimodal interaction3. End-users of multimodal applications• Interacting with the multimodal application• Adaptation at run-time

Run- time

Design time

OI platformRuntime kernel

21

Outlines

• Kernel• Overview• OI Components• CIDL, PDCL

• Current Features

• Next Features

22

Overview

• Useful high level programming language support:• C/C++, Java, Matlab, Python

• Plug n Play behavior.

• Lightweight skeleton platform.

23

Components

• Platform does not specify/define a Component Model (!!)

• Platform sees the external software as components

• components are assumed to behave like objects.

• Software/Component delivered as reusable independent unit with exported/imported I/O interfaces.

• Platform Kernel aims for the interaction of heterogeneous components

24

Component: OI point of view

Facets can be seen as services provided by a component

A Component is a set of facets

This is how any external software is seen by the Kernel

25

Reusable Independent Unit

• Software

• With API

• With installation/configuration doc

• Without explicit dependencies with other components

• Tested and validated

• Much likely a library (!!)

•  Impossible to test new modalities with an exe, for instance.

26

Exported/Imported IO

• Components only declare communication interface

• Enforce the requirement for independence

• Export IO (functions, etc.) when providing a functionalityDisplay an image

• Import IO for requesting a functionalityAn event handler

27

OpenInterface CIDL

• Designed to describe Independent and Reusable component.• (Intentionally) Not as expressive as WSDL (etc.)• Restricted on purpose to cover common attributes

of various programming languages• Describes component interface using a

programming language independent description (XML).• Not intended to be written by human• Generation tools are not yet available though

(being developed)

28

I/O Interface

• Functions provided by a component facet

• 3 types of functions

• SinkWith no return argument

• SourceWith return argument

CallbackWith no return argument, for importing external

functions

29

I/O interface: types

• Only supported languages common types

• Common primitives types

• Bool, integer(8,16,32,64bit), float, double, String

• Array of primitives types

• Complex types

• Objects

30

Runtime, Pipeline

• Pipeline description language to setup a running application initial connections.

31

Pipeline configuration

• Components• Remote or local

• Connections• Simple direct link• Multicast (Publisher/Subscriber)• (Temporal) Fusion

• Execution• Pin initialization values• Threaded execution

32

Current Features

• Supported Platform

• Linux and Windows• Supported Languages (with minor restrictions)

• Java• C/C++•Matlab

• Description languages

• CIDL, interfaces description• PDCL, static pipeline description

• Temporal Fusion components• Simple components distribution • Specification on how to write a language plug-in• Specification on how to write custom connectors

33

Next Features

• Installation Packages • Windows installer, debian package

• Pipeline • Kernel API component for controlling application dynamic

connections• Description languages tools

• Semi-Automatic CIDL generator•Graphical tool for pipeline composition

• Supported Platform• Other OS

• Supported Languages

• .NET framework• Connectors• More connectors plugins

OI platformRepository of modalities

35

Summary

• Device Components• Filter/Translator Components• Debugger Components (Development)• Connector Components• Future Work

Device components

37

Device Components

Component NameSource Provider

Integrator

OS State

Win32MouseDriverEmul UCL UCL Win Stable

XWindowMouseDriverEmul UCL UCL Linux Stable

Win32KeyboardDriverEmul UCL UCL Win Stable

IEEEFrameGrabber UCL UCL Linux Stable

ARToolkit UCL UCL Win/linux Stable

SpeechRec UCL UCL Win/Linux Stable

Shake UGLA UJF Win Beta

SpyTagClient MULT UCL Win/Linux Beta 

MultitelSpeechRecognition MULT UCL Win Beta

FingerTracker UJF UJF Win Beta

Filter/Debugger/Connector Components

39

Filter/TranslatorComponents

Component Name Source Provider Integrator OS State

Type&ValueFilter UJF UJF Win Stable

EventsGenerator UJF UJF Win Stable

Example: Int 1 string go

Example: Int 1 string zoom 10 string in

40

DebuggerComponents

Component Name Source Provider Integrator OS State

DataViewer UJF UJF Win Beta

Osciloscope UGLA UGLA Win Beta

Logger (data in a file) UGLA UGLA Win/Linux Beta

41

Connector Components

Multicast UCL UCL OI Stable

Complementarity Fusion UCL UCL OI Stable

RedundanceEquivalence Fusion UCL UCL OI Beta

Component Name Source Provider Integrator OS State

42

Components in Progress (UGLA)

• Abstract Devices•Gesture Language Technique

•Spatial interaction device•Pointing / panning / zooming•Lots of variants

•Earcon device

•Tacton device

• Wrappers

•Phidget OI wrapper

•ARToolkit OI wrapper

43

Futur work (UGLA)

• Generic GPS device

• Smell device

• Physiological sensors

• Context-sensing device

•Audio recognisers

•ambient noise level detector

•Vibration recogniser

•Light level recogniser

•Proximity sensor

• [...]

OI platformGraphical Editor

45

Content

• Introduction• Demo• Roadmap• Conclusion

46

Introduction

• What? : Graphically create an assembly of (OI) components

• At first: Technical (PDCL) Description

• Objective: OI Interface

• Not only a graphical editor but an IDE

• Actually: static assembly of components

• Next version: dynamic assembly of components

• Why?:

• Fast prototyping

• Runtime Testing and Monitoring

47

Demo

• Fast prototyping

• Create assembly

• Run the assembly

• Change Multimodal Interaction

48

Roadmap

• Introspection

• Runtime connection

• Monitoring of the components execution

• Usage data capture

• Link with Conceptual Model

• Multimodal interaction

• Multiple OS support

Thank you!