© Fraunhofer IGD Special considerations for Standards in Mobile AR International Workshop on AR...

transcript

Special considerations for Standards in Mobile ARInternational Workshop on AR Standards – 11. + 12. October 2010, SeoulTimo.Engelke@igd.fraunhofer.de - Institute for Graphics Research IGD

Fraunhofer IGDInstitute for Computer Graphics Research

Mission: Technologies and Applications of Visualization, Interaction and Communication in the field of ICT

Director: Prof. Dr. techn. Dieter W. Fellner 4 Locations: Darmstadt, Rostock (Germany), Singapore and Graz

(Austria) 180 Researchers (FTE) 14,7 Mio. € Turnover -> Department Virtual and Augmented Reality

Dep. Virtual & Augmented RealityResearch Topics

Computer Vision

Computer Graphics

Interaction Technologies

Overview

The aim of the presentation is triggering consciousness about the idea of AR and presenting an open approach practically ..

Where are the problems in standardization of AR? What are the common structures and where are variations? What is the role of those structures in the context of AR? Technological hardware issues Why and what kind of standards are there and do we need? Our approach to overcome the mentioned issues and help

developing standards

Cannot cover the whole horizon of problems .. But can try …

ProblemsGeneral goal of having standard(s) which apply to general needs of AR.

What is the problem? There are tracking technologies for AR, but they are in an early

stage. Every application type needs its custom strategy, technology and specialized content.

What do we want to cover with a standard? Due to the complexity of many parties involved in the field of AR and

the high potential of possibilities. The terminology AR is not clearly differentiated generally and can

lead to confusion. Which sense is augmented by what and through what? 2D,3D, training, leading, informing, disinforming, helping, covering

etc.. Diverging hardware and software technologies Bringing the paradigm to content masses

CommonTechnologies

Arising potentials of technological developmentin contrast to commonly used technologies in apps Mass -

Solution

Customized Mass-Solution

CommonTechno-logies Cutting

Edge-Solution

Scientific SpecificLaboratory-Solutions

Development

Provide tools for everyone

Layar, Wikitude etc..

Branded…

i.e. iLiving

Complexit

Augmented RealityAbstract Building Blocks

ContextEngine

Content

AnnotationEngine

ContextInformation

Interaction Engine

ApplicationModel

Reference Model Annotation Model

Sensors

Output

InteractionDevice

Computer Vision Based Augmented Reality

Camera

VisionEngine

Content

3D AnnotationEngine

Extrinsics (+ Intrinsics,Occlusions,Lights,Models etc..)

Interaction Engine

Image+Intrinsics

Display Image

Screen

ApplicationModel

Role of Reference Model

Retrieval of the camera Pose (6Dof) Find the absolute coordinates of user and objects

GPS, Compass, Acceleration Data Use prototype model for registration (i.e. Poster, Marker, Lines…)

Representation Absolute in latitude, longitude, altitude, roll, pitch, yaw Relative in Cartesian coordinates and angles (or axis angle, quaternion..) In relation to some object or object group – real or virtual?

Interaction Find differences of formulated and real world (POI,AOI,VOI)

This creates potential content for interaction Track movable and deformable objects Detect user interaction/behavior using sensors (Hands, gestures etc..)

Allow feedback of generated data

Role of Visual Annotations in ARBuilding elements in GUIs

Visual Aids Direct Visual Aids (DVA): Superimposed 3D-models/animations

(direct association with object) Indirect Visual Aids (IVA): “Annotations” related to tracked object

(reminder)

Mental Model Building User’s internal representation of the task Can be improved by providing context information

Passive Parts Information about the task in general Higher-level instructions global picture

Collaborative Annotations (expert trainer)

Role of the interaction model

Customizing the Annotation Model Manipulation Scene Graph Interactive Content

Customizing the Reference Model Dynamically change tracking behavior and point of focus (POI) Allow interchange in between Annotation and Registration Model Use of declarative formulation as far as possible. Allow scripting for specialization Tracking and Events: 2D-Projected Positions with distance hint Detection Events (i,e. Object enters/leaves display) Interaction of user/objects with the scene

Content must be delivered free or encrypted.. Who will do this?

Hardware technology issues on mobile devices

Low computational resources in relation to modern PCs Limited memory OpenGL ES Different camera modules (Intrinsics) Sensor support and calibration Partly supported hardware acceleration Software Interfaces and Languages Symbian (C++, Microkernel, Qt) Android (Java, C++ through NDK) iPhone (Obj-C, C++ support)

Algorithms need to be optimized for HW Costly and hard to maintain large set of handsets in one common

Code-Base

Open StandardsSustainability for AR-ContentApplications and generation tool chains should be based on.. X3D (ISO/IEC 19775:2004) HTML5 ECMAScript (ISO/IEC 16262:2002) JavaScript CSS STEP-Format KML

Benefits Sustainability Interoperability Porting Rapid development

X3DOM (x3dom.org)HTML5/X3D Integration Model

Simple car configurator to interactively change color and rims

N900 with miniB

X3DOM (x3dom.org)HTML5/X3D Integration ModelX3DOM: experimental open-source framework and runtime One way for integration of HTML5 and declarative 3D Ongoing discussion in the Web3D and W3C communities

Declarative content design Allows embedding 3D data directly into the HTML-DOM First approaches of CSS integration Application concepts map well to generic scene-graph

Runtime independent (supports native implementation, plugin, WebGL)

Suitable for desktop and mobile systems alike

instantReality – mobileAR - BrowserExtendable generic AR-Web-Browser for mobile devices based on HTML5 and Javascript

mobileAR – Open Architecture Proposal for open realization of declarative approaches

Generic Application (arBrowser)OS

WEB-View (HTML5 +JS)

DOM-Declarative

JavaScript

SVG A2ML

Native Interface

VisionRender Sound

Open Annotation Open Context

Content

AR-KML

Sensors

ualiza

tion o

develo

mobileAR – AR – Browser for mobile devicesAdditionally to Web-Browser a renderer and vision engine is integrated. Extends the internal Web-Browser with a simple but powerful API Interaction and visualization can be done declarative in HTML and/or

via JS Can render X3D via native renderer or even X3Dom Generic Vision Interface Integrates Vision Processor (VisionLib) for

Fiducials (Markers, Image-Markers) Poster-Tracking Object recognition Panorama and 3D-Model Based scene tracking Server based Tracking (bidirectional streaming)

Additional Features (about to come…) Remote Video & Audio Support Scenarios Encryption and packaging

Camera

Live/Pause

dImage

3D-View

optional

DebugView

WEB-View

HTML5+JS

Next steps…Identification and unification of common technology data Annotation Data Reference Data

Create open framework for declarative approaches Open Annotation Engine Open Context Engine Unify Vision and Sensor based approaches KML in the context of industrial maintenance applications Evaluate using show-cases

Extend native interfaces Incooperate Sound A2ML Better content management (Offline Usage, Encryption)

Make it free for masses for developing

Happy Workshopping

THANK YOU !

Building Block Hierarchy – What data to processContext EngineBased on available sensors in device

ContextEngine

LocalizationEngine

VisionEngine

AuditiveEngine

Other sensors..

GeolocationEngine

Building Block Hierarchy - DerivativesContext EngineBased on capabilities of device

AnnotationEngine

2D-AnnotationEngine

VideoEngine

AuditiveEngine

3D-AnnotationEngine

GeolocationEngine

Sound-Annotation

Engine

Parties involved in the game of AR

Physical Player The user of the app Network Provider Producer of Cellphones Application Developer Content

developer/provider Technology developer AR Standardization

…some of them want to

Learn something Network Save costs Save resources Save time Earn money Have fun Sustainability

Identification of needs...

App Devel

ContentProvide

ContentCreator

User(s)

CommunicationProvide

DeviceCreator

Techno-logy

The user of the app Network Provider Producer of Cellphones Application Developer Content developer/provider Technology developer AR Standardization

Applications (Fallback Model in Action)

IE8 Plugin (OpenGL and Raytracing) and native iOS app with x3dom layer

Runtime independent – supports (native implementation,) plugin, and WebGL

Geo Located - Augmented RealityAbstract Building Blocks

LocalizationEngine

Content

AnnotationEngine

Position

InteractionEngine

ApplicationModel

Localization Sensors

Output

InteractionDevice

Geo located and Vision Based Augmented RealityTechnically Abstract Building Blocks

Camera

LocalizationEngine

Content

2D AnnotationEngine

Extrinsics

Interaction Engine

Display ImageScreen

ApplicationModel

Localization Sensors Intrinsics

Source: Layar

© Fraunhofer IGD Special considerations for Standards in Mobile AR International Workshop on AR...

Documents