Mobile Mixed Reality in DIEM

Tivit Results and Business Seminar12.4.2011

Charles WoodwardVTT

• General goals– Location based <-> around shopping mall, or city– Close range <-> inside shop, products

Tracking for Mobile Mixed Reality

Nokia CityScene

Nokia CityScene (contd.)

• Department of Media Technology– Research on real-time camera tracking,

joint ISMAR tracking contest entry with VTT– Implemented generic 3D reconstruction

software package– Research on fast and scalable geometric

computer vision algorithms

• Department of Information and Computer Science

– Research on contextual information retrieval and multimodal relevance feedback

– Developed methods for face, object and context recognition

– Studied selection of relevant information based on current context and use history

– Implementation of the second generation platform of the Virtual Laboratory Guide on N900 mobile device

Research at Aalto

• Annual academic and industrial tracking competition.– A simulated storage room with many objects and tricky features– Reference points (and their global coordinates) are provided– Calibration phase can be done e.g. on a previous day– On contest run, participant will receive the list of object coordinates– The user has to pick the objects in the given order

• Participation by VTT and Aalto's joint solution– Five competitors (incl. Fraunhofer, Metaio, ...)– We had the only mobile phone implementation (1st ever)– We managed to identify all items (results/accuracy unknown)

ISMAR 2010 Tracking Competition

TrackingServer

ReconstructionServer 1

ReconstructionServer 2

…

ROS

ImageDatabase

– Client sends an image– Server replys with

6-DOF pose and 2D-3D point pairs

– Server can use the imageto improve databases and reconstructions

• Still required– Local recovery based on

feature descriptors– Query server for more

information while tracking– Possibly other sensors

UI for data manipulation

Tracking Solution for ISMAR 2011

• Overview– Server provides global

initialization and recovery

– Client tracks locally

Mobile Outdoors Tracking

• VTT's background work at AR4BC project– Augmented Reality for Building and Construction– Augmented visualization of 4D Building Information Models

• Now implemented for mobile phone– N900, client/server solution– RIFF algorithm by NRC and Stanford– Speed ups and accuracy improvements by VTT

Photorealistic Rendering for Augmented Reality

• Challenges, e.g.– Materials, incl. reflections,

transparency– Lighting: direction, intensity,

colors, soft shadows– Accounting for real world lighting

conditions– Matching computer graphics with

webcam video quality

• Comparisons– Left - standard 3D graphics– Right - using VTT methods

ALVAR Subroutine Library

• "A Library for Virtual and Augmented Reality”– Download at www.vtt.fi/multimedia/alvar.html

• Modules– Basic / Pro -> Desktop (Windows / Linux) – Mobile (Symbian & Maemo, iPhone, …)– Render (Windows, ...)

• Tracking features– Accurate & fast marker pose estimation, error

concealment; marker fields, ; optical flow tracking, hiding of markers, etc.

– Markerless tracking: based on 2D templates, or 3D features ("SLAM"); sensor fusion

• Users– OEM licensing e.g. by Vuzix (data glasses)– Columbia Univ. (Goblin XNA / Microsoft)– MIT (6th Sense), hundreds of others

MMR – Current & Next Steps @ TUT/IHTE

• Online survey for people using existing AR applications on iPhone or Android (ongoing)

• E.g. Layar, WikiTude, Junaio, WhereMark, TattooAR, StickyBits, MagiCam …• Focus: benefits & usefulness of AR, user experience, UI & functionality

issues• Targeting several hundred respondents

• Pilot study of the NRC’s CityScene panorama-based mixed reality client (Summer 2011)

• A 2-week study in U.S. with various user groups creating content on the mixed reality panoramas and reporting their experiences

• In collaboration with VTT & Aalto, developing an AR pilot in verkkokauppa.com shop in Ruoholahti (Spring 2011)

• Goal: to concept and implement a functional MAR service during 3rd project period of DIEM

• Concepting design implementation user experience evaluation with end-users

11

TUT / Human-centered technology 4/11/2011

Contact:

Prof. Charles WoodwardVTT Technical Research Centre of Finland

Vuorimiehentie 3, Espoo, FinlandTel: +358 40 500 1514

charles.woodward@vtt.fihttp://www.vtt.fi/multimedia