Augmented Reality with .NET

casey chesnutbrains-N-brawn.com

Dallas C# SIGJanuary 2008

brains-N-brawn.com• Pervasive Computing

– Tablet PC (MVP 03)

– Compact Framework (MVP 04)

– Advanced Web Services (MVP 05)

– Media Center (MVP 06)

– Speech / Office Communication Server (MVP 07)

– Location Based Services– Artificial Intelligence– 3D– Image Processing– Augmented Reality

Source

• http://www.brains-N-brawn.com/wpfAugReal/• http://www.brains-N-brawn.com/wpf3dPart• Environment : 32-bit Vista, VS 2005, .NET 3.0,

Logitech QuickCam Pro Web Cam for Notebooks

Audience

• DirectShow?

• 3D?

• WPF?

• Augmented Reality?

Outline

• Intro to Augmented Reality

• DirectShow

• Augmented Reality (AR) Libraries

• WPF 3D

Introduction

• Augmented Reality (AR) - is the combination of real world data with virtual data.

Audience

• What are some current examples?

Current Examples

• Fighter Pilot HUDs• QuickCam video conferencing• PS3 Eye of Judgment• Sports : Football, Hockey, Nascar• Aircraft maintenance• ARQuake• Car GPS HUD• …• Star Wars chess!

Videos

• QuickCam

• Eye of Judgment

Steps

• Acquire real-world data (camera, HMD, mobile devices)

• Recognize real-world data (computer vision, face detection, fudicial markers, GPS)

• Determine relative position of real-world marker to input source.

• Display composite of real-world data and virtual object (monitor, HMD, mobile device, projector).

Acquire real-world data

• OpenCV / SharperCV

• DirectShow / DirectShow .NET

• Windows Media Foundation

• …

DirectShow

• Used by Windows Media Player, Media Center Edition, Windows Media Encoder, Movie Maker, …

• User for audio/video playback, DVD, TV tuners, web cams, media capture, media transcoding, …

DirectShow

• COM

• GraphEdt tool

• Graph, Filter, Pin, Configuration

• Filters (Source, Transform, Sink)

• DirectShow .NET (C# wrapper)– http://directshownet.sourceforge.net/

• Learning the filters is key

• Not WPF 3D friendly

DirectShow .NET

• C# wrapper

• Wraps a majority of DirectShow

• Many samples available, along with GraphEdt.exe, and MSDN documentation

• Also has beta wrappers for Windows Media Lib and Windows Media Foundation

Demo

• GraphEdt

• /dshowPlay

• /wpfArBasic - capture

Recognize real-world data

• Computer Vision

• Face tracking

• AR Libraries (fudicial markers)

• GPS

• …

AR Libraries

• ArToolKit

• ArTag

• ArToolKitPlus– http://studierstube.icg.tu-graz.ac.at/

handheld_ar/artoolkitplus.php

• Studierstube Tracker

• Goblin XNA (coming)

ArToolKitPlus

• Optimized for mobile devices

• C++ Lib

• GPL, no longer under development– Studierstube Tracker is the latest

• Not well documented, but ArToolKit is

• Single, Multi-config, Multi-distinct tracking

• Markers (ID, BCH, Patterns)

Videos

• ARTag Magic Lens

• Studierstube Mobile

Demo

• /wpfArBasic – AR tracking

• /wpfArTest

Composite real and virtual data

• GDI

• OpenGL / Tao

• Direct3D

• XNA

• WPF 3D

• …

WPF 3D

• Viewport3D

• MatrixCamera

• Light

• ModelVisual3D / Model3DGroup

• Material

• Transform3D

• HitTest

WPF 3D

• 3D Programming for Windows (Petzold)

• Windows SDK Samples

Demo

• Simple WPF 3D

• /wpfArBasic – 3D model

• /wpfArVideo

Future

• Augmented Reality @ Home (book)– Mostly OpenGL and C++, but at least one C#

and Tao sample. Based on ArTag library.

• Goblin XNA (AR library)– For PC use, not on 360 until XNA is extended

• …

Questions