Date post: | 10-Jan-2017 |
Category: |
Technology |
Upload: | dominik-helleberg |
View: | 188 times |
Download: | 3 times |
with Project Tango
Dominik Helleberg
https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4thttp://bitstars.com/glossary/augm ented-reality/http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG
Augmented Reality
https://www.google.de/trends/explor e#q=Augmented%20Reality&cm pt=q&tz=Etc%2FGMT-1
Augmented Reality
https://www.google.de/trends/explor e#q=Virtual%20reality
Augmented Reality
http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.png
Augmented Reality
https://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality
https://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality
http://www.magicleap.com/#/home
Augmented Reality
https://www.youtube.com/watch?v=kPMHcanq0xM#t= 41
Augmented Reality
https://www.youtube.com/watch?v=CHV9xTGD4Ws
Augmented Reality
http://www.digi-capital.com/news/wp-content/upl oads/2015/04/ARVR-Forecas t.jpg
Augmented Reality
Augmented Reality - Basics
https://en.wikipedia.org/wiki/Augmented_reality
Augmented reality (AR) is a live direct or indirect view of a
physical, real-world environment whose elements are
augmented (or supplemented) by computer-generated
sensory input such as sound, video, graphics or GPS data.
-> It’s important to understand the “context”.
Augmented Reality - Examples
By Rama - http://fr.wikipedia.org/wiki/Image:Hud_on_the_cat.jpg, Public Domain, https://commons.wikimedia.org/w/index.php?curid=129590By Sebastian Klein/Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=3608534
HUD Displays
Context:- Vehicle (Position, Speed, Engine)
Augmented Reality - Examples
http://www.technologybloggers.org/gadgets/google-android-powered-glasses/Von Tim.Reckmann - Eigenes Werk, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=32750308
Weather Forecast(Google Glass)
Context:- Location (coarse)- Time- Weather forecast
Augmented Reality - Examples
https://commons.wikimedia.org/wiki/File:Wikitude_World_Browser_@Salzburg_Old_Town_3.jpg
Wikitude LocationsContext:
- Location (coarse)
- Direction (compass / angle)
- Location of POIs
Augmented Reality - Examples
Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
Virtual objects
Context:- exact, realtime location and
direction of the mobile device
- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Augmented Reality – Visual FX
https://vimeo.com/155092798
Augmented Reality – Virtual Objects
Context:- exact, realtime location and direction of the mobile device- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Augmented Reality – Virtual ObjectsRealtime location / camera chracteristics
https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan
Augmented Reality – VFXRealtime location / camera chracteristics
Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
Augmented Reality – Virtual ObjectsRealtime location / camera chracteristics
Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
Augmented Reality – Virtual ObjectsRealtime location / camera chracteristics
Augmented Reality – Virtual Objects
Context:- exact, realtime location and direction of the mobile device
- camera characteristics
- 3D model of the virtual objects- 3D model of the real objects
- Light model of the real world
Augmented Reality – Virtual Objects3D Model
https://www.blender.org/download/demo-files/ -> BMW Demo by Mike Pan
Augmented Reality – Visual FX3D Model
https://vimeo.com/155092798
Augmented Reality – Virtual Objects
Context:- exact, realtime location and direction of the mobile device
- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Augmented Reality – Virtual Objects3D Model of the real world
Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
Augmented Reality – Virtual Objects3D Model of the real world
https://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality – Virtual Objects3D Model of the real world
https://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality – Virtual Objects3D Model of the real world
https://github.com/stetro/project-tango-poc
Augmented Reality – Virtual Objects3D Model of the real world
https://vimeo.com/155092798
Augmented Reality – Virtual Objects
Context:- exact, realtime location and direction of the mobile device
- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Augmented Reality – Virtual ObjectsLight model
Augmented Reality – Virtual ObjectsLight model
Von Meximex - Used the app and took a picture, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=23501409
Augmented Reality – Virtual ObjectsLight model
https://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality – Virtual ObjectsLight model
http://gurneyjourney .blogspot.de/2012/01/reference-balls.htmlhttp://imgur.com/a/P2Nbp
Augmented Reality – Virtual Objects
Context:- exact, realtime location and direction of the mobile device
- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Project Tango
https://www.google.com/atap/proj ect-tango/
https://www.google.com/atap/proj ect-tango/http://www.androidheadlines.com/2014/02/googl es-project-tango-phone-renders-full-3d-maps-heres-look-like.html
Project Tango
2014Public announcement of
“Project Tango” and release of the “peanut”
prototype
2014 / 2015“Yellowstone” Tablet
released an sold 3000 times
Sept 2016Phab2 Pro
https://www.google.com/atap/proj ect-tango/
Project Tango
• NVIDIA Tegra K1 with 192 CUDA cores
• 7.02” 1920x1200 HD IPS display (323 ppi)
• Dual-band Wi-Fi (2.4GHz/5GHz)
• Bluetooth 4.0
• 4 GB RAM
• Accelerometer
• Ambient light
• Barometer
• Compass
• aGPS
• Gyroscope
https://www.google.com/atap/proj ect-tango/
Project Tango
Infrared Laser3D depth sensing
Motion tracking camera
4 MP 2µm RGB-IR pixel sensor
Project Tango – Motion Tracking
Project Tango – Motion Tracking
Project Tango – Depth Sensing
https://www.google.com/atap/project-tango/about-project-tango/index.html
• Tango’s Depth sensor detects the
range from the device to surfaces within
0.5 – 4 meters
• It generates a point cloud which can be
used to generate a mesh or enhance
AR-Apps
DEMO
Project Tango – Depth Sensing
• BACKUP
Project Tango – Area Learning / indoor navigation
Context:- exact, realtime location and direction of the mobile device
- camera characteristics
- 3D model of the virtual objects
- 3D model of the real objects
- Light model of the real world
Project Tango – Area Learning / indoor navigation
https://www.google.com/atap/project-tango/about-project-tango/index.html
• Tango recognizes or “learns” significant
“markers” of an areas and exports them into an
ADF (“Area Description File”)
• A Tango app may load an ADF file and try to
match visible markers against the ADF file
• After “matching” on an ADF file, Tango can
determine it’s position in local ADF coordinates
https://developers.google.com/proj ect-tango/apis/overview
Project Tango – Getting started
Unreal 4 engine
Project Tango – Getting started JAVA API
Rajawali Framework
Camera Objects / Textures Lights
Rajawali Utils
Tango Java APIs Java
Sup
port
Lib
Tango Service
Sensor Data
Application Logic
Android APIs
https://github.com/Rajaw ali/Rajawali
Project Tango – Getting started JAVA API
https://github.com/Rajaw ali/Rajawali
DirectionalLight light = new DirectionalLight(1, 0.2, -1);light.setColor(1, 1, 1);light.setPower(0.8f);light.setPosition(3, 2, 4);getCurrentScene().addLight(light);
// Set-up a material: green with application of the light and// instructions.Material material = new Material();material.setColor(0xff009900);try {
Texture t = new Texture("instructions", R.drawable.instructions);material.addTexture(t);
} catch (ATexture.TextureException e) {e.printStackTrace();
}material.setColorInfluence(0.1f);material.enableLighting(true);material.setDiffuseMethod(new DiffuseMethod.Lambert());
// Build a Cube and place it initially in the origin.mObject = new Cube(CUBE_SIDE_LENGTH);mObject.setMaterial(material);mObject.setPosition(0, 0, -3);mObject.setRotation(Vector3.Axis.Z, 180);getCurrentScene().addChild(mObject);
Project Tango – Getting started AR Scene
Project Tango – Getting started AR Scene
public class ARActivity extends Activity {
@Bind(R.id.gl_main_surface_view)RajawaliSurfaceView mainSurfaceView;
protected void onCreate(Bundle savedInstanceState) {
pointCloudManager = new TangoPointCloudManager();renderer = new SceneRenderer(this);
mainSurfaceView.setSurfaceRenderer(renderer);
...
}
Project Tango – Getting started JAVA API
Project Tango – Getting started JAVA API
@Overrideprotected void onResume() {
tango = new Tango(ARActivity.this, new Runnable() {
@Overridepublic void run() {
try {connectTango();connectRenderer();isConnected = true;
} catch (TangoOutOfDateException e) { }}
});
Project Tango – Getting started JAVA API
private void connectTango() {
TangoConfig config = tango.getConfig(TangoConfig.CONFIG_TYPE_DEFAULT);
config.putBoolean(TangoConfig.KEY_BOOLEAN_LOWLATENCYIMUINTEGRATION, true);config.putBoolean(TangoConfig.KEY_BOOLEAN_DEPTH, true);config.putBoolean(TangoConfig.KEY_BOOLEAN_COLORCAMERA, true);
tango.connect(config);
...
Project Tango – Getting started JAVA API
tango.connectListener(framePairs, new OnTangoUpdateListener() {@Overridepublic void onPoseAvailable(TangoPoseData pose) { }
@Overridepublic void onFrameAvailable(int cameraId) {
if (cameraId == TangoCameraIntrinsics.TANGO_CAMERA_COLOR) {isFrameAvailableTangoThread.set(true);mainSurfaceView.requestRender();
}}
@Overridepublic void onXyzIjAvailable(TangoXyzIjData xyzIj) {
pointCloudManager.updateXyzIj(xyzIj);}
Project Tango – Getting started JAVA API
private void connectRenderer() {
renderer.getCurrentScene().registerFrameCallback(new ASceneFrameCallback() {
@Overridepublic void onPreFrame(long sceneTime, double deltaTime) {
...tango.connectTextureId(
TangoCameraIntrinsics.TANGO_CAMERA_COLOR,renderer.getTextureId());
Project Tango – Getting started JAVA API
if (isFrameAvailableTangoThread.compareAndSet(true, false)) {
mRgbTimestampGlThread =tango.updateTexture(TangoCameraIntrinsics.TANGO_CAMERA_COLOR);
}
TangoPoseData lastFramePose = tango.getPoseAtTime(mRgbTimestampGlThread, FRAME_PAIR);
if (lastFramePose.statusCode == TangoPoseData.POSE_VALID) {// Update the camera pose from the rendererrenderer.updateRenderCameraPose(lastFramePose, mExtrinsics);mCameraPoseTimestamp = lastFramePose.timestamp;
}
Project Tango – Getting started JAVA API
public class SceneRenderer extends RajawaliRenderer {@Overrideprotected void initScene() {
ScreenQuad backgroundQuad = new ScreenQuad();
Material tangoCameraMaterial = new Material();tangoCameraMaterial.setColorInfluence(0);tangoCameraTexture =
new StreamingTexture("camera",(StreamingTexture.ISurfaceListener) null);
tangoCameraMaterial.addTexture(mTangoCameraTexture);backgroundQuad.setMaterial(tangoCameraMaterial);
getCurrentScene().addChildAt(backgroundQuad, 0);
Project Tango – Getting started JAVA API
public void updateRenderCameraPose(TangoPoseData devicePose, DeviceExtrinsics extrinsics) {
Pose cameraPose = ScenePoseCalculator.toOpenGlCameraPose(devicePose, extrinsics);
getCurrentCamera().setRotation(cameraPose.getOrientation());getCurrentCamera().setPosition(cameraPose.getPosition());
}
Augmented Reality with Tango
DEMOs
Augmented Reality - Conclusion
http://mscorpnews.blob.core.windows.net/ncmedia/2015/01/Microsoft-HoloLens-MixedWorld-RGB.pnghttps://news.microsoft.com/microsoft-hololens-press-materials/
Augmented Reality - Conclusion
https://de.wikipedia.org/wiki/Erweiterte_Realit%C3%A4thttp://bitstars.com/glossary/augm ented-reality/http://www.augmentedplanet.com/2010/04/get-real-redefining-augmented-reality-games/ardronei phone.JPG
Context:- exact, realtime location and direction of the mobile device 👍
- camera characteristics 👍
- 3D model of the virtual objects 👍
- 3D model of the real objects
- Light model of the real world 👎
Project Tango
Project Tango – Lessons Learned
Lessons learned- Point Cloud calculations are hard
- 3D is tricky sometimes
- which API to choose?
- Motion Tracking works really well
- ”closed” APIs make life hard
- Environments change more then we recognize
- Automated Testing is nearly impossible / debugging is tricky
It is a LOT of fun….
Augmented Reality - Demos
http://mashable.com/2016/02/22/project-tango-indoor-navigation/#xOEcMWsfpZqK
Augmented Reality - Demos
Augmented Reality - Demos
Project Tango
Questions?