BUILDING BUSINESS APPLICATIONS WITH HOLOLENS

AND CLOUD

“HoloLens Evangelist”

I am responsible for architecture related business for our project services

department at ETTU. I’m involved as architect at different companies like Jumbo

Maritime, PGGM, Gemeente Amersfoort and others

Alexander Meijers

ABOUT ME Solutions Architect for ETTU

Founder of the Mixed Reality User Group

MixUG: http://www.mixug.nl

Email: a.meijers@ettu.nl or alexander@appzinside.com

Twitter: @ameijers

Blog: http://www.appzinside.com

5INTRO ETTU

TODAY’S TALKAGENDA Realities explained

Microsoft HoloLens

Fields of Application

HoloLens capabilities Gestures

Sound

Spatial understanding

Local storage

Cloud Data

Connectivity

User Experience

Security

Slider

Content Ads

REALITIES EXPLAINED

HOLOLENS & CLOUD 7

REALITIESEXPLAINED

Merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interactin real time.

An overlay of synthetic content on the real world that is anchored to and interacts with the real world

MIXED REALITY

Direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data

An overlay of content on the real world where that content is not anchored to or part of it

AUGMENTED REALITY

Generation of realistic images, sounds and other sensations that replicate a real environment or create an imaginary setting

An immersive experience created entirely from computer-generatedContent. Also similar to 360 degree video

VIRTUAL REALITY

HOLOLENS & CLOUD 8

EVERYTHINGBECOMES MIXED REALITY

Virtual Reality devices SteamVR AltSpaceVR October/November 2017

IMMERSIVE HEADSETS

Mixed Reality Device Developer and Commercial Suite

version Since October 2016

HOLOLENS2019

HoloLens v3

MICROSOFT HOLOLENS

HOLOLENS & CLOUD 11

MICROSOFT HOLOLENS

First Holographic computer running on Windows 10

Contains a CPU, GPU and HPU

Human understanding

Gesture and voice input

Gaze tracking

Spatial sound

Uses wi-fi to connect and has no wires attached

Design for comfort

HOLOLENS & CLOUD 12

SENSORS, OPTICS ANDSPEAKERS

Environment camera

Depthcamera

Videocamera

Spatial SoundWith

speakers

High definitionlenses

“DEMONSTRATION OF HOLOLENSAPPLICATIONS”

“Holograms”

HOLOLENS & CLOUD 15

MICROSOFT HOLOLENSEDITIONS

Kiosk mode Limiting running apps to enable demo or showcase experiences

MDM Manage Multiple HoloLens devices through solutions like Microsoft

Intune

Business Store Portal Private enterprise store for your companies apps

Secure boot Only boot software which is trusted for your organization

Windows Update for Business Controlled OS updates to device

FIELDS OF APPLICATION

HOLOLENS & CLOUD 17

FIELDS OF APPLICATIONHoloLens

Enables the user to visualize

surrounding which is not

directly visible

Get visual insight of an

object or model

Modelling of non-existing

objects

VISUALIZATION

Enrich real-world

environment objects with

information

Show information as part of

the real-world on a wall or

table

Show additional information

to objects, persons or

activities

PROVISIONING

OF INFORMATION

Support through linked

view, voice and gestures

Meetings around

Holographic objects

Working together on

projects whereby

visualization is an important

part

Working together when

distance is involved

WORKING

TOGETHER

Holographic media

Enrich media with

additional holographic

objects

MEDIA AND

ENTERTAINMENT

Experience learning

Getting more realistic

teaching by moving 2D

presentations in books to

3D holographic models

EDUCATION

Play games as part of your

real-world environment

Game interaction with your

walls, ceiling and floor

GAMING

HOLOLENS & CLOUD 18

HOLOLENS EXAMPLES

HOLOLENS & CLOUD 19

HOLOLENS EXAMPLES

HOLOLENS & CLOUD 20

HOLOLENS EXAMPLES

HOLOLENS & CLOUD 21

HOLOLENS EXAMPLES

HOLOLENS & CLOUD 22

HOLOLENS EXAMPLES

HOLOLENS & CLOUD 23

HOLOLENS EXAMPLES

“DEMONSTRATION OF SPATIAL DESIGN CONCEPT”

“HoloLens app for creating neighbourhood facilities”

INPUT CONTROLS

HOLOLENS & CLOUD 26

INPUT CONTROLSGAZE

The direction of the HoloLens pointing is called the gaze

In principle the direction in which you look

Hits an object or part of the environment

Used for selecting or pointing

HOLOLENS & CLOUD 27

INPUT CONTROLSGESTURES

Recognize input by tracking the position of either or both hands within the gesture frame

Bloom gesture

Click gesture Press and release

Click gesture with hold Manipulation

Navigation

HOLOLENS & CLOUD 28

INPUT CONTROLSSPEECH

Use words or sentences to control your environment

English language only at the moment

Hey Cortana

SPATIAL SOUND

HOLOLENS & CLOUD 30

SPATIAL SOUNDPOSSIBILITIES

Simulates 3D sound using direction, distance and environmental simulations

Above, below, behind, to the side, etc.

How?

Gaze Mixing Attach sound to holographic objects Works also when object is not in line of sight Used to draw attention

Audio Haptics Selecting an object Hands within the gesture frame

Immersion Ambient sounds surrounding the user Background music

SPATIAL UNDERSTANDING

HOLOLENS & CLOUD 32

SPATIAL UNDERSTANDINGPERCEPTION

Spatial perception and mapping

Rooms

Different ways of detecting and working with spatial perception

Meshes & planes

Spatial perception is the ability to be aware of

your relationships with the environment around

you and with yourself

HOLOLENS & CLOUD 33

SPATIAL UNDERSTANDINGPERCEPTION

Device scans your environment and builds a digital model in real time

It allows HoloLens to see different surfaces, walls and ceiling

Possible to simulate a physical space E.g. Projecting a terain over your floor

“DEMONSTRATION OF SPATIAL MAPPINGHOLOGRAM”

“Spatial Understanding”

OTHER TECHNOLOGIES

HOLOLENS & CLOUD 36

TECHNOLOGIESVUFORIA

Allows you to develop holographic apps which recognize specific things in the environment and attach experiences to them

Recognizes specific images and objects in the environment

Connects holograms to targets

Vuforia engine Extend the capabilities of your holographic app

Targets can be stored on Device or in the Cloud

HOLOLENS & CLOUD 37

TECHNOLOGIESSHARING EXPERIENCES

Two or more people are experiencing the same holographic objects

Accomplished by anchor sharing

Use cloud to send anchors over to other devices

CLOUD & DATA

HOLOLENS & CLOUD 39

CLOUD & DATA

The workforce in your application is defined by the

amount and type of data which is transferred from

and to the HoloLens application

DATA

Use of certain data, latency when connecting

services and security will influence the user

experience

USER EXPERIENCE

Location and preferences of your customer defines

the ability of your HoloLens application to connect

to services and is divided into different scenarios

CONNECTIVITY

Data transferred need to be secured depending on

the type of data. Also authentication differs from

online services and available customer security.

SECURITY

APPLICATION REQUIREMENTS

COSTS

HOLOLENS & CLOUD 40

APPLICATIONREQUIREMENTS

What is the goal of your application?

Are you storing any data? Storage on device

Storage in the Cloud

What type of data are you storing? Common application data

Holograph objects

Sensory data

Are you depending on any (external) data?

Need the data to be exposed to other services or applications?

Where is the device used?

What kind of access do we have with external sources? Continuously

Hybrid

Offline

What is the impact on user experience of HoloLens users? Security

Latency with connected sources

HOLOLENS & CLOUD 41

DATALOCAL STORAGE

Only allowed to store files to a number of local folders

Save or open from another App like OneDrive

HoloLens does not support connecting external hard drives or SD Cards

There is no File Explorer application You could use the Windows Device Portal

HOLOLENS & CLOUD 42

DATA SERVICESCONNECTING DATA TO HOLOLENS

Online Services Bing

Other third-party services

Azure Services Custom services

Bots / AI

Azure Cognitive Services

Office 365 Microsoft Graph

SharePoint Online

Special Vuforia Cloud Recognition Service

“DEMONSTRATION OF HOLOMARKER”

“HoloLens app using SharePoint Online as registration service for tagging improvements”

HOLOLENS & CLOUD 44

CONNECTIVITYCONNECTION SCENARIOS

Preloading data to device and

post saving data to Cloud

No internet whatsoever

Keep application small by

getting only relevant data

OFFLINE

Connection not always present

or very low bandwidth

Application runs in different

environments with different

connectivity

Caching mechanism for

availability of data and

performance

HYBRID

Continuously connected

Get only data when needed

Caching mechanism for

performance

ONLINE

HOLOLENS & CLOUD 45

CLOUD & DATASECURITY CONSIDERATIONS

Do you need to secure your content? Building services

Is your content available for other applications?

Is your content anonymous available for the big world? E.g. a service which delivers certain data from your organization or type

of services

GDPR is not an illusion, data need to be secured depending on the content

HOLOLENS & CLOUD 46

CLOUD & DATAIDENTITY & AUTHENTICATION

Anonymous External Providers(Facebook, Twitter, Google, etc.)

Azure Active Directory Custom Providers Keys

Online services X X X X X

Office 365Microsoft Graph

X

Azure Services(AI/Bots/Custom)

X X X

Office 365SharePoint Online

X

Azure Cognitive Services X

Vuforia Cloud Recognition Service

X

“DEMONSTRATION OF OFFICE SKETCH”

“HoloLens app accessing Microsoft Graph”

HOLOLENS & CLOUD 48

USER EXPERIENCEUSER PERSPECTIVE

Ensure the application is comfortable to the user

Avoid abrupt movements

Allow them to see the world in the background

Animation from down, left or right or fade in

Move content slowly and smoothly toward the user

Keep a minimal distance of 3 meters

Design your application from an user’s unique point of view Determine if the user moves around, user is sitting down or standing

The user is the camera! Let them control the movement

HOLOLENS & CLOUD 49

USER EXPERIENCEHOLOGRAPHIC FRAME

Use the center of the holographic frame for your holographic objects

Help the user through guideness Arrows pointing to a specific location

Light trails

Pointers

Spatial sound

Voice prompts

Make sure your holograms fit the frame and be part of the real-life environment

They will feel natural

HOLOLENS & CLOUD 50

USER EXPERIENCEAPPLICATION

Make sure that your application uses background processes to prevent latency

Only load data which you actually need

Keep traffic to a minimum when accessing services

Keep framerates up to 60 fps

Use optimized assets specifically build for HoloLens

“DEMONSTRATION OF HOLOMAPS”

“HoloLens app using Bing as a service”

HOLOLENS & CLOUD 52

CLOUD & DATACOSTS

Devices Number of devices

Development or Commercial Suite edition

Additional hardware / software Sensors to enhance application or collect data

Embedded software/hardware

Additional cost for solution development to support hybrid and offline scenarios

Holograms

External services Data storage

Retrieval of data

Use of a services

WRAP-UP

HOLOLENS & CLOUD 54

WRAP-UPTAKEAWAYS

HoloLens is at the moment the best mixed reality device forbuilding applications for small to large business

The inbuild gestures, speech, spatial sound and spatialmapping give the user that true immersive experience forbuilding business applications

Consider application requirements and the cost

When building applications keep the four in mind; Data, Connectivity, User Experience and Security

THANKS FOR WATCHING!

SharePoint Unite

Kampenringweg 45b, Gouda

0182-686 000

a.meijers@ettu.nl / alexander@appzinside.com

Contact me:

facebook.com/alexandermeijers.5

@ameijers

Follow me on: