CGT 101Recent Advances in CGBedrich Benes, Ph.D.Purdue UniversityDepartment of Computer Graphics Technology andDepartment of Computer Science
© Bedrich Benes
CG and Other Fields• Comp. Graphics describes the world visually• Aims at perceived visual quality
(not ONLY realism, also art)
• Computer vision attempts to reconstruct world (world computer model)
• Image processing works with images
© Bedrich Benes
World Description
CG and Other Fields
Geometry
ImageLight
MaterialsTexturesMaterialsTextures
RenderingRendering
ModelingModeling
LightingLighting
Image Processing
Image Processing
Computer GraphicsComputer Graphics
Reconstruction and Computer Vision © Bedrich Benes
CG and Other FieldsExamples: • Computer Vision:
• 3D geometry reconstruction from images• Texture capture• Scanning 3D world• Image understanding
© Bedrich Benes
CG and Other FieldsExamples: • Image processing
• Image compression • Video codecs• Image inpainting• Computational cameras
© Bedrich Benes
Recent Advances
• Virtual Reality• Augmented Reality• Non‐Photorealistic Rendering• Artificial Intelligence
© Bedrich Benes
Virtual Reality
© Bedrich Benes
Scene
The Rendering ProcessWorld Description - Scene
Geometry
Light
MaterialsTextures
ModelingModeling
LightingLighting
World Description - Scene
Geometry
Light
MaterialsTextures
Modeling
Lighting
ImageCamera model
© Bedrich Benes
Virtual RealityLeft image
Right image
Scene
Left camera
Right camera
Left eye
Right eye
© Bedrich Benes
Virtual Reality• The rendering process is doubled• There are two cameras
approx. distance of the human eyes apart• They render two images• The images are displayed to left and right eye• The human brain perceives it as 3D• The viewers is immersed
into the virtual environment
© Bedrich Benes
Virtual Reality• It needs precise head tracking
• Head movement is transferred to the dual‐camera movement
• The environment is simulatedall is synthetic
© Bedrich Benes
Virtual Reality• How do we deliver the data to each eye?• …by using special displays
• Shutter Glasses
• Head Mounted Displays (HMD)
© Bedrich Benes
Shutter Glasses• They turn on and off the display for
each eye at high frequency• A specialized display displays
the left and the right images in fast sequence
• It needs to be synchronized
https://en.m.wikipedia.org/wiki/Active_shutter_3D_system
© Bedrich Benes
Shutter Glasses• Nvidia 3D Vision (2008)
• USB‐support• Stereoscopic gaming kit
© NVidia
© Bedrich Benes
Shutter Glasses• The Cave
• Four large active displays• Active shutter glasses• Can move the screen walls
L, U, Flat
© Visibox Inc
© Purdue University
© Bedrich Benes
Shutter Glasses• Challenges
• Limited view (the screen)
• Limited motion
• Limited field of view
© Bedrich Benes
Head Mounted Displays• Double display • Worn on head• Precise head tracking• Good sound• Some support eye‐tracking
© Oculus
© Bedrich Benes
Head Mounted Displays• Challenges
• Motion sickness
• Heavy (it is getting better, but…)
• Needs huge computational power
© Bedrich Benes
VR• VR Challenges in General
• Lag • There is a delay between the movement and display
• Limited field of view• We perceive in the corners of our eyes too
• Full immersion is only visual• You cannot feel the weight of the objects.
• And all the imprecisions • Colors, sound, tracking…
© Bedrich Benes
Augmented Reality
© Bedrich Benes
Augmented Reality• AR displays additional data in real world
• Usually an image in the real world
• Provides some additional insight
• A popular ones are real‐time translators
© Bedrich Benes
Augmented Reality
© Bentley https://www.youtube.com/watch?v=5HV3fcTvZk0
© Bedrich Benes
Augmented Reality
© Pepsi https://www.youtube.com/watch?v=Go9rf9GmYpM © Bedrich Benes
Augmented Reality
© Microsoft
• Technology• Head mounted
transparent displayse.g., Microsoft HoloLens
• Cellphones
© BGR.com © macworld.com
© Bedrich Benes
Augmented Reality• Challenges
• 3D positioning precisiondepends on sensors and cameras
• Speed of renderingdepends on the CPU/GPU
• Data availabledepends on the internet connection
© Bedrich Benes
Non‐photorealistic rendering
© Bedrich Benes
Non‐photorealistic Rendering• Usually we aim for photorealism• Always we aim for fast displaying
• In some contexts non‐photorealism is better
• Stylized rendering• Artistic rendering
© Bedrich Benes
NPR• Toon shading
displays objects similar to cartoon animation(strong color changes, silhouettes)
© Bedrich Benes
NPR• Artistic rendering
attempts to simulate artistic style
© Bedrich Benes
NPR• Sketch‐based rendering
simulates human sketches
Interactive Line Art Rendering of Freeform Surfaces. Computer Graphics forum, Vol 18, No 3, pp 1-12, September 1999.
© Bedrich Benes
NPR• Illustrative visualization of volumetric data
• Needs smart processing of volumesand feature extraction
© Anna Vilanova© Bedrich Benes
NPR• Nowadays
Strongly supported by deep learning
• Many of these approaches are image processing techniquesSupported in image processing softwaree.g., Adobe Photoshop
© Bedrich Benes
NPR
© Bedrich Benes
Deep Learning
© Bedrich Benes
Deep Learning• DL belongs to machine learning (ML)
and ML belongs to artificial intelligence (AI)
• Game changing technology• DL
• Works on large datasets (usually)• Requires huge computational power (usually)• Provides end‐to‐end solutions
© Bedrich Benes
Deep Learning• Traditional ML creates rules for decisions
e.g., if is it round, it is a circle
• DL uses massive datasets to find answers• Show it 10,000 cats and it will recognize cat
© Bedrich Benes
Deep Learning• In the core is a deep neural network
© M. Mitchell Waldrop © Bedrich Benes
Deep LearningIt is a two step process
1) Training
2) Inference
© Bedrich Benes
Deep Learning1) TrainingRequires a DL (with certain architecture)Requires initial weights of neuronsRequires large dataset of labelled data
© Bedrich Benes
Deep Learning1) Training
zero
one
nine
Training data
© MINST dataset
Labels
© Bedrich Benes
Deep Learning1) Training
The DL is shown batches of pairs [label, data]DL assigns its weights by using back propagation algorithmFor large datasets this can take daysGPUs are commonly used for this task
Cat
© Bedrich Benes
Deep Learning2) Inference
The DL is shown an unknown data setIt will assign it a label
Inference is very fast (in orders of milliseconds)
Cat
© Bedrich Benes
Deep Learning• Why it all works?• The DNN generalizes the data• Each layer holds higher level of abstraction
• Overfitting problem – it can “memorize” the input data and will not generalize at all(if you memorize the homework you will fail the exam)
© Bedrich Benes
Deep Learning• Convolutional NN
• Image classification• Image segmentation
• Recurrent neural networks • sequences in time• Translators
© Bedrich Benes
Deep Learning• Generative Adversarial Networks
• Can complete images
© Phillip Isola et al.
https://www.youtube.com/watch?v=5w685udM838&feature=youtu.be© Bedrich Benes
Deep Learning ‐ Reinforcement Learning
© www.openAI.comhttps://www.youtube.com/watch?v=kopoLzvh5jY
© Bedrich Benes
Food for thought…
© https://autonomousweapons.org/https://www.youtube.com/watch?v=TlO2gcs1YvM © Bedrich Benes
Use what you have learned to make the world a better place