Post on 03-Apr-2018
transcript
7/28/2019 Computer Graphics & Computer Vision
1/49
Page 1
Computer Graphics and Computer Vision
Professor Demetri Terzopoulos
Graphics and Vision: A Unified ViewGraphics and Vision: A Unified ViewThis major field is about Computers and ImagesThis major field is about Computers and Images
Computer Graphics (CG)Computer Graphics (CG)
Computational models images and videosComputational models images and videos
SynthesisSynthesis
Forward problemForward problem
Computer Vision (CV)Computer Vision (CV)
Images and videos computational modelsImages and videos computational models
AnalysisAnalysis
Inverse problemInverse problem
7/28/2019 Computer Graphics & Computer Vision
2/49
Page 2
History of Computer Graphics and Computer VisionHistory of Computer Graphics and Computer Vision
Two PhD projects at MIT in the early 1960sTwo PhD projects at MIT in the early 1960s Ivan E. Sutherland, 1963Ivan E. Sutherland, 1963
Sketchpad, a manSketchpad, a man--machine graphical communication systemmachine graphical communication system
Lawrence G. Roberts,Lawrence G. Roberts, 19631963
Machine perception of threeMachine perception of three--dimensional solidsdimensional solids
CG and CV have developed as independent fieldsCG and CV have developed as independent fields
,,
Exciting synergies and crossExciting synergies and cross--fertilization with other fieldsfertilization with other fields Physics, biology, cognitive science, artificial intelligence, art, .Physics, biology, cognitive science, artificial intelligence, art, .
What is an Image / Video?What is an Image / Video?Array of pixels (Array of pixels (oneone orormmoorree numbers)numbers)
A video is a time sequence of imagesA video is a time sequence of images
How they are formed:How they are formed:
Objects in the world (static or dynamic)Objects in the world (static or dynamic)
Illumination (light sources)Illumination (light sources)
Imaging device (eye, camera)Imaging device (eye, camera)
Illumination
Camera
We want to synthesize and analyze themWe want to synthesize and analyze them
7/28/2019 Computer Graphics & Computer Vision
3/49
Page 3
Why? Images and movies are everywhere!Why? Images and movies are everywhere! Entertainment: Motion pictures & GamesEntertainment: Motion pictures & Games
Virtual worlds (especially for movies and games)Virtual worlds (especially for movies and games)
Industrial designIndustrial design
Scientific and medical visualizationScientific and medical visualization
HumanHuman--computer interactioncomputer interaction
Fine artsFine arts
Etc.Etc.
RoboticsRobotics
AutomotiveAutomotive
Visual Surveillance / BiometricsVisual Surveillance / Biometrics
Industrial inspectionIndustrial inspection
Medical imagingMedical imaging
Remote sensingRemote sensing
Image and video retreivalImage and video retreival Etc.Etc.
For a deeper understanding of the physical world and living systems, including the human brainFor a deeper understanding of the physical world and living systems, including the human brain
Computer GraphicsComputer GraphicsThe art and science of creating imagery by computerThe art and science of creating imagery by computer
Three main research themesThree main research themes
ModelingModeling
How do we model (mathematically represent) objects?How do we model (mathematically represent) objects?
How do we construct models of specific objects?How do we construct models of specific objects?
AnimationAnimation
How do we represent the motions of objects?How do we represent the motions of objects?
How do we ive animators control of this motion?How do we ive animators control of this motion?
RenderingRendering
How do we simulate the realHow do we simulate the real--world behavior of light?world behavior of light?
How do we simulate the formation of images?How do we simulate the formation of images?
7/28/2019 Computer Graphics & Computer Vision
4/49
Page 4
Standard Display DevicesStandard Display Devices
PlasmaLCD
(Liquid Crystal Display)
CRT
(Cathode Ray Tube)
Exotic Display DevicesExotic Display Devices
ImmersiveImmersiveImmersiveImmersive HeadHead--MountedMountedHeadHead--MountedMounted
AutostereoscopicAutostereoscopicAutostereoscopicAutostereoscopic VolumetricVolumetricVolumetricVolumetric
7/28/2019 Computer Graphics & Computer Vision
5/49
Page 5
MoviesMovies
To reality and beyond !To reality and beyond !
MoviesMoviesSpecial effectsSpecial effects
7/28/2019 Computer Graphics & Computer Vision
6/49
Page 6
MoviesMovies
Digital CompositingDigital Compositing
7/28/2019 Computer Graphics & Computer Vision
7/49
Page 7
Digital compositingDigital compositing
Digital CompositingDigital Compositing
7/28/2019 Computer Graphics & Computer Vision
8/49
Page 8
CartoonsCartoons
GamesGamesFocus on interactivityFocus on interactivity
7/28/2019 Computer Graphics & Computer Vision
9/49
Page 9
GamesGames
Metal GearMetal Gear
Solid 4Solid 4
7/28/2019 Computer Graphics & Computer Vision
10/49
Page 10
ComputerComputer--Aided DesignAided Design
Precision modelingPrecision modeling
Engineering visualizationEngineering visualization
ComputerComputer--Aided DesignAided DesignIts not just about visualizationIts not just about visualization
Simulation is also usefulSimulation is also useful
7/28/2019 Computer Graphics & Computer Vision
11/49
Page 11
Visualization: ScientificVisualization: Scientific
Molecular Biology
Visualization: MedicalVisualization: Medical
7/28/2019 Computer Graphics & Computer Vision
12/49
Page 12
Visualization: ArchitecturalVisualization: Architectural
http://www.diamondschmitt.com/
Visualization: InfoVisualization: Info
Smith and Fiore
7/28/2019 Computer Graphics & Computer Vision
13/49
Page 13
Graphical User InterfacesGraphical User Interfaces
3D User Interfaces3D User Interfaces
Steven Schkolne
Sculpture Art
7/28/2019 Computer Graphics & Computer Vision
14/49
Page 14
Digital ArtDigital Art
Genetically evolvedGenetically evolved
Carl Sims
Digital ArtDigital Art
The Highest-Grossing Film of All Time, Jason Salavon^2nd
7/28/2019 Computer Graphics & Computer Vision
15/49
Page 15
Elements of CGElements of CG
The graphics pipelineThe graphics pipeline
Modeling Animation Rendering
ModelingModelingRepresenting objects geometrically on a computerRepresenting objects geometrically on a computer
Polygon meshes
Texture maps
ur ace pa c es
Parametric curves
7/28/2019 Computer Graphics & Computer Vision
16/49
Page 16
ModelingModeling
Alternative representationsAlternative representations
Voxels
Blobs
Subdivision surfaces
ModelingModelingAltering geometric modelsAltering geometric models
Affine transform
Ying, Kristjansson, Biermann, Zorin
7/28/2019 Computer Graphics & Computer Vision
17/49
Page 17
Scanning ShapesScanning Shapes
Digital Michaelangelo Project
Plant ModelingPlant ModelingP. Prusinkiewicz, Univ. of CalgaryP. Prusinkiewicz, Univ. of Calgary
7/28/2019 Computer Graphics & Computer Vision
18/49
Page 18
Plant ModelingPlant Modeling
RenderingRenderingKey elementsKey elements
Illumination
Geometric Shape& Reflectance
Camera
Pixels
7/28/2019 Computer Graphics & Computer Vision
19/49
Page 19
Camera ModelCamera Model
RenderingRenderingDraw visible surfaces onto displayDraw visible surfaces onto display
Camera(view direction)
ImagePlane
Clipping ScanConversion
Camera(eye position)
7/28/2019 Computer Graphics & Computer Vision
20/49
Page 20
Reflectance ModelingReflectance Modeling
Reflectance ModelingReflectance Modeling
Diana the Huntress, Henrik Wann Jensen
7/28/2019 Computer Graphics & Computer Vision
21/49
Page 21
Subsurface ScatteringSubsurface Scattering
Translucency and varied levels of light penetration can be created usingTranslucency and varied levels of light penetration can be created usingsubsurface scattering effects (nVIDIA)subsurface scattering effects (nVIDIA)
RenderingRenderingNonNon--photorealistic renderingphotorealistic rendering
Aaron Hertzmann
7/28/2019 Computer Graphics & Computer Vision
22/49
Page 22
Rendering: InformationRendering: Information
Agrawala and Stolte Mapblast.com
AnimationAnimationKeyframe animationKeyframe animation
Keyframing
7/28/2019 Computer Graphics & Computer Vision
23/49
Page 23
AnimationAnimation
Motion captureMotion capture
Animating Golem in LOTRAnimating Golem in LOTR
7/28/2019 Computer Graphics & Computer Vision
24/49
Page 24
Markerless Motion CaptureMarkerless Motion Capture
Max Plank Institute of Informatics, GermanyMax Plank Institute of Informatics, Germany
AnimationAnimationPhysical simulationPhysical simulation
Dynamics R. Fedkiw et al., Stanford U.
7/28/2019 Computer Graphics & Computer Vision
25/49
Page 25
Atlanta in MotionAtlanta in Motion
J. Hodgins, et al.,J. Hodgins, et al.,Georgia TechGeorgia Tech
NBCNBC --Los AngelesLos AngelesCoverage of Prof. Petros Faloutsos digital stuntmanCoverage of Prof. Petros Faloutsos digital stuntman
7/28/2019 Computer Graphics & Computer Vision
26/49
Page 26
Falling SidewaysFalling Sideways
Falling Backward, Rolling Over, Rising, andFalling Backward, Rolling Over, Rising, and
Balancing in GravityBalancing in Gravity
and Iand I cancan get up!get up! Help, Ive fallen! Help, Ive fallen!
7/28/2019 Computer Graphics & Computer Vision
27/49
Page 27
Rising From a Supine PositionRising From a Supine Position
Virtual CelebrityVirtual CelebrityVirtual Celebrity Productions, LLCVirtual Celebrity Productions, LLC
7/28/2019 Computer Graphics & Computer Vision
28/49
Page 28
FacialFacialAnatomyAnatomy
Skin Model
Muscle Model
Epidermis
Dermis
Muscle Layer
RealReal--Time Facial SimulationTime Facial Simulation
7/28/2019 Computer Graphics & Computer Vision
29/49
Page 29
NeckNeck--HeadHead--Face AnimationFace Animation
AnimationAnimationBehavioral animationBehavioral animation
7/28/2019 Computer Graphics & Computer Vision
30/49
Page 30
Evolved Virtual CreaturesEvolved Virtual Creatures
Karl Sims, 1994Karl Sims, 1994
Genetic algorithmsGenetic algorithms
Mimic biological evolutionMimic biological evolution
AnimationAnimation
Pixar: Luxo Jr. (1986)Pixar: Luxo Jr. (1986)
7/28/2019 Computer Graphics & Computer Vision
31/49
Page 31
AnimationAnimation
Example: Geris GameExample: Geris Game -- PixarPixar
AnimationAnimationThe AnimatrixThe Animatrix
Final Flight of the OsirisFinal Flight of the Osiris
7/28/2019 Computer Graphics & Computer Vision
32/49
Page 32
Reality is *Very* ComplexReality is *Very* Complex
Virtual RealityVirtual RealityArtificial life and other natural phenomenaArtificial life and other natural phenomena
Discuss laterDiscuss later
7/28/2019 Computer Graphics & Computer Vision
33/49
Page 33
Many Open Research ProblemsMany Open Research Problems
How to model/render/animate a complex scene?How to model/render/animate a complex scene?
Computer VisionComputer VisionRelated fieldsRelated fields
Image processingImage processing
Pattern recognitionPattern recognition
Visual perceptionVisual perception
Image understandingImage understanding
True IU seems to involve a great deal of human intelligenceTrue IU seems to involve a great deal of human intelligence
Automated s stems are still far from human erformanceAutomated s stems are still far from human erformance
Some good solutions in constrained special casesSome good solutions in constrained special cases
(e.g., inspection: IC manufacturing, circuit boards)(e.g., inspection: IC manufacturing, circuit boards)
Inverse problems are generally tougher to solveInverse problems are generally tougher to solve
7/28/2019 Computer Graphics & Computer Vision
34/49
Page 34
WHAT DO YOU SEE?WHAT DO YOU SEE?
AND NOW?AND NOW?
7/28/2019 Computer Graphics & Computer Vision
35/49
Page 35
Vision Research ThemesVision Research Themes
Edge and region extractionEdge and region extraction
Image segmentationImage segmentation
Shape from XShape from X
ContoursContours
ShadingShading
StereoStereo
MotionMotion
Object trackingObject tracking
Object recognition (including faces)Object recognition (including faces) Event and activity recognitionEvent and activity recognition
Interactive Image Analysis Using SnakesInteractive Image Analysis Using Snakes
7/28/2019 Computer Graphics & Computer Vision
36/49
Page 36
Retinal Angiogram SegmentationRetinal Angiogram Segmentation
Cartographic Modeling: RiversCartographic Modeling: Rivers
P. Fua. Fast, Accurate and Consistent Modeling of Drainage andSurrounding Terrain. International Journal of Computer Vision, 1998.
7/28/2019 Computer Graphics & Computer Vision
37/49
Page 37
Cartographic Modeling: Buildings and RoadsCartographic Modeling: Buildings and Roads
Automated Image RetreivalAutomated Image Retreival
Find pictures of theFind pictures of the
Sydney operaSydney opera
househouse
in a large databasein a large database
on the webon the web
7/28/2019 Computer Graphics & Computer Vision
38/49
Page 38
Space Robotics: Mars RoverSpace Robotics: Mars Rover
SelfSelf--Driving Cars: Intelligent TransportationDriving Cars: Intelligent TransportationProfessor Stefano SoattoProfessor Stefano Soatto
7/28/2019 Computer Graphics & Computer Vision
39/49
Page 39
Visual ServoingVisual Servoing
Tom Drummond, Univ. of Cambridge
ModelModel--Based Heart Motion TrackingBased Heart Motion Tracking
7/28/2019 Computer Graphics & Computer Vision
40/49
Page 40
3D Reconstruction3D Reconstruction
From image(s) to model(s)From image(s) to model(s) Inverting 3D projectionInverting 3D projection
ImageImage--based modelingbased modeling
Viewpoint
Image
3D model
Cooking with KurtCooking with KurtModelModel--based scenebased scene
reconstruction/animationreconstruction/animation
7/28/2019 Computer Graphics & Computer Vision
41/49
Page 41
Human Modeling from VideoHuman Modeling from Video
M. Dimitri jevic, S. Ilic, and P. Fua CVPR04, S. I lic, and P. Fua PAMI06M. Dimitri jevic, S. Ilic, and P. Fua CVPR04, S. I lic, and P. Fua PAMI06
PerformancePerformance--Based AnimationBased Animation[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]
7/28/2019 Computer Graphics & Computer Vision
42/49
Page 42
PerformancePerformance--Based AnimationBased Animation
[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]
PerformancePerformance--Based AnimationBased Animation[Pighin, Szeliski, Salezin, 1999][Pighin, Szeliski, Salezin, 1999]
7/28/2019 Computer Graphics & Computer Vision
43/49
Page 43
ALife Modeling PyramidALife Modeling Pyramid
Computational model of animals and humans
BehaviorBehavior
LearningLearning
CognitionCognition
PhysicsBiomechanics / LocomotionBiomechanics / LocomotionPerceptionPerception
Body
An Artificial Fish ModelAn Artificial Fish Model
Innate Characteristics
!Motor Controllers
Behavior Routines
Action SelectionLearning
OptimizationSensors
Perception
Motor
Display Model
Behavior
Biomechanical Model
7/28/2019 Computer Graphics & Computer Vision
44/49
Page 44
ALife Characters in Interactive GamesALife Characters in Interactive GamesThese autonomous characters are aliveThese autonomous characters are alive
CyberLifes Creatures (1996)
Nintendogs (2005)
7/28/2019 Computer Graphics & Computer Vision
45/49
Page 45
ALife for VisionALife for Vision
Deformable organisms for automated medical imageDeformable organisms for automated medical imagesegmentationsegmentation
Memory andprior knowledge
Plan or schedule
Skeleto
n
uscles
Interactions withother cr eatures
Musc le actuation causesshapedeformation
BrainPerception
Perceptual attentionmechanism
Memory andprior knowledge
Plan or schedule
Skeleto
n
uscles
Interactions withother cr eatures
Musc le actuation causesshapedeformation
BrainPerception
Perceptual attentionmechanism
Sensors
and limbsUnderlying
medial based Shaperepresentation
Sensors
and limbsUnderlying
medial based Shaperepresentation
fornix
2
3N-2
1
genu
spleniumrostrum
body
NN-1
upper/right
lower/leftfornix
2
3N-2
1
genu
spleniumrostrum
body
NN-1
upper/right
lower/left
Active Visual TrackingActive Visual Tracking
StereoRetinalImages
Retinal Backprojection ofColor Model
ObliqueView
TopView
7/28/2019 Computer Graphics & Computer Vision
46/49
Page 46
Active Vision SystemActive Vision System
Color Models
Stabilization
Module
Foveation
Module
Retinal ImageStream
Further VisualAnalysis
Optical flow minimization Color histogramintersection & backprojection
aze on ro
LocomotionControl
SelfSelf--Animating Virtual PedestriansAnimating Virtual Pedestrians
In a Virtual Train StationIn a Virtual Train Station (Wei Shao)(Wei Shao)
Autonomous pedestrian behaviorAutonomous pedestrian behavior
Old Penn Station in NYC
Concourses
Main Waiting Room
7/28/2019 Computer Graphics & Computer Vision
47/49
Page 47
Simulation of Pedestrian Activity in the Train StationSimulation of Pedestrian Activity in the Train Station
Virtual VisionVirtual Vision
Simulated visual sensor networksSimulated visual sensor networks
7/28/2019 Computer Graphics & Computer Vision
48/49
Page 48
Persistently Observing a PedestrianPersistently Observing a Pedestrian
CS Faculty Conducting CG and CV ResearchCS Faculty Conducting CG and CV Research
StanleyStanley OsherOsher (CV(CV -- Mathematics)Mathematics)
StefanoStefano SoattoSoatto (CV(CV -- Computer Science)Computer Science)
DemetriDemetri TerzopoulosTerzopoulos (CG & CV(CG & CV -- Computer Science)Computer Science)
ZhuowenZhuowen TuTu (CV(CV -- Neurology)Neurology)
AlanAlan YuilleYuille (CV(CV -- Statistics)Statistics)
SongSong--Chun Zhu (CVChun Zhu (CV -- Statistics)Statistics)
7/28/2019 Computer Graphics & Computer Vision
49/49
Courses OfferedCourses Offered
CS 174ACS 174A Introduction to Computer GraphicsIntroduction to Computer Graphics
CS 174BCS 174B ImageImage--Based Modeling and RenderingBased Modeling and Rendering
CS 174CCS 174C Computer AnimationComputer Animation
Plus several graduate courses in graphics and vision:Plus several graduate courses in graphics and vision:
Artificial Life for Computer Graphics and VisionArtificial Life for Computer Graphics and Vision
Machine VisionMachine Vision
Humanoid Character SimulationHumanoid Character Simulation
Building an Autonomous RobotBuilding an Autonomous Robot
Etc.Etc.
Thank you!Thank you!