94
Neural Rendering Chuan Li Lambda Labs Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang
Neural Rendering
Chuan Li
Lambda Labs
Collaborators: Thu Nguyen-Phuoc, Bing Xu, Yongliang Yang, Stephen Balaban, Lucas Theis, Christian Richardt, Junfei Zhang, Rui Wang, Kun Xu, Rui Tang
Model Pictures
Forward (Computer Graphics)
Model Pictures
Forward (Computer Graphics)
Inverse (Computer Vision)
Integral of the incident radians
BRDF
32K SPP Ray Tracing (90 mins 12 CPU Cores)The Tungsten Renderer
P0
P1
P0
P1
R01 | T
01
P0
P1
Inverse (Computer Vision)
R01 | T
01
P0
P1
P2
R12 | T12
Inverse (Computer Vision)
R01 | T
01
P0
P1
Building Rome in a DaySameer Agarwal, Noah Snavely, Ian Simon, Steven M. Seitz and Richard Szeliski
Model Pictures
Sub-module
End-2-End
Differentiable Rendering
1 SPP
2048 SPP
Sub-modules
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
Sub-modules
Value Network
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
Sub-modules
Value Network
Policy Network
Mastering the game of Go with deep neural networks and tree searchDavid Silver et al.
2^15 SPP4 SPP
Value Networks
Denoising
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
4 SPP Denoised1 sec 2080 Ti
32K SPP Ray Tracing90 mins 12 cores CPU
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
loss
DecoderEncoderInput x
Ref
Output
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
L1 VGG Loss
L1 VGG Loss + GAN
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
loss
DecoderEncoderDiffuseInput x Diffuse
Output
DecoderEncoderSpecular
Input x SpecularOutput Ref
Output
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
Auxiliary
loss
DecoderEncoderDiffuseInput x Diffuse
Output
DecoderEncoderSpecular
Input x SpecularOutput Ref
Output
Albedo, normal, depth
Auxiliary
Conv
LeakyReLU
Conv
x
Element-wiseBiasing
Conv
Auxiliary
LeakyReLU
Conv
Conv
LeakyReLU
Conv
x
Element-wiseBiasing
Element-wiseScaling
Conv
Auxiliary
LeakyReLU
Conv
Conv
LeakyReLU
Conv
x
Element-wiseBiasing (OR)
Element-wiseScaling (AND)
Denoise comparison 4 SPP
Adversarial Monte Carlo denoising with conditioned auxiliary feature modulationB Xu et al. Siggraph Asia 2019
2^15 SPP
Value Networks
Denoising
Policy Networks
Same SPP
4 SPP
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
incidence radiance map
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
Neural Importance SamplingThomas Müller et al. ACM Transactions on Graphics 2019
Model Pictures
Sub-module
End-2-End
Differentiable Rendering
Ray TracingImage Centric
RasterizationObject Centric
Ray TracingImage Centric
RasterizationObject Centric
Visibility
Ray TracingImage Centric
RasterizationObject Centric
Shading
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
Neural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
VisibilityNeural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
VisibilityNeural Voxels
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Neural Voxels
32 x 32 x 32 x 16
3D Encoder
3D-2D
32 x 32 x 512
Neural Pixels
2D Decoder
ShadingNeural Voxels Visibility
MSE pixel loss
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Contour
Toon
Ambient OcclusionRenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Thu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
3D Encoder
3D-2D Neural Pixels
2D Decoder
TextureNetwork
NeuralTexture Voxels
or
Neural Voxels
Channel-wise Concatenation
64 x 64 x 64 x 4
64 x 64 x 64 x 1
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Same shape, different textures
Same texture, different shapes
RenderNet: A deep convolutional network for differentiable rendering from 3D shapesThu Nguyen-Phuoc et al. NeurIPS 2018
Depth Map Voxel Point Cloud Mesh
Memory Good Very Poor Poor Very Good
NN friendly Great Yes No Enemy
Rasterization a RGB point cloud
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
Rasterization a neural point cloud
(First three PCA dimensions of the neural descriptor)
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
Neural Point-Based GraphicsKA Aliev et al, arxiv 2019
RB
G ra
ster
izat
ion
Neu
ral r
aste
rizat
ion
Neural 3D Mesh RendererH Kato et al, CVPR 2018
Deferred Neural Rendering: Image Synthesis using Neural Textures
J Thies et al, Siggraph 2019
Model Pictures
Sub-module
End-2-End
?
TargetApproximation
TargetApproximation RenderedApproximation
Loss
Back-propagate
TargetRenderedApproximation
Approximation
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagate
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagateFor Free
Loss
TargetRenderedApproximation
UpdatedApproximation
Back-propagate
Expensive
Loss
TargetRenderedApproximation
DecoderEncoder
Reconstruction Rendering
Human perception imposes coordinate frame on objects
Inductive Bias: Separate Appearance from Pose
Learning 3D representation from natural images without 3D supervision
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
Conditional GANs
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
Info GANs
Conditional GANs
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
3D StyleGAN
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
Real/Fake
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
RenderNet3D Generator
A representation that is unbreakable under 3D rigid-body transformations
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
z1 z2
Shape Controller Texture Controller
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
HoloGAN: Unsupervised learning of 3D representations from natural imagesThu Nguyen-Phuoc et al, ICCV 2019
Model Pictures
Forward (Computer Graphics)
Inverse (Computer Vision)
Model Pictures
Sub-module for Ray Tracing (Value / Policy Networks)
End-2-End Rasterization (Depthmap, Voxel, Point Cloud, Mesh)
Differentiable Rendering (Representation Learning)
Thu Nguyen-Phuoc Bing Xu Yongliang Yang Stephen Balaban
Lucas Theis Christian Richardt Junfei Zhang Rui Wang Kun Xu Rui Tang
