Learning to Shade Hand-drawn Sketches
CVPR 2020 Oral Presenter: Yu Han
2020/04/05
1
Outline • Authors • Background • Network • Experiment • Conclusion
2
Qingyuan Zheng* · University of Maryland
3
Zhuoru Li* · Project HAT
4
Adam Bargteil · Assistant professor at University of Maryland
· Phd at UCB · Postdoc at CMU Graphics Lab
Research Interest: · Computer graphics and animation, especially using physics-based animation
5
Outline • Authors • Background • Network • Experiment • Conclusion
6
Background• Pix2pix
7
Image-to-Image Translation with Conditional Adversarial Networks. In CVPR, 2017.
Background• DeepNormal
8
Deep Normal Estimation for Automatic Shading of Hand-Drawn Characters. In ECCV, 2018.
Background• Sketch2Normal
9
Interactive Sketch-Based Normal Map Generation with Deep Neural Networks. In SIGGRAPH, 2018.
Background• Squeeze and Excitation(SE) Block
10
SE Net:Squeeze-and-Excitation Networks. In CVPR, 2018.
Outline • Authors • Background • Network • Experiment • Conclusion
11
Network
12
Data
13
· 1160 paired hand-drawn line drawings and shadows · Tagged with a lighting direction · 8x3+2=26 directions
Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function
14
Generative Network
15
Two parts: • Shape net: encodes the underlying 3D structure from 2D sketches • Render net: renders artistic shadows based on the encoded structure
Generative Network
16
Render net: • Self-attention: enhance the visual reasoning 3D structure from 2D sketches. • SE blocks: filter out unnecessary features • Extract two supervision side outputs, s1 and s2
Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function
17
Discriminator Network
18
• Self-attention: make discriminator sensitive to the distant features
Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function
19
Loss Function
20
Outline • Authors • Background • Network • Experiment • Conclusion
21
Experiment
22
• Shift, zoom in/out, and rotate to augment the dataset • Input: 320x320 • 80 000 iterations
Experiment
23
Experiment
24
Experiment
25
Experiment
26
User Study
27
All people.
People with drawing experience.
Ablation Study
28
Ablation Study
29
Calculate the first two orders of the distance between distributions in Gaussian space.
Outline • Authors • Background • Network • Experiment • Conclusion
30
Conclusion • New dataset
• Propose a network that “understands” the structure and 3D spatial
relationships implied by line drawings and produces highly-detailed
and accurate shadows
31
Thank you
32