Post on 24-Jun-2020
transcript
Chenglei Wu2, Derek Bradley1, Pablo Garrido3, Michael Zollhöfer3,
Christian Theobalt3, Markus Gross1,2, Thabo Beeler1
1Disney Research 2ETH Zurich 3Max Planck Institute for Informatics
Model-Based Teeth Reconstruction
Michael Zollhöfer Model-Based Teeth Reconstruction 2
MOTIVATION
Michael Zollhöfer Model-Based Teeth Reconstruction 2
Michael Zollhöfer Model-Based Teeth Reconstruction 3
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 3
Michael Zollhöfer Model-Based Teeth Reconstruction 4
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 4
Berard16
Michael Zollhöfer Model-Based Teeth Reconstruction 5
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 5
Berard16
Bermano16
Michael Zollhöfer Model-Based Teeth Reconstruction 6
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 6
Beeler12
Berard16
Bermano16
Michael Zollhöfer Model-Based Teeth Reconstruction 7
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 7
Beeler12
Berard16
Bermano16
Nagano15
Michael Zollhöfer Model-Based Teeth Reconstruction 8
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 8
Beeler12
Berard16
Bermano16
Nagano15
Hu14
Michael Zollhöfer Model-Based Teeth Reconstruction 9
Teeth are missing!
MOTIVATION
Input Image Beeler11
Michael Zollhöfer Model-Based Teeth Reconstruction 10
MOTIVATION
Input Image Beeler11
Teeth are missing!
Michael Zollhöfer Model-Based Teeth Reconstruction 11
Complex Appearance
MOTIVATION
Input Image Beeler11
Teeth are missing!
Michael Zollhöfer Model-Based Teeth Reconstruction 12
Complex Appearance
Featureless
MOTIVATION
Input Image Beeler11
Teeth are missing!
Michael Zollhöfer Model-Based Teeth Reconstruction 13
Complex Appearance
Occlusions
Featureless
MOTIVATION
Input Image Beeler11
Teeth are missing!
Michael Zollhöfer Model-Based Teeth Reconstruction 14
Complex Appearance
Occlusions
Featureless
Input Image Our Reconstruction
Teeth are missing!
MOTIVATION
Michael Zollhöfer Model-Based Teeth Reconstruction 15
RELATED WORK
Michael Zollhöfer Model-Based Teeth Reconstruction 16
Medical Dentistry
RELATED WORK
From CT images [Omachi07, Yanagisawa14]
Michael Zollhöfer Model-Based Teeth Reconstruction 17
Medical Dentistry
RELATED WORK
From CT images [Omachi07, Yanagisawa14]
Intra-Oral Scanners[3M True Definition, iTero, 3Shape TRIOS]
Michael Zollhöfer Model-Based Teeth Reconstruction 18
Medical Dentistry
RELATED WORK
From CT images [Omachi07, Yanagisawa14]
Intra-Oral Scanners[3M True Definition, iTero, 3Shape TRIOS]
Photogrammetric Reconstruction
Shape from Shading[Carter10, Farag13, Mostafa14]
Michael Zollhöfer Model-Based Teeth Reconstruction 19
Medical Dentistry
RELATED WORK
From CT images [Omachi07, Yanagisawa14]
Intra-Oral Scanners[3M True Definition, iTero, 3Shape TRIOS]
Photogrammetric Reconstruction
Shape from Shading[Carter10, Farag13, Mostafa14]
Contours and Feature Points[Zheng11]
Michael Zollhöfer Model-Based Teeth Reconstruction 20
Medical Dentistry
RELATED WORK
From CT images [Omachi07, Yanagisawa14]
Intra-Oral Scanners[3M True Definition, iTero, 3Shape TRIOS]
Photogrammetric Reconstruction
Shape from Shading[Carter10, Farag13, Mostafa14]
Contours and Feature Points[Zheng11]
Statistical Models[Mehl05, Buchaillard07]
Michael Zollhöfer Model-Based Teeth Reconstruction 21
HUMAN TEETH
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 22
HUMAN TEETH
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 23
HUMAN TEETH
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 24
OVERVIEW
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 25
OVERVIEW
Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 26
OVERVIEW
Scans
Contour Extraction
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 27
OVERVIEW
Teeth Prior Model
Scans
Dental Scans
Contour Extraction
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 28
OVERVIEW
Teeth Prior Model
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 29
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Static Images/Video Clip
Dental Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 30
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 31
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 32
TEETH MODEL
• 86 high resolution plaster cast scans
Michael Zollhöfer Model-Based Teeth Reconstruction 33
TEETH MODEL
• 86 high resolution plaster cast scans
Michael Zollhöfer Model-Based Teeth Reconstruction 34
Teeth Segmentation Fitted Template
PREPROCESSING
Michael Zollhöfer Model-Based Teeth Reconstruction 35
Input Scan Teeth Segmentation Fitted Template
PREPROCESSING
Michael Zollhöfer Model-Based Teeth Reconstruction 36
Input Scan Teeth Segmentation Fitted Template
PREPROCESSING
Michael Zollhöfer Model-Based Teeth Reconstruction 37
Input Scan Teeth Segmentation Fitted Template
PREPROCESSING
Michael Zollhöfer Model-Based Teeth Reconstruction 38
𝐙𝛕 = 𝐒 𝑻τ 𝐀𝛕 +
𝐢
𝛂𝛕𝐢𝐁𝛕𝐢
PARAMETRIC TEETH MODEL
Michael Zollhöfer Model-Based Teeth Reconstruction 39
𝐙𝛕 = 𝐒 𝑻τ 𝐀𝛕 +
𝐢
𝛂𝛕𝐢𝐁𝛕𝐢
PARAMETRIC TEETH MODEL
𝐀𝛕
Michael Zollhöfer Model-Based Teeth Reconstruction 40
𝐙𝛕 = 𝐒 𝑻τ 𝐀𝛕 +
𝐢
𝛂𝛕𝐢𝐁𝛕𝐢
PARAMETRIC TEETH MODEL
𝛂𝛕𝐢𝐁𝛕𝐢
𝐀𝛕
shape of tooth 𝛕
Michael Zollhöfer Model-Based Teeth Reconstruction 41
𝐙𝛕 = 𝐒 𝑻τ 𝐀𝛕 +
𝐢
𝛂𝛕𝐢𝐁𝛕𝐢
PARAMETRIC TEETH MODEL
𝛂𝛕𝐢𝐁𝛕𝐢
𝑻τ
𝐀𝛕
position of tooth 𝛕 shape of tooth 𝛕
Michael Zollhöfer Model-Based Teeth Reconstruction 42
𝐙𝛕 = 𝐒 𝑻τ 𝐀𝛕 +
𝐢
𝛂𝛕𝐢𝐁𝛕𝐢
PARAMETRIC TEETH MODEL
𝛂𝛕𝐢𝐁𝛕𝐢
𝑻τ
𝐀𝛕𝐒
Scale and position position of tooth 𝛕 shape of tooth 𝛕
Michael Zollhöfer Model-Based Teeth Reconstruction 43
GAUSSIAN DISTRIBUTION
𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
Michael Zollhöfer Model-Based Teeth Reconstruction 44
GAUSSIAN DISTRIBUTION
• All dimensions modeled via multivariate Gaussians
𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
Michael Zollhöfer Model-Based Teeth Reconstruction 45
GAUSSIAN DISTRIBUTION
• All dimensions modeled via multivariate Gaussians
𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
Michael Zollhöfer Model-Based Teeth Reconstruction 46
GAUSSIAN DISTRIBUTION
• All dimensions modeled via multivariate Gaussians
𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
𝒑 𝑿
Teeth Prior
Michael Zollhöfer Model-Based Teeth Reconstruction 47
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 48
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 49
CONTOUR EXTRACTION
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 50
CONTOUR EXTRACTION
Scans
• Teeth are featureless
Michael Zollhöfer Model-Based Teeth Reconstruction 51
CONTOUR EXTRACTION
Scans
• Teeth are featureless
Except the silhouette
Michael Zollhöfer Model-Based Teeth Reconstruction 52
CONTOUR EXTRACTION
Scans
• Teeth are featureless
Except the silhouette
.
Michael Zollhöfer Model-Based Teeth Reconstruction 53
CONTOUR EXTRACTION
Scans
Boosted Edge Learning (BEL)
• Teeth are featureless
Except the silhouette
.
Michael Zollhöfer Model-Based Teeth Reconstruction 54
FEATURE FUSION
Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 55
FEATURE FUSION
Scans
Input
Michael Zollhöfer Model-Based Teeth Reconstruction 56
FEATURE FUSION
Scans
Input
BEL
Michael Zollhöfer Model-Based Teeth Reconstruction 57
FEATURE FUSION
Scans
Input
BEL
Gabor
Michael Zollhöfer Model-Based Teeth Reconstruction 58
FEATURE FUSION
Scans
Input
BEL
Gabor
Fusion
Michael Zollhöfer Model-Based Teeth Reconstruction 59
FEATURE FUSION
Scans
Input
BEL
Gabor
Fusion
Formulas in the paper!
Michael Zollhöfer Model-Based Teeth Reconstruction 60
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 61
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 62
• Given detected contours 𝑪
TEETH FITTING
Michael Zollhöfer Model-Based Teeth Reconstruction 63
• Given detected contours 𝑪
• Find parameters: 𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
TEETH FITTING
Michael Zollhöfer Model-Based Teeth Reconstruction 64
• Given detected contours 𝑪
• Find parameters: 𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
𝑿∗ = 𝒂𝒓𝒈𝐦𝐚𝐱𝑿𝒑 𝑿 𝑪
= 𝒂𝒓𝒈𝐦𝐚𝐱𝑿[ 𝒑 𝑪 𝑿 ⋅ 𝒑 𝑿 ]
MAXIMUM A POSTERIORI (MAP) PROBLEM
Michael Zollhöfer Model-Based Teeth Reconstruction 65
• Given detected contours 𝑪
• Find parameters: 𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
𝑿∗ = 𝒂𝒓𝒈𝐦𝐚𝐱𝑿𝒑 𝑿 𝑪
= 𝒂𝒓𝒈𝐦𝐚𝐱𝑿[ 𝒑 𝑪 𝑿 ⋅ 𝒑 𝑿 ]
Teeth Edge Likelihood
MAXIMUM A POSTERIORI (MAP) PROBLEM
Michael Zollhöfer Model-Based Teeth Reconstruction 66
• Given detected contours 𝑪
• Find parameters: 𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
𝑿∗ = 𝒂𝒓𝒈𝐦𝐚𝐱𝑿𝒑 𝑿 𝑪
= 𝒂𝒓𝒈𝐦𝐚𝐱𝑿[ 𝒑 𝑪 𝑿 ⋅ 𝒑 𝑿 ]
Teeth Edge Likelihood
Teeth Prior
MAXIMUM A POSTERIORI (MAP) PROBLEM
Michael Zollhöfer Model-Based Teeth Reconstruction 67
• Given detected contours 𝑪
• Find parameters: 𝑿 = (𝐒 , 𝑻τ, {𝛂𝛕𝐢 })
𝑿∗ = 𝒂𝒓𝒈𝐦𝐚𝐱𝑿𝒑 𝑿 𝑪
= 𝒂𝒓𝒈𝐦𝐚𝐱𝑿[ 𝒑 𝑪 𝑿 ⋅ 𝒑 𝑿 ]
Teeth Edge Likelihood
Teeth Prior
MAXIMUM A POSTERIORI (MAP) PROBLEM
Solve with EM-Algorithm
Michael Zollhöfer Model-Based Teeth Reconstruction 68
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 69
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 70
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 71
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 72
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 73
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 74
EM ALGORITHM
Michael Zollhöfer Model-Based Teeth Reconstruction 75
E-STEP
Michael Zollhöfer Model-Based Teeth Reconstruction 76
TEETH EDGE LIKELIHOOD
𝒑 𝑪 𝑿
Teeth Edge Likelihood
Michael Zollhöfer Model-Based Teeth Reconstruction 77
TEETH EDGE LIKELIHOOD
𝒑 𝑪 𝑿
Teeth Edge Likelihood
𝒑𝒑 𝑪 𝑿
=
Michael Zollhöfer Model-Based Teeth Reconstruction 78
TEETH EDGE LIKELIHOOD
𝒑 𝑪 𝑿
Teeth Edge Likelihood
𝒑𝒑 𝑪 𝑿
𝒑𝒏 𝑪 𝑿
=
⋅
Michael Zollhöfer Model-Based Teeth Reconstruction 79
TEETH EDGE LIKELIHOOD
𝒑 𝑪 𝑿
Teeth Edge Likelihood
𝒑𝒑 𝑪 𝑿
𝒑𝒏 𝑪 𝑿
=
⋅
Formulas in the paper!
Michael Zollhöfer Model-Based Teeth Reconstruction 80
M-STEP
Michael Zollhöfer Model-Based Teeth Reconstruction 81
M-STEP
Michael Zollhöfer Model-Based Teeth Reconstruction 82
ITERATE
Michael Zollhöfer Model-Based Teeth Reconstruction 83
DONE
Michael Zollhöfer Model-Based Teeth Reconstruction 84
COLOR AND GUMS
Michael Zollhöfer Model-Based Teeth Reconstruction 85
COLOR AND GUMS
• Projective Texturing
Michael Zollhöfer Model-Based Teeth Reconstruction 86
Template
Gum Line
COLOR AND GUMS
• Projective Texturing
• Fit 3D gum template via Laplacian deformation
Michael Zollhöfer Model-Based Teeth Reconstruction 87
Template
Gum Line
Result
COLOR AND GUMS
• Projective Texturing
• Fit 3D gum template via Laplacian deformation
Michael Zollhöfer Model-Based Teeth Reconstruction 88
OVERVIEW
Teeth Prior Model 3D Reconstruction
Scans
Teeth Reconstruction
Contour Extraction
Dental Scans
Static Images/Video Clip
Michael Zollhöfer Model-Based Teeth Reconstruction 89
OVERVIEW
Teeth Prior Model 3D Reconstruction
Static Images/Video Clip
Teeth Reconstruction
Contour Extraction
Dental Scans
Michael Zollhöfer Model-Based Teeth Reconstruction 90
MULTI-VIEW IMAGES
Michael Zollhöfer Model-Based Teeth Reconstruction 91
MONOCULAR VIDEO
Michael Zollhöfer Model-Based Teeth Reconstruction 92
APPLICATION: TOOTH RESTORATION
1 tooth missing 2 teeth missing 4 teeth missing
Michael Zollhöfer Model-Based Teeth Reconstruction 93
APPLICATION: TOOTH RESTORATION
1 tooth missing 2 teeth missing 4 teeth missing
Michael Zollhöfer Model-Based Teeth Reconstruction 94
APPLICATION: TOOTH RESTORATION
1 tooth missing 2 teeth missing 4 teeth missing
Michael Zollhöfer Model-Based Teeth Reconstruction 95
APPLICATION: TOOTH RESTORATION
1 tooth missing 2 teeth missing 4 teeth missing
Michael Zollhöfer Model-Based Teeth Reconstruction 96
First non-invasive teeth reconstruction approach
CONCLUSION
Michael Zollhöfer Model-Based Teeth Reconstruction 97
First non-invasive teeth reconstruction approach
Content creation(i.e. digital actor)
CONCLUSION
Michael Zollhöfer Model-Based Teeth Reconstruction 98
First non-invasive teeth reconstruction approach
Content creation(i.e. digital actor)
Dentistry(i.e tooth restoration)
CONCLUSION
THANK YOU!
P. Garrido M. ZollhöferC. Wu D. Bradley T. BeelerC. Theobalt M. Gross
Michael Zollhöfer Model-Based Teeth Reconstruction100
APPENDIX