To understand Leonardo da Vinci drawings
using 3D techniques
Marco Gaiani, Fabrizio Ivan Apollonio, Roberta Barsanti, Roberto Palermo
8 MAY 2019 – RIJKSMUSEUM, AMSTERDAM
Alma mater studiorum A.D. 1088
Università di Bologna
2
Introduction
Working group
Alma Mater Studiorum – University of Bologna
Fabrizio Ivan Apollonio, Giovanni Bacci, Andrea Ballabeni, Marco Gaiani,
Simone Garagnani
Gallerie degli Uffizi - Florence
Roberto Palermo
Museo Leonardiano - Vinci
Roberta Barsanti
Relio
Marco Bozzola
3
Introduction
Outline
The problem
Fine drawings visualization, understanding and analysis
The solution
New application to analyze drawings as graphic artifacts
ISLe – InSightLeonardo
Development 2011-2019
Case study
Leonardo da Vinci drawings
4
Case study
Study of human body proportionsKnown as
The Vitruvian man
Pen and ink with wash over metalpoint on white paper,
345x246 mm
Gabinetto Disegni e Stampe, Gallerie
dell’Accademia, Venice
Recto
In partnership with:
Acquisition 2014
Probably the most famous existing drawing
5
The back of the drawing unveils the copying work of
the painter Giuseppe Bossi at the beginning of the XIX
century
Verso
Case study
Study of human body proportionsKnown as
The Vitruvian man
6
Case study
Landscape, 5 August 1473
Pen and two colors of brown
iron gall ink, red chalk, lead tip
on paper 196 x 287 mm., 1473
Gabinetto Disegni e Stampe
degli Uffizi, Florence, inv. 8P
Recto:
Landscape of the Arno River
and Valley and Marmore
waterfall
Acquisition 2018
7
Case study
Landscape, 5 August 1473
Pen and two colors of brown
iron gall ink, red chalk, lead tip
on paper 196 x 287 mm.
Gabinetto Disegni e Stampe
degli Uffizi, Florence, inv. 8P
Verso:
Landscape, man running,
some geometric figures
8
Case study
Study for the Background of the Adoration of the Magi
Metalpoint, pen and brown ink,
brush and brown wash, traces
of white gouache highlights,
over stylus and compass
construction, pinpricks for
measurement, 163 x 290 mm.
Gabinetto Disegni e Stampe
degli Uffizi, Florence, inv.
436E
Recto
Acquisition 2018
9
Leonardo da Vinci, The Adoration of the Magi
Unfinished painting. Oil and some tempera
grassa on wood, 246×243 cm, 1481-1482
Galleria degli Uffizi, Florence
Case study
Study for the Background of the Adoration of the Magi
10
Case study
Study for the Background of the Adoration of the Magi
Metalpoint, reworked with pen
and brown ink, brush and
brown wash, traces of white
gouache highlights, over stylus
and compass construction,
pinpricks for measurement, 163
x 290 mm.
Gabinetto Disegni e Stampe
degli Uffizi, Florence, inv.
436E
Verso
11
The new application
ISLe - InSightLeonardo
GoalTo improve tracings and hatchings analysis to understand the artist
graphic method and the drawing features
MethodExperience the drawing as if it were in your hands
OutputDigital copy of the physical drawing perceptually indistinguishable
from the original & manipulable using gestures
TargetScholars and exhibition & museums visitors
Solution
Acquisition, real-time rendering & interaction
of the drawing’s total appearance
12
ISLe background
Main observations
The spectrum of colors in a drawing is
much narrower than the spectrum of
colors in a painting
Multispectral techniques not offer
significant advantages
No advantages for tracing
definitions
Color gamut
Colorimetric reproduction
13
ISLe background
Main observations
Thickness left by pens and pencils 5÷10 μm
Laser scanner techniques are not effective (max lateral spatial
resolution possible 50-60 μm)
CG simulation of
surface features from
images
The Vitruvian Man
Depth information
14
ISLe background
Main observations
Appearence structure modeling
Macrostructure (polygons)
Mesostructure (small details and bumps)
Microstructure (light reflection orientation)
S.H.Westin, J.Arvo, K.E.Torrance, Predicting reflectance functions from complex
surfaces, 1992
15
ISLe
Workflow features – three modules
ACQUISITION
• SHAPE
• COLOR
• BRDF
1. Accurate and safe on-site image capture of drawing shape and surface
reflectance with a faithful 3D reconstruction of its features
16
ISLe
Workflow features – three modules
ACQUISITION
• SHAPE
• COLOR
• BRDF
VISUALIZATION
• VIEWER
• RENDERING
• VISUALIZATION DEVICE
2. High fidelity real-time rendering of the total appearence of the drawing:
resolution of 50 mm; accurate color reproduction on a 100% sRGB capable display
17
ISLe
Workflow features – three modules
ACQUISITION
• SHAPE
• COLOR
• BRDF
VISUALIZATION
• VIEWER
• RENDERING
• VISUALIZATION DEVICE
INTERACTION
• INTERACTION DEVICE
• INTERACTION TECHNIQUES
• ADDED TOOLS
• SEMANTIC INFORMATION ORGANIZATION
3. Adaptation of traditional multitouch interaction paradigm to fit the exploration
of 2D-3D contents to minimize uncommon gestures
18
Developed workflow
Data capture
1. System characterization
2. Resolution characterization
3. Color correction
4. Image post-processingCIEXYZ to sRGB
Sharpen
Camera selection Sensor quality
Noise measurement
Light selection &
positioning
Target 18% Light uniformity
ISO 12233 MTF test
X-Rite CC
CIEXYZ color space
Color correction
Color accuracy
19
Developed workflow
Resolution required 8000 x 5500 pixels
Color quality required: 48-bit depth
Large format scanning back based Rencay
DiRECT Camera System 24k³
Sensor: trilinear ON semiconductor 8000 CCD
Maximum optical resolution: 8000 X 13000 pixels
Maximum resolution: 24000 X 39000 pixels
Color depth: 48 bit
Pixel size: 9 μm (H) x 9 μm (V)
Resolution limit: 55 lp/mm with contrast > 60%
Sensor area: 72 x 118 mm
Capture device
Chip back Scan back
20
Developed workflow
Lighting system – 2014 solution
Four Osram Studioline L-55W fluorescent continuous lights
CCT 5600°K
Operating temperature 22°C
Luminous Flux 3800 lm
Output Color Rendering Index (CRI) 85%
Spectral power distribution (SPD)
Presence of frequencies in which the response is limited
UV internally used to excite the phosphors, could be source of damaging UV
21
Developed workflow
Lighting system – 2018 solution
16 High Flux single LED white light Relio2
CCT 4000°K
Operating temperature: 20°C
Luminous Flux 430 lm
Output Color Rendering Index (CRI) > 95%
Spectral power distribution (SPD)
Continuous spectrum of visible light
No IR & UV wavelengths light components
22
Developed workflow
Lighting system – 2018 solution
Relio LED solution advantages
- Light spectral properties are consistent from shot to shot and year to year
- Lumens/watt is stable
- Low power and consumption
- Limited weight (2,5 kg whole system)
- Quick setup and limited space required
24
Developed workflow
Evaluation of the levels of illumination during the photoshot
2014 solution - Four Osram Studioline L-55W fluorescent continuous lights
Results
2018 solution - 16 High Flux LED white light Relio2
25
Developed workflow
Five Rules of Colorimetric Imaging
(1) Lighting correlated color temperature (CCT) near 5000 K (D50 workflow)
(2) Optimal exposure
(3) Profile based on minimizing ∆E with outstanding lightness accuracy
(4) Independent validation using target not used for camera profiling
(5) Encoding space does not clip scene colors
Roy Berns
Scientific Imaging of Cultural Heritage: Minimizing visual editing and relighting
2+3D Photography Practice and Prophecies, 2015
Same target used for calibration
+
Expert observers judgment
26
Developed workflow
Color correction - SHAFT (SAT & HUE Adaptive Fine Tuning)
Features
- Target based technique
- Based on Bruce Fraser's calibration procedure for successive approximations
from RAW images (ACR-scripts)
- Developed in MATLAB supported by DCRaw
(www.cybercom.net/~dcoffin/dcraw/)
- CIEDE2000 colour-difference formula to evaluate color error
- Usable alone or coupled with a polynomial regression
M. Gaiani, A. Ballabeni, SHAFT (SAT & HUE Adaptive Fine Tuning), a new automated
solution for target-based color correction, 2018
Gaiani et al., “Securing Color Fidelity in 3D Architectural Heritage Scenarios”, 2017
27
Color correction
Target selection
(1) to establish accurate color image capture - part of a calibration function
(2) to evaluate deviation from the desired capture of color image information
X-Rite ColorChecker Classic
Full Size: 8.5”x11”
24 Patches - 6 Neutral Gray
Problem
CC for subjects predominated by near neutrals and limited color
Typical solution
collection-specific color test targets
TrialsDon Williams, Targeting for Important Color Content: Near Neutrals and Pastels, 2012
Unstable results
Candidate colors selection is not easy and can lead to even greater errors
Solution based on observer judgement & weighted version of SHAFT
to better fit color existing in the drawings
Judgment on results of differently weighted CC images against the drawing and a
manually CC image on different monitors & room lighting condition
28
Color accuracy & exposure evaluation
Results - Florence (no weights)
Mean color accuracy ΔE*00 = 2.31
Mean lightness accuracy ΔL*00 = 0.59
Exposure error 0.02 f-stop
29
Results - Florence (weighted)
Mean color accuracy ΔE*00 = 2.25
Mean lightness accuracy ΔL*00 = 0.63
Exposure error 0.00 f-stop
Color accuracy & exposure evaluation
30
Developed workflowDeveloped workflow
Resolution characterization - ISO 12233:2000
MTF results - Florence
Average spatial resolution (horizontal)
▪ Rising edge distance of 10-90% = 4.76 pixels
▪ MTF50 = 0.179 line pairs/pixel
▪ MTF10 = 0.452 line pairs/pixel
▪ Effective resolution: 575 Px/inch finer
detail of 75 µm
Requirements
Leonardo tracing of a minimum of 90 µm thick
565 ppi are needed
31
Developed workflow
Material modeling – Macroscale
Surface shape
Assimilated to a polygonal plane (16384 triangles)
32
Developed workflow
Material modeling - Mesoscale
Multitexture method with four map
▪ Albedo
▪ Normals
▪ Depth
▪ Specular
33
Developed workflow
Material modeling - Mesoscale
Normal map reconstruction: photometric stereo techniques
Four Light Imaging
4 images with constant illumination from 4 directions, ≈orthogonal to each other
Solution implemented
B.D. Cox & R.S. Berns, Imaging artwork in a studio environment for
computer graphics rendering, 2015
34
Developed workflow
Material modeling - Mesoscale
Normals reconstruction: results
35
Developed workflow
Material modeling - Mesoscale
Normals reconstruction: results
Improved
Cox & Berns
Old
technique
36
Developed workflow
Material modeling – Mesoscale/Macroscale
Height map reconstruction
1. Gradient map reconstruction
2. Displacement in the normal direction of the drawing plane
Height map from uncorrect normal map
Problem
Normal map inaccuracies caused by
- not perfectly regular distribution of the lighting despite
the flat fielding regularization
- inaccuracies of the position measurements of the four
lights (i.e. violation of the condition of distant light)
37
Developed workflow
Material modeling – Mesoscale/Macroscale
Height map reconstruction
Process
1. Lighting decay correction using the lightness channel on the gray card images
2. Normal map reconstruction using Cox & Berns technique on the correct images
3. Gradient map reconstructon using the Knald gradient operator, which allows to
minimize cumulative errors due to residual noise (https://www.knaldtech.com)
4. Displacement in the normal direction of the drawing plane
Displacement calibration
1. Conversion of the gradient map into a modeled surface using an algorithm that
follows an error minimization principle with Grasshopper & Kangaroo plug-in
2. Calibration against results of 3 test-case papersheet with shape configurations
similar to those of drawings reconstructed with automatic photogrammetry techniques
Height map from correct normal map
Accuracy achieved: 50 mm
38
Developed workflow
Material modeling – Mesoscale/Macroscale Results
PhotoRendered
with Corona render
Study for the Background of the Adoration of the Magi
39
Developed workflow
Material modeling - Microscale
Paper BRDF (Bidirectional reflectance distribution function) and shader
Inspired to M. Papas, K. de Mesa, H. W. Jensen, A Physically-Based BSDF for
Modeling the Appearance of Paper, 2014
Exploiting the Walt Disney physically-based shading framework
B. Burley, SIGGRAPH 2012 & 2015
Implemented in the Unity rendering engine system
www.unity3D.com
40
Developed workflow
Material modeling - Paper BRDF
micro-surface
subsurface
retro-reflection sheen
diffuse
Paper modeled as material
- highly scattering
- optically thick
- which exhibits a combination of subsurface scattering, specular reflection,
retroreflection, surface sheen, and transmission effects
41
Developed workflow
Material modeling - Paper BRDF
BRDF model
- Reflection & refraction: microfacet model of the rough glass (Walter et al., 2007;
Torrance & Sparrow 1967; Cook & Torrance, 1981)
- Subsurface scattering:
▪ A BSDF reduction of the (Donner & Jensen, 2005) multi-layer model based on
the assumption that the length of the mean free path is orders of magnitude
smaller than the thickness of paper and the spatial resolution of the sensor
▪ The single scattering theory of (Hanrhan & Krueger, 1993)
42
Developed workflow
Material modeling results – Microscale
Old shader New shader
Paper
43
Iron gall ink
Old shader New shader
Developed workflow
Material modeling results – Microscale
44
Results
Landscape, 5 August 1473
Demo
45
Whats happen?
Perfecto e virtuale
Exhibition
24 OCTOBER 2014 - 6 JANUARY 2015
Arco di Augusto - Fano
.20.000 visitors!The opening day
Paola Salvi in front to the table
Paola Salvi, The Midpoint of the Human Body in the
Leonardo’s Drawings and in the Codex Huygens, 2016
Whats happen?
Perfecto e virtuale
One year later
Paola Salvi in front to the table
Whats happen?
Leonardo a Vinci. Alle origini del genio
The opening day
Exhibition
15 APRIL 2019 - 15 OCTOBER 2019
Museo Leonardiano, Vinci
Whats happen?
Leonardo a Vinci. Alle origini del genio
In print
Marco Gaiani, Fabrizio Ivan Apollonio, Giovanni Bacci, Andrea Ballabeni, Marco
Bozzola, Riccardo Foschi, Simone Garagnani, Roberto Palermo, Vedere dentro i
disegni Un sistema per analizzare, conservare, comprendere comunicare i disegni
di Leonardo, 2019
New insights on the Landscape, 5 August 1473, using ISLe!
49
Thank you for your attention!
Marco Gaiani [email protected]
Fabrizio Ivan Apollonio [email protected]
Roberta Barsanti [email protected]
Roberto Palermo [email protected]
