IIIT
Procedural 3D Building Reconstruction
using Shape Grammars and Detectors
Markus Mathias
Anđelo Martinović
Julien Weissenberg
Luc Van Gool
May, 19th 2011
IIIT2/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
IIIT3/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
IIIT4/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
IIIT5/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
IIIT6/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
Inverse ProceduralModeling System
IIIT7/31
System Overview
Colonnade --> split(x){ Column | { Column }* | Column } Column --> split(y){ ~1 : Shaft | Capital }
Inverse ProceduralModeling System
IIIT8/31
Modelling: Procedural Modelling
• Create models from set of rules separate from evaluation engine
• Generating large scenes with only a few amount of rules
• Models are compact and semantically meaningful
• Used for plant modelling, façades and even entire cities including street network, building, façades, trees
• BUT: Focus so far mostly on creation of new virtual buildings, not existing ones
Images courtesy of www.procedural.com
IIIT9/31
Pictures
Grammar
?
How can we use prior knowledge encoded by the grammar?
Inverse Procedural Modeling
IIIT10/31
Pictures
Grammar
How can we use prior knowledge encoded by the grammar?
Asset detectors!
Asset detectors relate image areas with grammar symbols
Inverse Procedural Modeling
IIIT11/31
GrammarInterpreter
VisionModule
AssetDetectors
3D ReconstructionModule
Inverse Procedural Modeling System
IIIT12/31
GrammarInterpreter
VisionModule
AssetDetectors
3D ReconstructionModule
Inverse Procedural Modeling System
IIIT13/31
Grammar Interpreter
Analyzes input grammar Extracts semantic information CGA shape:
Standardized description Powerful shape operations Readable by humans Tool for rendering available (Cityengine)
GrammarInterpreter
VisionModule
Symbol listMatching symbols
Structural InformationEstimated attributes
IIIT14/31
Temple Grammar Example
Colonnade --> split(x){ ~ columnSpacing : Column | { ~ columnSpacing : Column }*
| ~ columnSpacing : Column }
Column --> split(y){ ~1 : Shaft | capitalHeight : Capital }
Capital --> split(y){ ~echinusPartsHeight: i(echinusAsset) | ~abacusPartsHeight : i(abacusAsset) }
Shaft --> i(shaftAsset)
Colonade
IIIT15/31
GrammarInterpreter
VisionModule
AssetDetectors
3D ReconstructionModule
Inverse Procedural Modeling System
IIIT16/31
Asset Detector
Felzenszwalb's part based Detector is trained on a diverse set of examples, e.g. Doric, Corinthian and Ionic capitals
High confidence detections through the reconstruction process Retraining of the detector including these detections leads to
specialized detector
IIIT17/31
GrammarInterpreter
VisionModule
AssetDetectors
3D ReconstructionModule
Inverse Procedural Modeling System
IIIT18/31
3D reconstruction module
Using SfM web service ARC3D:● Sparse point-cloud● Camera calibrations ● Only used to support our system and not providing parts
for output model
ARC3D
IIIT19/31
GrammarInterpreter
VisionModule
AssetDetectors
3D ReconstructionModule
Inverse Procedural Modeling System
IIIT21/31
Vision Module
Accumulation of detections in 3D
Parameter estimation for Grammar Interpreter● Lot size, Asset size, Repeat distances, asset color
IIIT22/31
Vision Module
Substantiation of the semantic information coming from the grammar interpreter:
● Spatial relation of detected assets
● Similarity detection if asset appears in “repeat”
IIIT23/31
Vision module - Results
Determined parameters are translated by the grammar interpreter to the appropriate grammar attributes:
Detections with high confidence are included into the detector training set:
189+188 (new detections) for capitals 204+124 (new detections) for shafts Specialized column and shaft detector for Greek Doric temples: Detection rate for capitals increased by 7.31% (FP at 2.2) Detection rate for shafts increased by 14.89% (FP at 5.4)
IIIT30/31
Future Work
Registration of model to the point-cloud: Measurement of the accuracy of the estimated model Adjustment of remaining parameters
Extension to use the system on houses of different styles: Haussman Neo-classical Renaissance