Unwarping paper: A differential geometric approachgumerov/PDFs/CVL_032803.pdfUnwarping paper: A...

transcript

UnwarpingUnwarping paper: A differential paper: A differential geometric approachgeometric approach

Nail Gumerov, Ali Zandifar, Ramani Nail Gumerov, Ali Zandifar, Ramani Duraiswami and Larry S. DavisDuraiswami and Larry S. Davis

Perceptual Interfaces and Reality LabPerceptual Interfaces and Reality LabMarch 28, 2003March 28, 2003

03/28/2003 Perceptual Interfaces and Reality Lab (PIRL) 2

OutlineOutline

MotivationMotivationStatement of the ProblemStatement of the ProblemPrevious WorksPrevious WorksDefinition of Applicable SurfaceDefinition of Applicable SurfaceFull Full IntegrabilityIntegrability of the of the PDEPDE’’ssForward ProblemForward ProblemInverse ProblemInverse ProblemFuture Works Future Works ReferencesReferences

Motivation (I)Motivation (I)

To develop a “Seeing-Eye” video-based interface for the visually impaired to access environmental information which leads to a way of independence. We are concerned about those daily activities which involved with interpreting “Environmental texts” or “Scene Texts”There are vast types of texts in outdoor and indoor scene located not necessarily on flat surfacesOff-the-shelf OCR fails to interpret them!!

Motivation (II)Motivation (II)

Various Scene textVarious Scene text

MannMann--BrookBrook--Fogarty[Fogarty[’’99] Table99] Table

Motivation (III)Motivation (III)

Statement of problemStatement of problem3D structure understanding and unwarpingscene texts from single views for OCR purposes

Previous work: Understanding Previous work: Understanding 3D structure (I)3D structure (I)

Multiple or Single views

StereopsisStructure from Motion; optical flow or epipolar constraints PolemanPoleman--Kanade[Kanade[‘‘97], Dellaert97], Dellaert--Seitz[Seitz[’’00],00],……Challenges: good reliable features to track, correspondence Challenges: good reliable features to track, correspondence problem and occlusionsproblem and occlusions

Multiple views

Previous works: Understanding 3D Previous works: Understanding 3D structure (II)structure (II)-- Single viewsSingle views

Shape from Shading Oliensis[Oliensis[’’91], Forsyth91], Forsyth--Ponce[Ponce[’’00],00],……Light source model and reflection model in outdoor scenesLight source model and reflection model in outdoor scenes

Shape from Structured Light WangWang--Mitchie[Mitchie[’’8787’’], ], ……Not useful in outdoor scene texts and sensitive to other light Not useful in outdoor scene texts and sensitive to other light sources in indoor scenesources in indoor scene

Shape from Occluding contours Koenderink[Koenderink[’’84][84][’’90]90]No exact solution but gives qualitatively good resultsNo exact solution but gives qualitatively good results

Shape from Texture RosenholtzRosenholtz--Malik[Malik[’’97], 97], Aloimonos[Aloimonos[’’88], Riberio[88], Riberio[’’89], Garding[89], Garding[’’92]92]

Texture identification and modeling of scene texts and occlusionTexture identification and modeling of scene texts and occlusions s 3D Structure from surface properties

Differential geometric properties of a surface is sufficient forDifferential geometric properties of a surface is sufficient for the the exact 3D structure solution exact 3D structure solution

Previous works: Previous works: UnwarpingUnwarpingInterpolation

Rigid mapping Wolberg[Wolberg[’’90]90]Deformable mapping and elastic matching Terzopolus[Terzopolus[’’96], Blongie[96], Blongie[’’02], 02], ……Need two images, features and correspondence Need two images, features and correspondence throughout whole patchthroughout whole patch

ReconstructionUnwarping using 3D range Data Pilu[Pilu[’’01], 01], Brown[Brown[’’01],01],……Metric rectification Zissermann[Zissermann[’’98][98][’’00], Pilu[00], Pilu[’’01],01],……Assumption: Planarity of an objectAssumption: Planarity of an objectUnwarping using surface propertiesOnly boundaries of a patch in image and parametric Only boundaries of a patch in image and parametric planeplane

Parametric surface representationParametric surface representation

( , ) ( ( , ), ( , ), ( , ))r r u v X u v Y u v Z u v= =Consider a smooth and parametric surface S in 3D:Consider a smooth and parametric surface S in 3D:

Assume 1Assume 1stst and 2and 2ndnd derivatives of r exist w.r.t u and v:derivatives of r exist w.r.t u and v:, , , ,u v uu uv vvr r r r r

DefinitionsDefinitions

First Fundamental form of a surface:First Fundamental form of a surface:

Second Fundamental form of a surface:Second Fundamental form of a surface:

2 2| | , , | |u u v vE r F r r G r= = =

. , ..

u u uu u v uv

v v vv

L r n r n M r n r nN r n r n= − = ⋅ = − = ⋅= − = ⋅

where n is normal vector of a surface:where n is normal vector of a surface:

| |u v

r rnr r×

Applicable Surface (I)Applicable Surface (I)

Isometric ( preserved length) and Isometric ( preserved length) and conformal (preserved angle) with conformal (preserved angle) with flat surfaceflat surface

Zero Gaussian Curvature: At least in one principal Zero Gaussian Curvature: At least in one principal direction the curvature is zero; e.g. cylindrical surfacedirection the curvature is zero; e.g. cylindrical surface

2 2| | 1, 0, | | 1u u v vE r F r r G r= = = = = =

1 2 2 0LN MK k kEG F

−= = =

Applicable Surface (II)Applicable Surface (II)

2 0uu uv vvn ar br cr

We can show from derivatives of isometric and We can show from derivatives of isometric and conformal properties and zero Gaussian curvature conformal properties and zero Gaussian curvature that:that:

Inverse ProblemInverse Problem

TheoremaTheorema EgregiumEgregium (Gauss)(Gauss)

If surface S is obtained from surface S0 by smooth bending(without stretching), at which the first quadratic form (E,F,G) is preserved, the Gauss curvature of the surface S is the same as the Gauss curvature of S0.

Basic Basic PDEPDE’’ssEquations:

Conventional Methods:

• Finite Difference,• Spectral Methods,…

These are high order nonlinear PDEs

Full Full IntegrabilityIntegrability of the of the PDEPDE’’ssWe found a general analytical solution and showedFull integrability of these equations.

Reformulation:

Jacobian of transform (u,v)→(Wu,Wv) is zero! Thus,

Full Full IntegrabilityIntegrability of the of the PDEPDE’’ssFurther, we have degenerativity of the following Jacobians:

This shows that the mapping function, t=t(u,v), is universal for all coordinates:

Full Full IntegrabilityIntegrability of the of the PDEPDE’’ss

Implicitdefinitionof t(u,v).

Relation to WaveRelation to Wave(Hyperbolic) Equations(Hyperbolic) Equations

Linear Wave EquationLinear Wave Equation

v = v0- c0u

u = 0u = 1u = 2u = 3

v0v0-c0v0-2c0v0-3c0u

v Characteristic plane

Perturbationspropagate

v + c0u = const

Wave EquationWave Equation

We have:

(Riemann-Hopf equation):

v = v0- c(t)ut

v0breaking wave

v + c0u = constform of Riemann-Hopf equation

Characteristic PlaneCharacteristic Plane

A’’

B’’ C’’

D’’

Characteristic Plane= Undistorted Flat Plane

Along characteristics:t=const

Full Integral for Applicable Full Integral for Applicable SurfaceSurface

Other equations

If functions are found, the entire solution is known

Forward (Warping) ProblemForward (Warping) ProblemGiven a 3D curve, specified by equation

Given a 2D curve in the (u,v)-plane, specified by equations

Determine applicable surface,

such that

Forward (Warping) ProblemForward (Warping) Problem

vA’’

Γ’’

Γ X Y

Ω’’Ω

Forward (Warping) ProblemForward (Warping) ProblemSolution:

1). Use natural parametrization

2). Solve ODEs(analytical solution is also available)

Length of theCurves is the same!

Forward (Warping) ProblemForward (Warping) Problem

Inverse ProblemInverse ProblemGiven a closed 2D curve in the image plane, specified by equations

Given a 2D curve in the (u,v)-plane, specified by equations

Determine applicable surface,

such that

Here Fx and Fy are given camera equations.Also a few correspondence points are known.

A ’’

B ’’ C ’’

D ’’

v A ’’

B ’’ C ’’

D ’’A ’’

B ’’ C ’’

D ’’

Γ23’’

A’’

B’’C’’

D’’

Γ11’’

Γ12’’

Γ13’’

Γ22’’

Γ21’’

Γ11’

Γ12’

Γ13’Γ23’

Γ22’Γ21’

Patcht = t1

t = t2

Inverse ProblemInverse Problem1). Guess initial break point (for rectangular page this is one of the corners2). Specify initial conditions (currently we need 2 free parameters)3). Solve system of ODE’s with these initial conditions

4). Find the difference of solution and data in the correspondencepoints: try to find proper initial conditions.5). If impossible change guess 1).6). Use other available information for determination of initial conditionsby minimization the error function.

Inverse ProblemInverse Problem1). ODE solver works. Tests are performed.2). For synthetic initial data obtained from the forward problem producesacceptable results.

Current state:

Problems (working on):

• Number of necessary feature points and initial conditions.• Algorithm stability.• Accuracy of the solver near the corner points.• Feature point matching technique.• Incorporation of apriori information to minimize the error

AmbiguitiesAmbiguitiesMethod relies on an image of the boundarySome deformations can lead to same images of boundary– Consider a rectangular sheet of paper

– Both deformations lead to same image in cross-section– Deformation can only be determined by inside data– More generally, we need to derive necessary and sufficient

conditions for the applicable surfaces to identify non-uniqueness of the solution.

3D Curve Warping

B” C”

2D Patch Warping+

2D Patch Unwarping

Image Acquisition

Boundary Detection

Future Works (I)Future Works (I)Sensitivity and Sensitivity and Error AnalysisError Analysis

Choice of boundary valuesChoice of boundary valuesError in Boundary extractionError in Boundary extractionError in Error in unwarpingunwarping problemproblemError in Model (Camera or Error in Model (Camera or Surface property)Surface property)

Future Works (II)Future Works (II)

Boundary Extraction Problem

Future Works (III)Future Works (III)

Shape from Texture is an alternative to understanding Shape from Texture is an alternative to understanding 3D structure and 3D structure and unwarpingunwarping scene texts using scene texts using differential geometric Propertiesdifferential geometric PropertiesChallenges:Challenges:

Texture identification and modeling for a font style Texture identification and modeling for a font style Hypothesized slant and tilt as the gradient of texture Hypothesized slant and tilt as the gradient of texture deformationdeformationShape from texture problemShape from texture problem

ConclusionsConclusions

Developed a method to determine structure from differential geometry constraints and some image informationCorrespondence freeThe solution is newGeneral approach may also be promising for other types of surfacesSubject of research

ReferencesReferences

1.1. N. Gumerov, A. Zandifar, R. Duraiswami and L.S. N. Gumerov, A. Zandifar, R. Duraiswami and L.S. DavisDavis, “ Structure of Applicable Surfaces from Single Views”, sent to ICCV 2003.

2.2. J.J. J.J. KoenderinkKoenderink, “Solid Shape”, MIT press 1990.3.3. M. Do M. Do CarmoCarmo, “Differential Geometry of Curves and

Surfaces”, Prentice Hall 1976.

THANK YOU!

Unwarping paper: A differential geometric approachgumerov/PDFs/CVL_032803.pdfUnwarping paper: A...

Documents