April 12, 2023

1

Pattern Recognition Group

Normalized averaging

using adaptive applicability

functions

Presented at SCIA 2003

Tuan Q. Pham and Lucas J. van Vliet

with applications in image reconstruction

from sparsely and randomly sampled data

April 12, 2023 2

Overview

• Normalized averaging

• Local structure adaptive filtering

• Experimental results

• Comparison with diffusion-based image inpainting

• Directions for further research

April 12, 2023 3

Normalized averaging (Knutsson’93)•Weighted average filtering:

•Normalized averaging = weighted average + signal/certainty principle:

•each signal s is associated with a certainty c•s & c have to be processed separately

where s :signal, c :certainty, a :filter, r :result, * :convolution

( . )s c ar

c a

NA with Gaussian applicability (σ =

1)

input with 10% original pixels

Gaussian smoothing (σ = 1)

*r s a

April 12, 2023 4

Normalized averaging: An exampleReconstruction from 10% random pixels

Nearest neighbor interpolation NA with adaptive applicability

April 12, 2023 5

Image reconstruction using Adaptive Normalized Averaging

Input Image (sparsely & randomly sampled)

Normalized

Averaging

Structure Analysis

Output Image (with local structure extended into missing regions)

σ = 1

Adaptive applicability

April 12, 2023 6

Local structure adaptive filtering

•Local structure from the structure tensor

•orientation φ = arg( )•anisotropy A = (λu - λv)/(λu + λv) •curvature κ = ∂φ /∂•scale rdensity = sample density

•Scale-adaptive curvature-bent anisotropic Gaussian kernel with scales in 2 orthogonal directions:

kernel aligns withlocal structure

T uu vvT T Tu vI I

u

v

212

y x

u v

(1 )u densityC A r (1 )v densityC A r

where C ~ SNR ~ degree of structure enhancement

April 12, 2023 7

Sample Density Transform

•Definition: Smallest radius of a pillbox, centered at each pixel, that encompasses total certainty of at least 1

•Role: Automatic scale selection of the applicability in the NA equation to avoid unnecessary smoothing

Adap. Norm. Avg.Lena with missing hole Density transform NA with Gaussian(σ=1)

April 12, 2023 8

4x4 super-resolution from 4 noisy frames• 4 input LowRes captured with fill-factor = 25%, intensity noise

(σ=10), registration noise (σ=0.2 LR pitch)

1 of 4 input 64x64 LR SR using triangulation SR using adaptive NA

• 16 times upsampling from only 4 frames. How is it possible: along linear structures, only 4 samples are enough for 4x super-resolution

April 12, 2023 9

Scale in perpendicular directionScale along linear structuresSample density

Orientation Anisotropy Curvature

April 12, 2023 10

Comparison with image inpainting• Image inpainting (Sapiro) = diffusion with level line evolution

• also extending orientation into the missing regions

• slow due to iterative nature

• poor result for large holes

input inpainting 110 iters (6

min)

inpainting + texture

synthesis

Adapt. Norm. Avg.

0 iters (6 sec)

April 12, 2023 11

Directions for Further Research

• Applications• Image filtering (noise/watermark removal, edge

enhancement...)• Image interpolation from sparsely and randomly sampled

data (image inpainting, image fusion, super-resolution...)

• Further improvements• Scale-space local structure analysis.• Detect multiple orientations using orientation space. • Robust neighborhood operator than the weighted mean.

April 12, 2023 12

Image inpainting of thin scribbles

Adaptive Normalized Averaging (10 sec)

input

inpainting

April 12, 2023 13

Simultaneous geometry/texture inpainting

geometry

Adaptive NA

(1 min)

textureinput

April 12, 2023 14

Inpainting of ambiguous discontinuity

inpainting Adaptive NA (1 sec)

original input