Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28 th, 2004 serene...

Composition-Guided Image Acquisition

Serene BanerjeePh.D. Defense, April 28th, 2004 http://www.ece.utexas.edu/~serene

Committee Members: Prof. Ross Baldick

Prof. Alan C. Bovik Prof. Brian L. Evans (Advisor)

Prof. Wilson S. GeislerProf. Joydeep Ghosh

Prof. Robert W. Heath, Jr.

Computer Engineering Curriculum TrackDept. of Electrical and Computer EngineeringThe University of Texas at Austin

4/28/2004 Composition-Guided Image Acquisition 2

“One day Alice came to a fork in the road and saw a Cheshire cat in a tree.

‘Which Road do I take?’ she asked.

‘Where do you want to go?’ was his response.

‘I don’t know,’ Alice answered.

‘Then,’ said the cat, ‘it doesn’t matter.”

Lewis Carroll Alice in Wonderland


Outline

Introduction Motivation Overview of contributions Summary of previous research for main subject detection

Contributions Online main subject detection Aesthetic enhancements, given main subject Blur background objects merging with main subject

Conclusions


Motivation

Problem: Amateur photographers take unappealing pictures (e.g. personal and business use)

Help users take better pictures with digital cameras

Main subjectcropped

Too muchbackground

No foreground / background distinction


Enhance Picture Appeal

Improving photograph appeal [Savakis, Etz & Loui; 2000]

Photographic composition Objective measures People/expression

Examples of photographic composition rules

Rul

e-of

-thi

rds

Amateur Placement Professional

Blu

r ba

ckgr

ound

Amateur Shot Professional

Avo

id M

erge

r

Amateur Shot Professional


Enhance Acquired Picture Appeal Goal: Provide well-composed alternative pictures

during image acquisition in digital still cameras Solution: Framework for in-camera automation of

photographic composition rules Acquire picture user intended to take Locate main subject by combining optical and digital

image processing on a supplementary picture Apply composition rules to user-intended picture

Place main subject according to rule-of-thirds Blur entire background given main subject location Blur background objects that merge with main subject

User takes intended picture and framework also returns three alternative pictures


Offline Main Subject Detection

Neural network based training [Luo, Etz, Singhal & Gray; 2000-2001]

Cluster multi-level wavelet coefficients [Wang et al.; 1999-2001]

Iterative classification from variance maps [Won, Pyan & Gray; 2002]

Algorithm Training complexity Runtime complexity

Neural network Difficult to form widely applicable training set

High (e.g. feature extraction, grouping)

Wavelet-based No training required High (e.g. wavelet, k-means clustering)

Variance-based No training required High (e.g. iterations, watershed)


Automating Composition Rules

Detect main subject

Rule-of-thirds

Background blur

Mitigate merger

Original color image

Generated picture with

rule-of-thirds

Generated picture with

blur

Generated picture without

mergers

In-camera online framework

Provide alternatives to user during image acquisition

One-pass low-complexity algorithms [Banerjee & Evans; 2003-04]

Independent of scene content and setting

Amenable to fixed-point implementation

Match processing on digital still cameras

Supplementary picture


Digital Still Cameras

Converts optical image to electric signal

Software control Shutter aperture and speed Focus Zoom White balance

Additional hardware could control Camera angle Aspect ratio: landscape or

portrait


Outline

Introduction Contributions

Online main subject detection In-camera segmentation of the main subject Low-complexity one-pass algorithm Amenable to implementation in digital still cameras

Aesthetic enhancement, given main subject Mitigation of mergers with background objects

Conclusions


Online Main Subject Detection

Auto-focus main subject Take supplementary picture

Open shutter aperture (takes 1s) to blur objects not in focus

In-focus edges stronger than out-of-focus edges

Process supplementary picture to find main subject mask Enhance in-focus edges Detect strong edges Close boundary

Contribution #1

3x3 Highpass

filter

Detect sharper

edges

Close boundary

Auto-focus filter

Open shutter for blur

Scene

Binary main subject mask

Compute intensity

Supplementary picture


Supplementary picture has intensity function, I IH and IL are highpass and lowpass versions

For background image, contribution from IL is greater

Goal: Identify pixels contributing high frequencies I is modeled as mixture of IH and IL

Highpass filtering of I enhances main subject edges

Main Subject Detection: Formulation

LH Ik

kI

kI

11

1

Contribution #1

where k 1


Step 1: Enhance In-focus Edges

Subtract smoothed image from sharpened one Strong edges in main subject, weak edges in background

Σ

Supplementary image

Lowpass image

Highboost image

+

-

Contribution #1

Edge-enhanced image with

stronger main subject edges


Step 2: Detect Strong Edges

Canny edge detector detects strong edges [Canny; 1986] Selects weak edges only if they are connected to strong

edges

Laplacian of Gaussian detector [Burt & Adelson; 1983] Selects edges based on zero crossings of second derivative Either detects weak and strong edges or eliminates weak

edges from main subject (depends on threshold)

Contribution #1

Canny edge detector Laplacian of Gaussian


Step 3: Generate Mask

Goal: Generate closed contour from strong edges Gradient vector flow [Xu, Yezzi & Prince; 2001]

Balances forces Internal: spline characteristics External: normal of gradient of detected strong edges

Outer boundary of detected sharp edges is initial contour Change shape of initial contour, depending on gradient

Approximate lower complexity method Select leftmost & rightmost “ON” pixel and

make row pixels in between them “ON” Can detect convex regions but fails at concavities

Contribution #1


Main Subject Detection Results

Supplementary image

Step 1: Edge map

Step 2: Strong edge detection

Step 3a: Gradient of strong edges

Step 3b: Gradient vector flow field

Step 3c: Initial contour

Step 3d: Contour after 5 iterations (not mandatory)

Main subject mask

Contribution #1


Implementation Complexity

Per-pixel complexity for algorithm [Banerjee & Evans; 2003-04]

Multi-level wavelet based [Wang, Lee, Gray, Wiederhold; 1999-2001]

Variance of multi-level wavelet coefficients: ~2X increase k-means clustering: 2(image size)(no. of iterations)X increase

Iterative classification from variance maps [Won et al.; 2002]

Iterative maximum a posteriori segmentation: ~3X increase Watershed refinement: 6 passes per pixel

Contribution #1

Process/Operation Multiply-accumulates Compares Memory accesses

Pre-filtering (3x3) 9

Edge detection 9 2 5

Close boundary 2 1


Comparison With Previous MethodsOriginal image

Proposed algorithm[Banerjee & Evans; 2003-4]

Wavelet-based[Wang et al.; 1999-2001]

Variance maps[Won, Pyan & Gray; 2002]

Contribution #1


Limitations

Frequency-based features not applicable if Main subject does not have enough high frequencies Background not blurry enough

Could incorporate region-based features

Example of an image where the proposed algorithm fails to detect the main subject, the flower

Contribution #1


Outline


Main subject detection Aesthetic enhancement, given main subject

Reposition main subject to follow rule-of-thirds Simulate background blur for motion or clarity

Mitigation of mergers with background objects

Conclusions


Rule-of-Thirds

Better interaction of main subject with image background

Center of mass of main subject at 1/3 or 2/3 picture width (or height) from the left (or top) edge

Contribution #2

Main subject in center of picture

Main subject follows rule-of-thirds

Outdoor setting; the flower is main subject


Rule-of-Thirds Algorithm

Compute center-of-mass of main subject 2 multiply-accumulates, 1 memory read per pixel 1 division per image

Locate closest one-third corner 8 compares per image (4 comparisons of (x,y) points)

Shift picture so center-of-mass falls at desired corner Mirror undefined boundary pixels Best case: no change to image Worst case: 1/3 rows/columns need to be shifted Average (main subject in middle): shift 1/6 rows/columns 0 to 2 memory accesses per pixel

Contribution #2


Ideal Background Blur Example

Contribution #2

Background blur emphasizes main subject, the shell, and aids in constrained image communication

Indoor setting; no humans in picture


Simulated Background Blur

Possible camera blurs Background blur: shutter aperture Linear blur: subject or camera motion Radial blur: camera rotation Zoom: change in zoom

Digital alternatives Original image masked with detected main subject mask Region of interest filtering performed on non-masked pixels Complexity: 9 multiply-accumulates and 4 memory accesses

per pixel for convolution with symmetric 3x3 filter

Contribution #2


Results (1)

Supplementary image with main

subject(s) in focus

Detected main subject mask

Rule-of-Thirds: Main subject repositioned

Simulated background blur

Contribution #2

Outdoor setting; human main subject


Results (2)


subject(s) in focus




Contribution #2

Outdoor setting; human main subject


Results (3)


subject(s) in focus




Contribution #2

Indoor setting; no human subjects


Outline


Main subject detection Aesthetic enhancement, given main subject Mitigation of mergers with background objects

Framework for background analysis and merger detection Low-complexity one-pass algorithm for merger mitigation

Conclusions


Ideal Merger Mitigation Example

Contribution #3

Unwanted mergers avoided

Background bar merges with gymnast’s hand


Mitigation of Mergers: Overview

Goal: Identify background objects merging with main subject In-focus background object Connected to main subject mask Large area relative to image size

Merger detection Color segmentation based on hue Identify distracting background

object based on distance to main subject and frequency content

Blur merging background objects to induce a sense of distance

Contribution #3

Merging background objects: trees and bush

over right shoulder


Segmentation of Background Objects

Hues above histogram average are dominant hues Background is a mixture of dominant hues Thresholds: average of two consecutive dominant hues

Contribution #3

Background hues

Histogram of background hues and identified objects

Thresholds = {87, 151}


Merger Object Detection Define Frequency Inverse Distance Measure for

each disjoint background object Oi Decreases with nearest distance (di) from main subject Increases with high spatial frequency coefficients (ωi

H)

Merged object: Object with highest transform value

form lExponentia ),(

formDivision ),(

),(

formLinear y)(x,)),(1(

i

i

i

Oy)(x,

),(

Oy)(x,

Oy)(x,

yxdHii

i

Hi

i

Hiii

ieyx

yxd

yx

yxd

Contribution #3


Measure Selection

Linear, division, and exponential forms to combine High frequencies computed with residual in Gaussian

pyramid decomposition Euclidean distance measured from main subject mask

Attribute Linear Divisional Exponential

Computational complexity

Low High High

Merged object’s size

Large Small Small

Contribution #3


Merger Mitigation ResultsBackground tree and bush merging with main subject

High frequency and inv. distance values for

background

Blurred tree and bush appear to be farther away

Contribution #3


Per-pixel Implementation Complexity

Contribution #3

Process /Operation Multiply-accumulates

Compares Memory accesses

RGB to hue 3 6 4

Histogram and thresholding 1 2

RGB to intensity 2

Gaussian pyramid 9 4

Approx. inv. distance measure 2 1 2

Detect merged object 1 1

Gaussian pyramid reconstruction

9 1 5

TOTAL 27 11 15For comparison, JPEG compression takes ~60 operations/pixel


System Prototype

Generated picture with blur Merger mitigated picture

Measure how close rule-of-thirds

followed

Scene

Automate rule-of-thirds

Simulate background blur

Generated picture with rule-of-thirds

Binary main subject mask

Intensity Gaussian pyramid

Background segmentation

Inverse distance

transform

Grayscale image

X

Detect merging object

Grayscale image

Reconstruct color pyramid

Color Gaussian pyramid

Transform coefficients

3x3 Highpass

filter

Detect sharper

edges

Close boundary

Auto-focus filter

Open shutter for blur

Compute intensity

Original color image

Supplementary image


Conclusion

Contributions Combined optical/digital image acquisition Provide online feedback to amateur photographers Low-complexity one-pass method for main subject detection Rule-of-thirds: placement of the main subject on the canvas Simulated background blur: motion and depth-of-field Mitigation of mergers with background objects

Deliverables Prototype development for digital still image acquisition Copies of MATLAB code, slides, and papers, available at

http://www.ece.utexas.edu/~bevans/projects/dsc/index.html


Future Work

Automate other photographic composition rules Best zoom Available frames, lines of interest, best angle, balanced picture

Extension for video acquisition Frame-by-frame basis Compressed domain

Digital image stabilization: Subject mask as feature Potential research impact:

Video cameras, Surveillance, Image/video retrieval, Constrained image/video communication, Main subject detection for specific applications

Date post:	24-Dec-2015
Category:	Documents
Upload:	brian-oscar-mcdonald
View:	222 times
Download:	1 times

Composition-Guided Image Acquisition Serene Banerjee Ph.D. Defense, April 28 th, 2004 serene...

Documents