Agenda

• Project 2- Due this Thursday• Office Hours Wed 10:30-12• Image blending• Background– Constrained optimization

Recall: goal

Formulation: find the best patch f

• Given vector field v (pasted gradient), find the value of f in unknown region that optimize:

Pasted gradient Mask

Background

unknownregion

Notation• Destination image: f* (table)• Source image: g (table)• Output image: f (table)• W: list of (i,j) pixel coordinates from f* we want to replace• dW: list of (i,j) pixel coordinates on border of W• We’ll use p = (i,j) to denote a pixel location

– gp is a pixel value at p = (i,j) from source image,

– f W is the set of pixels we’re trying to find

Notation• Destination image: f* (table)• Source image: g (table)• Output image: f (table)• W: set of (i,j) pixel coordinates from f we want to replace (list of pairs)• dW: set of (i,j) pixel coordinates on border of W (list of pairs) • We’ll use p = (i,j) to denote a pixel location

– gp is a pixel value at p = (i,j) from source image,

– f W is the set of pixels we’re trying to find

With constraint that, for p in dW

sum over all pairs of neighbors in W

Optimization

What is optimal fW without above constraint?

What is known versus unknown?

Variational formulation of solution:The best patch is the one that produces the lowest score, subject to the constraint

Drop subscript

for all p in dOmega

Optimization

Pretend constraint wasn’t there: how to find lowest scoring fW?

1) Brute-force search-Keep guessing different patches f and score them

-Output the best-scoring one

2) Gradient descent-Guess a patch f. Update guess with f = f -

How to estimate gradient?

In general, we can always do it numerically

For above quadratic function, we can calculate in closed form

How to estimate gradient?

In general, we can always do it numerically

For above quadratic function, we can calculate in closed form

Constrained optimization

1) Brute-force search-Keep guessing different patches f and score them

-Output the best-scoring one

2) Gradient descent-Guess a patch f. Update guess with f = f -

What happens when gradient is zero?

Optimization

1) Brute-force search-Keep guessing different patches f and score them

-Output the best-scoring one

2) Gradient descent-Guess a patch f. Update guess with f = f –

3) Closed-form solution (for simple functions)

Constrained optimization

How to handle constraints?

1) Brute-force search-Keep guessing different patches f and score them

-Output the best-scoring one

2) Gradient descent-Guess a patch f. Update guess with f = f -

Correct fp = f*p after a gradient update

Constrained optimization

How to handle constraints?

1) Brute-force search-Keep guessing different patches f and score them

-Output the best-scoring one

2) Gradient descent-Guess a patch f. Update guess with f = f -

What happens when gradient is zero?

Lagrangian optimization

• If there was no constraint, we’d have a closed-form solution

• Is there a way to get closed-form solutions using the constraint?

Lagrangian optimizationmin f(x,y) such that g(x,y) = 0

Imagine we want to synthesize a “two-pixel” patch

Lagrangian optimizationmin f(x,y) such that g(x,y) = 0

and g(x,y) = 0

Write conditions with single equation(just for convenience)

At minimum of F, the its gradient is 0

Therefore, the following conditions hold

Multiple constraintsmin f(x,y) such that g1(x,y) = 0, g2(x,y) = 0

What is f(x,y) in our case? g1(x,y)?

Lagrangian optimization

for p in dW (border pixels)

for all other p in W

Since S is quadratic in f, the above yeilds a set of linear equationsAf =b

f = inv(A)b