+ All Categories
Home > Documents > Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the...

Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the...

Date post: 29-Dec-2015
Category:
Upload: bathsheba-owens
View: 214 times
Download: 1 times
Share this document with a friend
89
Ordinary Least-Squares Emmanuel Iarussi Inria
Transcript
Page 1: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

Ordinary Least-Squares

Emmanuel IarussiInria

Page 2: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

2

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

Surface reconstruction

Page 3: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

3

Color diffusion

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

Page 4: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

4

Shape registration

Many graphics problems can be seen as finding the best set of parameters for a model, given some data

Page 5: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

5

Temperature (a) Gas pressure (b)

Example: estimation of gas pressure for a given temperature sample

Page 6: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

6

Assuming linear relationship

Page 7: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

7

Assuming linear relationship

Page 8: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

8

Assuming linear relationship

Temperature measurements (m)

Pressure measurements (m)

Page 9: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

9

Assuming linear relationship

Temperature measurements (m)

Pressure measurements (m)

=

Page 10: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

10

Assuming linear relationship

The relationship might not be exact

Temperature measurements (m)

Pressure measurements (m)

=

Page 11: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

11

Assuming linear relationship

The relationship might not be exact

Temperature measurements (m)

Pressure measurements (m)

=

Match “as best as possible” the observations

Page 12: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

12

Assuming linear relationship

The relationship might not be exact Match “as best as possible” the observations

Temperature measurements (m)

Pressure measurements (m)

=

Page 13: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

13

Many of CG problems can be formulated as minimizing the sum of squares of the residuals

between some features in the model and the data.

or

Page 14: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

14

Matrix notation (observations)

2

2

)(

)()(

)()(

ab

abab

xxe

xx

xabxe

T

iii

We can rewrite the residual function using linear algebra as:

Page 15: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

15

Temperature

Gas pressure

Example: estimation of gas pressure for a given temperature and altitude samples

Altitude

Page 16: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

16

Multidimensional linear regression, using a model with n parameters

Page 17: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

17

Multidimensional linear regression, using a model with n parameters

before+ +…

Page 18: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

18

Multidimensional linear regression, using a model with n parameters

Temperature measurements (m)

Pressure measurements (m)

Altitudemeasurements (m)

Page 19: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

19

Multidimensional linear regression, using a model with n parameters

Temperature measurements (m)

Pressure measurements (m)

Altitudemeasurements (m)

=

Page 20: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

20

=

Page 21: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

21

=

Page 22: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

22

=

Page 23: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

23

Objective: find x subject to minimize:

Page 24: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

24

Objective: find x subject to minimize:

Page 25: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

25

Objective: find x subject to minimize:

Convex bowl function has minima when:

Page 26: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

26

Objective: find x subject to minimize:

Convex bowl function has minima when:

gradient

Page 27: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

27

Expanding square term

Page 28: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

28

Expanding square term

Page 29: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

29

Differentiating with respect to x (gradient)

Expanding square term

Page 30: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

30

Expanding square term

Differentiating with respect to x (gradient)

Page 31: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

31

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Differentiating with respect to x (gradient)

Page 32: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

32

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Issue: matrix multiplication

Differentiating with respect to x (gradient)

Page 33: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

33

Differentiating with respect to x

Expanding square term

Normal equation

x = (A.transpose()*A).solve(A.transpose()*b);

Issue: matrix multiplication Solution: find expression

Page 34: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

34

Let's go back to gradient = 0

Page 35: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

35

Let's go back to gradient = 0

Differentiating again with respect to x

Page 36: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

36

Let's go back to gradient = 0

Differentiating again with respect to x

Page 37: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

37

Let's go back to gradient = 0

Differentiating again with respect to x

hessian

Page 38: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

38

Let's go back to gradient = 0

Differentiating again with respect to x

hessian

Page 39: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

39

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

Normal equation:

Page 40: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

40

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

Normal equation:

Page 41: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

41

Let's go back to gradient = 0

Differentiating again with respect to x

x = (A.transpose()*A).solve(A.transpose()*b);

x = (H).solve(c);

Two alternatives for solving:

Constant terms in gradient

Page 42: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

42

Example: Diffusion

Page 43: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

43

Example: Diffusion

Minimize difference between neighbors

Page 44: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

44

Example: Diffusion

Minimize difference between neighbors

Page 45: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

45

Example: Diffusion

Minimize difference between neighbors

Page 46: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

46

Example: Diffusion

Minimize difference between neighbors

Page 47: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

47

Solving with normal equation:

1. Build A

Page 48: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

48

Solving with normal equation:

1. Build A

Page 49: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

49

Solving with normal equation:

1. Build A

Page 50: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

50

Solving with normal equation:

1. Build A

Page 51: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

51

Solving with normal equation:

1. Build A

Page 52: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

52

Solving with normal equation:

1. Build A

Page 53: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

53

Solving with normal equation:

2. Compute and

Page 54: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

54

Solving with normal equation:

3. Solve using:• Cholesky decomposition• Conjugate Gradient• …

x = (A.transpose()*A).solve(A.transpose()*b);

in Eigen (i.e):

Page 55: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

55

Solving with normal equation:

3. Solve using:• Cholesky decomposition• Conjugate Gradient• …

x = (A.transpose()*A).solve(A.transpose()*b);

in Eigen (i.e):

x wil have:

Page 56: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

56

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 57: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

57

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 58: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

58

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 59: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

59

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 60: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

60

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 61: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

61

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 62: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

62

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 63: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

63

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 64: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

64

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 65: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

65

Solving with Hessian:

Page 66: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

66

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 67: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

67

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 68: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

68

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 69: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

69

Solving with Hessian:

1. Gradient vector: first-order partial derivatives of energy terms

Page 70: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

70

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 71: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

71

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 72: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

72

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 73: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

73

Solving with Hessian:

2. Hessian matrix: second-order partial derivatives of energy terms

Page 74: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

74

Solving with Hessian:

Page 75: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

75

Solving with Hessian:

To recap:

Page 76: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

76

Solving with Hessian:

3. Build system

Page 77: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

77

Solving with Hessian:

3. Build system

Page 78: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

78

Solving with Hessian:

3. Build system: add constraint

Page 79: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

79

Solving with Hessian:

3. Build system: add constraint

Page 80: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

80

Solving with Hessian:

3. Build system: add constraint

Page 81: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

81

Solving with Hessian:

3. Build system: add constraint

Page 82: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

82

Solving with Hessian:

3. Build system: add constraint

Page 83: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

83

Solving with Hessian:

3. Build system: add constraint

Page 84: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

84

Solving with Hessian:

3. Build system: add constraint

Page 85: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

85

Solving with Hessian:

3. Build system: add constraint

Page 86: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

86

Solving with Hessian:

3. Build system: add constraint

Page 87: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

87

Solving with Hessian:Solving with Hessian:

4. Solve using:• Cholesky decomposition• Conjugate Gradient• …

Page 88: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

88

Solving with Hessian:Solving with Hessian:

x = (H).solve(b`);

in Eigen (i.e):

x will have:

4. Solve using:• Cholesky decomposition• Conjugate Gradient• …

Page 89: Ordinary Least-Squares Emmanuel Iarussi Inria. Many graphics problems can be seen as finding the best set of parameters for a model, given some data Surface.

89

References:

• Solving Least Squares Problems. C. L. Lawson and R. J. Hanson. 1974.

• Practical Least Squares for Computer Graphics. Fred Pighin and J.P. Lewis. Course notes 2007.


Recommended