Chapter 1Divide and Conquer
Part 2: Polynomial Multiplication
Algorithm TheoryWS 2013/14
Fabian Kuhn
Algorithm Theory, WS 2013/14 Fabian Kuhn 2
PolynomialsReal polynomial in one variable :
… 1 0
Coefficients of : , , … , ∈Degree of : largest power of in ( in the above case)
Example:
3 3– 15 2 18
Set of all real‐valued polynomials in : (polynomial ring)
Algorithm Theory, WS 2013/14 Fabian Kuhn 3
Operations: Addition
• Given: Polynomials , ∈ of degree
⋯
⋯
• Compute sum :
⋯ ⋯ ⋯
Algorithm Theory, WS 2013/14 Fabian Kuhn 4
Operations: Multiplication• Given: Polynomials , ∈ of degree
⋯
⋯
• Product ⋅ :
⋅ ⋯ ⋅ ⋯ ⋯
• Obtaining : what products of monomials have degree ?
For0 2 :
where 0 for .
Algorithm Theory, WS 2013/14 Fabian Kuhn 5
Operations: Evaluation• Given: Polynomial ∈ of degree
⋯
• Horner’s method for evaluation at specific value :
… ⋯
• Pseudo‐code:≔ ; ≔ ;
while 0 do≔ 1;≔ ⋅
end
• Running time:
Algorithm Theory, WS 2013/14 Fabian Kuhn 6
Representation of PolynomialsCoefficient representation:
• Polynomial ∈ of degree is given by its1 coefficients , … , :
⋯
• Example:3 15 18
• The most typical (and probably most natural) representation of polynomials
Algorithm Theory, WS 2013/14 Fabian Kuhn 7
Representation of PolynomialsProduct of linear factors:
• Polynomial ∈ of degree is given by its roots
⋅ ⋅ ⋅ … ⋅
• Example:3 2 3
• Every polynomial has exactly roots ∈ for which 0– Polynomial is uniquely defined by the roots and
• We will not use this representation…
Algorithm Theory, WS 2013/14 Fabian Kuhn 8
Representation of PolynomialsPoint‐value representation:
• Polynomial ∈ of degree is given by1 point‐value pairs:
, , , , … , ,
where for .
• Example: The polynomial
3 2 3is uniquely defined by the four point‐value pairs 0,0 , 1,6 , 2,0 , 3,0 .
Algorithm Theory, WS 2013/14 Fabian Kuhn 9
Operations: Coefficient RepresentationDeg.‐ polynomials ⋯ , ⋯
Addition:⋯
• Time:
Multiplication:
⋅ ⋯ , where
• Naive solution: Need to compute product for all 0 ,
• Time:
Algorithm Theory, WS 2013/14 Fabian Kuhn 10
Operations Point‐Value RepresentationDegree‐ polynomials
, , … , , , , , … , ,
• Note: we use the same points , … , for both polynomials
Addition:
, , … , ,
• Time:
Multiplication:
⋅ , ⋅ , … , , ⋅
• Time:
Algorithm Theory, WS 2013/14 Fabian Kuhn 11
Faster Multiplication?• Multiplication is slow Θ when using the standard
coefficient representation
• Try divide‐and‐conquer to get a faster algorithm
• Assume: degree is 1, is even• Divide polynomial ⋯ into 2
polynomials of degree ⁄ 1:
⋯⋯
⋅
• Similarly: ⋅ ⁄
Algorithm Theory, WS 2013/14 Fabian Kuhn 12
Use Divide‐And‐Conquer• Divide:
⋅ , ⋅
• Multiplication: ⋅ ⋅
• 4 multiplications of degree ⁄ 1 polynomials:
4 2
• Leads to Θ like the naive algorithm… (see exercises)
Algorithm Theory, WS 2013/14 Fabian Kuhn 13
More Clever Recursive Solution• Recall that ⋅ ⋅
• Compute ⋅ :
Algorithm Theory, WS 2013/14 Fabian Kuhn 14
Karatsuba Algorithm• Recursive multiplication:
⋅⋅
⋅
• Recursively do 3 multiplications of degr. ⁄ 1 ‐polynomials
3 2
• Gives: . (see exercises)
Algorithm Theory, WS 2013/14 Fabian Kuhn 15
Faster Polynomial Multiplication?Multiplication is fast when using the point‐value representation
Idea to compute ⋅ (for polynomials of degree ):
, of degree 1, coefficients
2 2 point‐value pairs , and ,
2 point‐value pairs ,
of degree 2 2, 2 1 coefficients
Evaluation at points , , … ,
Point‐wise multiplication
Interpolation
Algorithm Theory, WS 2013/14 Fabian Kuhn 16
Point‐Value Representation of , • Select points , , … , to evaluate and in a clever way
Consider the powers of the principle th root of unity:
Principle root of unity:
1, 1
Powers of (roots of unity):
1 , , … ,
Note: cos ⋅ sin
Algorithm Theory, WS 2013/14 Fabian Kuhn 17
Discrete Fourier Transform• The values for 0,… , 1 uniquely define a
polynomial of degree .
Discrete Fourier Transform (DFT):• Assume , … , is the coefficient vector of poly.
⋯
DFT ≔ , ,… ,
Algorithm Theory, WS 2013/14 Fabian Kuhn 18
Example• Consider polynomial 3 15 18
• Choose 4
• Roots of unity:
Algorithm Theory, WS 2013/14 Fabian Kuhn 19
Example• Consider polynomial 3 15 18
• 4, roots of unity: 1, , 1,
• Evaluate at :
, 1, 1 1,6
, , , 15 15
, 1, 1 1, 36
, , , 15 15
• For 3, 15,18,0 :, , ,
Algorithm Theory, WS 2013/14 Fabian Kuhn 20
Faster Polynomial Multiplication?
Idea to compute ⋅ (for polynomials of degree ):
, of degree 1, coefficients
2 2 point‐value pairs , and ,
2 point‐value pairs ,
of degree 2 2, 2 1 coefficients
Evaluation at points , , … ,
Point‐wise multiplication
Interpolation
Algorithm Theory, WS 2013/14 Fabian Kuhn 21
Properties of the Roots of Unity• Cancellation Lemma:
For all integers 0, 0, and 0, we have:
• Proof:
Algorithm Theory, WS 2013/14 Fabian Kuhn 22
Divide‐and‐Conquer Approach• Divide of degree 1 ( is even) into 2 polynomials of
degree ⁄ 1 differently than in Karatsuba’s algorithm
• ⋯ ⁄ (even coeff.)⋯ ⁄ (odd coeff.)
Algorithm Theory, WS 2013/14 Fabian Kuhn 23
Discrete Fourier TransformEvaluation for 0,… , 1:
⋅
⁄ ⋅ ⁄ if 2
⁄⁄ ⋅ ⁄
⁄ if 2
For the coefficient vector of :
DFT ⁄ ,… , ⁄⁄ , ⁄ , … , ⁄
⁄
⁄ , … , ⁄⁄⁄ , ⁄
⁄ , … , ⁄⁄
Algorithm Theory, WS 2013/14 Fabian Kuhn 24
ExampleFor the coefficient vector of :
DFT ⁄ ,… , ⁄⁄ , ⁄ , … , ⁄
⁄
⁄ , … , ⁄⁄⁄ , ⁄
⁄ , … , ⁄⁄
4:
Need: , and ,
(DFTs of coefficient vectors of and )
Algorithm Theory, WS 2013/14 Fabian Kuhn 25
Recursive StructureFor simplicity, we abuse notation in the following:
• Poly. ⋯ with coefficient vector Let DFT ≔ DFT
Recursive structure: • For 4:
DFT DFT ⋅ DFT
• General (assume is even): DFT
DFT /
⋅ DFT /
Algorithm Theory, WS 2013/14 Fabian Kuhn 26
Computation of • Divide‐and‐conquer algorithm for DFT :
1. Divide
1: DFT
1: Divide into (even coeff.) and (odd coeff).
2. Conquer
Solve DFT ⁄ and DFT ⁄ recursively
3. Combine
Compute DFT based on DFT ⁄ and DFT ⁄
Algorithm Theory, WS 2013/14 Fabian Kuhn 27
Analysis• : max. time to compute DFT :
2 2 , 1 1
• As for mergesort, comparing orders, closest pair of points:
⋅ log
Algorithm Theory, WS 2013/14 Fabian Kuhn 28
Small ImprovementPolynomial of degree 1:
⁄ ⋅ ⁄ if 2
⁄⁄ ⋅ ⁄
⁄ if 2
⁄ ⋅ ⁄ if 2
⁄⁄ ⁄ ⋅ ⁄
⁄ if 2
Need to compute ⁄ and ⋅ ⁄ for 0 ⁄ .
Algorithm Theory, WS 2013/14 Fabian Kuhn 29
Example
⋅
⋅
⋅
⋅
Algorithm Theory, WS 2013/14 Fabian Kuhn 30
Fast Fourier Transform (FFT) AlgorithmAlgorithm FFT(a)• Input: Array of length , where is a power of 2• Output: DFT
if 1 then return ; // ≔ FFT , , … , ;≔ FFT , , … , ;≔ ; ≔ 1;
for 0 to ⁄ 1 do //
≔ ⋅ ;≔ ; ⁄ ≔ ;≔ ⋅
end;return , , … , ;
Algorithm Theory, WS 2013/14 Fabian Kuhn 31
Example• 3 15 18 0, 0,18, 15,3
Algorithm Theory, WS 2013/14 Fabian Kuhn 32
Faster Polynomial Multiplication?
Idea to compute ⋅ (for polynomials of degree ):
, of degree 1, coefficients
2 2 point‐value pairs , and ,
2 point‐value pairs ,
of degree 2 2, 2 1 coefficients
Evaluation at , , … , using FFT
Point‐wise multiplication
Interpolation
Algorithm Theory, WS 2013/14 Fabian Kuhn 33
InterpolationConvert point‐value representation into coefficient representation
Input: , , … , , with for
Output: Degree‐ 1 polynomial with coefficients , … , such that
⋯ ⋯
⋮⋮⋯
linear system of equations for , … ,
Algorithm Theory, WS 2013/14 Fabian Kuhn 34
InterpolationMatrix Notation:
1 ⋯1 ⋯⋮ ⋮ ⋱ ⋮1 ⋯
⋅ ⋮ ⋮
• System of equations solvable iff for all
Special Case :
1 1 1 ⋯ 11 ⋯1 ⋯⋮ ⋮ ⋮ ⋱ ⋮1 ⋯
⋅⋮ ⋮
Algorithm Theory, WS 2013/14 Fabian Kuhn 35
Interpolation• Linear system:
⋅ ⟹ ⋅
, , ⋮ , ⋮
Claim:
Proof: Need to show that
Algorithm Theory, WS 2013/14 Fabian Kuhn 36
DFT Matrix Inverse
⋯ 1
⋯ ⋮ ⋯
⋅
⋯ 1 ⋯⋯ ⋯⋯ ⋯ ⋮ ⋯ ⋯
Algorithm Theory, WS 2013/14 Fabian Kuhn 37
DFT Matrix Inverse
,
ℓ
ℓ
Need to show ,1if0if
Case :
Algorithm Theory, WS 2013/14 Fabian Kuhn 38
DFT Matrix Inverse
,
ℓ
ℓCase :
Algorithm Theory, WS 2013/14 Fabian Kuhn 39
Inverse DFT
•
⋯
⋯ ⋮ ⋯
• We get ⋅ and therefore
1⋯ ⋅ ⋮
1⋅ ⋅
Algorithm Theory, WS 2013/14 Fabian Kuhn 40
DFT and Inverse DFTInverse DFT:
1⋅ ⋅
• Define polynomial ⋯ :
1⋅
DFT:• Polynomial ⋯ :
Algorithm Theory, WS 2013/14 Fabian Kuhn 41
DFT and Inverse DFT
⋯ , ⋅ :
• Therefore:, , … ,
1⋅ , , , … ,
1⋅ , , , … ,
• Recall:DFT , , , … ,
⋅ , , , … , ,
Algorithm Theory, WS 2013/14 Fabian Kuhn 42
DFT and Inverse DFT• We have DFT ⋅ , , , … , , :
DFT if 0
DFT if 0
• DFT and inverse DFT can both be computed using FFT algorithm in log time.
• 2 polynomials of degr. can be multiplied in time log .
Algorithm Theory, WS 2013/14 Fabian Kuhn 43
Faster Polynomial Multiplication?
Idea to compute ⋅ (for polynomials of degree ):
, of degree 1, coefficients
2 2 point‐value pairs , and ,
2 point‐value pairs ,
of degree 2 2, 2 1 coefficients
Evaluation at , , … , using FFT
Point‐wise multiplication
Interpolation using FFT
Algorithm Theory, WS 2013/14 Fabian Kuhn 44
Convolution• More generally, the polynomial multiplication algorithm
computes the convolution of two vectors:
, , … ,, , … ,
∗ , , … , ,
where, :
,
• is exactly the coefficient of in the product polynomial of the polynomials defined by the coefficient vectors and
Algorithm Theory, WS 2013/14 Fabian Kuhn 45
More Applications of ConvolutionsSignal Processing Example:• Assume , … , represents a sequence of
measurements over time• Measurements might be noise and have to be smoothed out• Replace by weighted average of nearby last and next
measurements (e.g., Gaussian smoothing):
1⋅
• New vector ′ is the convolution of and the weight vector1⋅ , , … , , 1, , … , ,
• Might need to take care of boundary points…
Algorithm Theory, WS 2013/14 Fabian Kuhn 46
More Applications of ConvolutionsCombining Histograms:• Vectors and represent two histograms• E.g., annual income of all men & annual income of all women
• Goal: Get new histogram representing combined income of all possible pairs of men and women:
∗
Also, the DFT (and thus the FFT alg.) has many other applications!