Reed-Muller codes for random errors and erasures

Based on:

Abbe-S-WigdersonSaptharishi-S-Volk

Reed-Muller code - RM(m,r)

Message: coefficient vector of a polynomial f(x1,…,xm) over 𝔽2 of degree ≤ r

Encoding: evaluations of f: f → (f(000),f(001),…,f(111))

Distance: Hamming distance between any two code words is ≥ 2m-r

Note: codewords form a linear space over 𝔽2

The Evaluation matrix (Sierpinski matrix)

{0,1}m

m-variate monomials in lexicographical order:

1, x1,x2,x1x2,x3,x1x3,…

1x1x2x1x2x3x1x3x2x3x1x2x3x4

00!0

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

11!1

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

1x1x2x1x2x3x1x3

Keep rows corresponding to low degree monomials (heaviest rows)

The Evaluation matrix (Sierpinski matrix)

Generating matrix of Reed-Muller code:codewords are linear combination of rows

1x1x2x3x4x5

{0,1}m00!0

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

11!1

⎛

⎝

⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟

1x1x2x1x2x3x1x3x2x3x1x2x3x4

Keep rows corresponding to low degree monomials (heaviest rows)

Reed-Muller code - RM(m,r)

Message: coefficient vectors of a polynomial f(x1,…,xm) over 𝔽2 of degree ≤ r

Encoding: evaluations of f: f → (f(000),f(001),…,f(111))

Minimum distance: d=2m-r

Most studied linear algebraic code

Around over 50 years yet fundamental questions are still open!

Why care about RM codes?Low degree polynomials ubiquitous in TCS:

lower bounds

derandomization

PCP

Hardness amplification

List decoding

Algorithms

Property testing

Extractors

…

Decoding RM codes

Decoding problem: decode corrupted codewords

Worst case behavior well understood:

Reed: Efficient decoding up to half min. distance

Gopalan-Klivans-Zuckerman, Bhowmick-Lovett: List decoding radius ≤ 2·dist

Can we do better for random errors?

Average case study of errors

50 years old open problem in coding theory:

How good are RM codes?

Do RM codes meet Shannon’s bounds for random errors and erasures?

Error Model ⟷ Channels

Binary Erasure Channel: symbol is replaced with ? (erased) with probability p

0 1 1 0 1 0 0 0 1 0

0 1 ? ? 1 0 ? 0 1 0

BEC

Error Model ⟷ Channels

Binary Erasure Channel: symbol is replaced with ? (erased) with probability p

Binary Symmetric Channel: symbol is flipped with probability p

0 1 1 0 1 0 0 0 1 0

0 1 0 1 1 0 1 0 1 00 1 0 1 1 0 1 0 1 0

BSC

Error Model ⟷ Channels

Binary Erasure Channel: symbol is replaced with ? (erased) with probability p

Binary Symmetric Channel: symbol is flipped with probability p

Shannon: maximal rate that enables decoding w.h.p. (capacity of channel) - best tradeoff between redundancy and robustness

BEC: R = 1 - p

BSC R = 1 - h(p) (h(x) = -xlog(x)-(1-x)log(1-x))

Major goal: design explicit codes that meet Shannon’s bound (with efficient encoding and decoding)

Average case behavior of RMDo Reed-Muller codes meet Shannon’s bound?I.e., can RM codes of rate R handle the same fraction of errors/erasures that random codes of rate R handle.

Problem related to

Rank of random evaluation matrices

Spaces of tensors over 𝔽2

Polynomial interpolation from noisy data

Sparse recovery

Learning

Polar codes

…

Polar Codes

Introduced by Arikan 2009 and now get all the rage in the coding theory community

Very similar to RM - messages are polynomials with respect to a different monomial basis (different choice of rows from the Serpiensky matrix)

Achieve capacity for all channels!

Due to strange basis (no simple description), Polar codes seem less natural than RM codes, yet are still morally similar

It would be much nicer to work with RM codes rather than Polar!

Our Results

Theorem 1 [ASW]: RM achieve capacity for BEC at low rate

Theorem 2 [ASW]: RM achieve capacity for BSC at low rate

Theorem 3 [ASW]: RM achieve capacity for BEC at high rate

Theorem 4 [ASW]: RM decodable from “many” errors at high rate (super polynomial in min. distance and polynomially smaller than what capacity achieving code can do)

Theorem 4’ [ASW,SSV]: If RM(m,m-r-1) achieves capacity for BEC, then can efficiently correct many errors in RM(m,m-2r-2).

Important tool: understanding weight distribution of RM codes (how many codewords of any given weight are there)

Theorem 5 [ASW]: Tighter bounds on weight distribution of RM codes (based upon and improves Kaufman-Lovett-Porat)

m≤ r

⎛⎝⎜

⎞⎠⎟

Results [ASW]

Note: minimum distance of RM(m,m-s) is 2s

Prior work: r=1 (folklore). r=2 (Helleseth, Klove, Levenshtein ’05)

Kumar-Pfister, Kudekar-Mondelli-Sasoglu-Urbanke 15: RM achieve capacity for the BEC at constant Rate !

BEC BSC

r = o(m) √ √

√√(# of errors)

r = m −O m / log(m)( )≈ m

r⎛⎝⎜

⎞⎠⎟

Acknowledgement

We would like thank Avi Wigderson, Emmanuel Abbe and Ilya Dumer for helpful discussionsand for commenting on an earlier version of the paper. We thank Venkatesan Guruswami andanonymous reviewers for pointing out the abstraction of Algorithm 1 given in the full version.

m/20 m

o(m) o(p(m/ log m))O(

pm)

Figure 1: Regime of r for which RM(m, r) is known to achieve capacity for the BEC

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

r = r/m, relative degree of RM(m, r)

num

ber

ofer

rors

corr

ecte

d

1 � r

h⇣

1�r2

⌘

Figure 2: The number of correctable errors as a function of the degree, in logarithmic scale, normalized by m. The redline corresponds to the algorithm given in [Dum06], which corrects ⇡ d log d ⇡ 2(1�r)m · (1 � r)m errors. The bluecurve corresponds to the algorithm given in this work, which corrects ⇡ 2h( 1�r

2 )m errors.

References

[ALM+98] Sanjeev Arora, Carsten Lund, Rajeev Motwani, Madhu Sudan, and Mario Szegedy.Proof Verification and the Hardness of Approximation Problems. J. ACM, 45(3):501–555, 1998.

[Arı08] Erdal Arıkan. A Performance Comparison of Polar Codes and Reed-Muller Codes.IEEE Communications Letters, 12(6):447–449, 2008.

[AS92] Noga Alon and Joel H. Spencer. The Probabilistic Method. John Wiley, 1992.

11

m/20 m

log m log m

o(p

m)

o(p

m/ log m)

Degree (r) of RM(m, r):

(see also Figure 2)

[Dum04, DS06, Dum06]:⇡ n/2 errors O(d log d) = O(2(1�r)m · (1 � r)m) errors for r = r/m

O(n log n) time algorithm

Our results:⇡ n/2 errors

O(n4) time algo.

(d log d)O(log m) errors

n1+o(1) time algo.

2h⇣

1�r2

⌘m errors for r = r/m

assuming RM achieve capacity for all rates

Figure 3: Comparison with [Dum04, DS06, Dum06]

[ASW15] Emmanuel Abbe, Amir Shpilka, and Avi Wigderson. Reed-Muller Codes for RandomErasures and Errors. In Proceedings of the 47th Annual ACM Symposium on Theory ofComputing (STOC 2015), pages 297–306, 2015. Pre-print available at arXiv:1411.4590.

[BF90] Donald Beaver and Joan Feigenbaum. Hiding instances in multioracle queries. InProceedings of the 7th Symposium on Theoretical Aspects of Computer Science (STACS 1990),pages 37–48, 1990.

[BFL91] Laszlo Babai, Lance Fortnow, and Carsten Lund. Non-Deterministic Exponential Timehas Two-Prover Interactive Protocols. Computational Complexity, 1:3–40, 1991. Prelim-inary version in the 31st Annual IEEE Symposium on Foundations of Computer Science(FOCS 1990).

[BL15] Abhishek Bhowmick and Shachar Lovett. The List Decoding Radius of Reed-MullerCodes over Small Fields. In Proceedings of the 47th Annual ACM Symposium on Theory ofComputing (STOC 2015), pages 277–285, 2015. Pre-print available at eccc:TR14-087.

[BV10] Andrej Bogdanov and Emanuele Viola. Pseudorandom Bits for Polynomials. SIAMJ. Comput., 39(6):2464–2486, 2010. Preliminary version in the 48th Annual IEEESymposium on Foundations of Computer Science (FOCS 2007). Pre-print available ateccc:TR14-081.

[CF07] Daniel J. Costello, Jr. and G. David Forney, Jr. Channel coding: The road to channelcapacity. Proceedings of the IEEE, 95(6):1150–1177, 2007.

[DK94] Iwan M. Duursma and Ralf Kotter. Error-locating pairs for cyclic codes. IEEE Transac-tions on Information Theory, 40(4):1108–1121, 1994.

[DS06] Ilya Dumer and Kirill Shabunov. Recursive error correction for general Reed-Mullercodes. Discrete Applied Mathematics, 154(2):253–269, 2006.

[Dum04] Ilya Dumer. Recursive decoding and its performance for low-rate Reed-Muller codes.IEEE Transactions on Information Theory, 50(5):811–823, 2004.

12

The algorithm

Enter linear algebraLet U = {u1,…,ut} ⊆ {0,1}m be the unknown locations of errors.

v ∈ U iff the following linear system, in undetermined coefficients of degree r polynomial f, is solvable:

For every monomial M of degree at most r:

∑i f(ui) = f(v) = 1 (f not trivial)

∑i M(ui)·f(ui) = M(v) (vr in span of {uir})

∑i M(ui)·xj(ui)·f(ui) = M(v) (if vj = 1)

∑i M(ui)·(1-xj)(ui)·f(ui) = M(v) (if vj = 0)

Last two equations: for each j, v spanned by those that agree with it on j’th coordinate

Crucial: Coefficients of equations can be computed given corrupted word!

Parity Check Matrix

Parity Check Matrix

H such that v in Code iff Hv = 0

H is generating matrix for the dual code C⊥ = {u : u⊥v for all v in C}

Fact RM(m,r)⊥ = RM(m,m-r-1)

I.e., P.C.M is an evaluation matrix (monomials vs. points)

Example PCM of RM(4,1):

1x1x2x3x4x1x2x1x3x1x4x2x3x2x4x3x4

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 11 1 1 1 1 1 1 1

1 1 1 1 1 1 1 11 1 1 1

1 1 1 11 1 1 1

1 1 1 11 1 1 1

1 1 1 1

⎛

⎝

⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟⎟

0000 0001 0010 0011 0100 0101 0110 0111 1000 1001 1010 1011 1100 1101 1110 1111

Dual of RM(4,1) is RM(4,4-1-1)=RM(4,2)v

v2

Correcting erasures and PCM

Number of erasures slightly less than number of rowsUnique solution (unique decoding) iff corresponding sub-matrix has full column-rank.

PCM0??11?0?

⎛

⎝

⎜⎜⎜⎜⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟⎟⎟⎟⎟

× =00000

⎛

⎝

⎜⎜⎜⎜⎜

⎞

⎠

⎟⎟⎟⎟⎟

Want to solve linear system

Back to the algorithm

Let U = {u1,…,ut} ⊆ {0,1}m be the unknown locations of errors.

v ∈ U iff the following linear system, in undetermined coefficients of degree r polynomial f, is solvable:

For every monomial M of degree at most r:

∑i f(ui) = f(v) = 1 (f not trivial)

∑i M(ui)·f(ui) = M(v) (vr in span of {uir})

∑i M(ui)·xj(ui)·f(ui) = M(v) (if vj = 1)

∑i M(ui)·(1-xj)(ui)·f(ui) = M(v) (if vj = 0)

Last two equations: for each j, v spanned by those that agree with it on j’th coordinate

Crucial: Coefficients of equations can be read from Hy !!

SummaryAbbe-S-Wigderson: RM codes achieve capacity for BEC for rates close to 0 or 1 and for BSC at rate close to 0

First improvement on a 50 years old problem

Kumar-Pfister, Kudekar-Mondelli-Sasoglu-Urbanke: RM achieve BEC capacity for constant rate

Saptharishi-S-Volk: efficient algorithm for correcting most error patterns of weight t in RM(m,m-2t)

Open problem: Do RM codes achieve capacity for the BSC?

Most interesting case: what can be said for the BSC for constant rate (degree m/2 ± O(√m))?

Open problem: Do RM codes achieve capacity for the BEC for all rates?

Acknowledgement

We would like thank Avi Wigderson, Emmanuel Abbe and Ilya Dumer for helpful discussionsand for commenting on an earlier version of the paper. We thank Venkatesan Guruswami andanonymous reviewers for pointing out the abstraction of Algorithm 1 given in the full version.

m/20 m

o(m) o(p(m/ log m))O(

pm)

Figure 1: Regime of r for which RM(m, r) is known to achieve capacity for the BEC

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

r = r/m, relative degree of RM(m, r)

num

ber

ofer

rors

corr

ecte

d

1 � r

h⇣

1�r2

⌘

Figure 2: The number of correctable errors as a function of the degree, in logarithmic scale, normalized by m. The redline corresponds to the algorithm given in [Dum06], which corrects ⇡ d log d ⇡ 2(1�r)m · (1 � r)m errors. The bluecurve corresponds to the algorithm given in this work, which corrects ⇡ 2h( 1�r

2 )m errors.

References

[ALM+98] Sanjeev Arora, Carsten Lund, Rajeev Motwani, Madhu Sudan, and Mario Szegedy.Proof Verification and the Hardness of Approximation Problems. J. ACM, 45(3):501–555, 1998.

[Arı08] Erdal Arıkan. A Performance Comparison of Polar Codes and Reed-Muller Codes.IEEE Communications Letters, 12(6):447–449, 2008.

[AS92] Noga Alon and Joel H. Spencer. The Probabilistic Method. John Wiley, 1992.

11

m/20 m

log m log m

o(p

m)

o(p

m/ log m)

Degree (r) of RM(m, r):

(see also Figure 2)

[Dum04, DS06, Dum06]:⇡ n/2 errors O(d log d) = O(2(1�r)m · (1 � r)m) errors for r = r/m

O(n log n) time algorithm

Our results:⇡ n/2 errors

O(n4) time algo.

(d log d)O(log m) errors

n1+o(1) time algo.

2h⇣

1�r2

⌘m errors for r = r/m

assuming RM achieve capacity for all rates

Figure 3: Comparison with [Dum04, DS06, Dum06]

[ASW15] Emmanuel Abbe, Amir Shpilka, and Avi Wigderson. Reed-Muller Codes for RandomErasures and Errors. In Proceedings of the 47th Annual ACM Symposium on Theory ofComputing (STOC 2015), pages 297–306, 2015. Pre-print available at arXiv:1411.4590.

[BF90] Donald Beaver and Joan Feigenbaum. Hiding instances in multioracle queries. InProceedings of the 7th Symposium on Theoretical Aspects of Computer Science (STACS 1990),pages 37–48, 1990.

[BFL91] Laszlo Babai, Lance Fortnow, and Carsten Lund. Non-Deterministic Exponential Timehas Two-Prover Interactive Protocols. Computational Complexity, 1:3–40, 1991. Prelim-inary version in the 31st Annual IEEE Symposium on Foundations of Computer Science(FOCS 1990).

[BL15] Abhishek Bhowmick and Shachar Lovett. The List Decoding Radius of Reed-MullerCodes over Small Fields. In Proceedings of the 47th Annual ACM Symposium on Theory ofComputing (STOC 2015), pages 277–285, 2015. Pre-print available at eccc:TR14-087.

[BV10] Andrej Bogdanov and Emanuele Viola. Pseudorandom Bits for Polynomials. SIAMJ. Comput., 39(6):2464–2486, 2010. Preliminary version in the 48th Annual IEEESymposium on Foundations of Computer Science (FOCS 2007). Pre-print available ateccc:TR14-081.

[CF07] Daniel J. Costello, Jr. and G. David Forney, Jr. Channel coding: The road to channelcapacity. Proceedings of the IEEE, 95(6):1150–1177, 2007.

[DK94] Iwan M. Duursma and Ralf Kotter. Error-locating pairs for cyclic codes. IEEE Transac-tions on Information Theory, 40(4):1108–1121, 1994.

[DS06] Ilya Dumer and Kirill Shabunov. Recursive error correction for general Reed-Mullercodes. Discrete Applied Mathematics, 154(2):253–269, 2006.

[Dum04] Ilya Dumer. Recursive decoding and its performance for low-rate Reed-Muller codes.IEEE Transactions on Information Theory, 50(5):811–823, 2004.

12

Thank You!