hannel coding is an error-control techniqueused for providing robust data transmissionthrough imperfect channels by adding redun-dancy to the data. There are two important
classes of such coding meth-ods: block and convolutional.For this tutorial, we focus onlinear block codes becausethey provide much insight and allow for a simple visuali-zation of the error detection/correction process.Forward error correction (FEC) is the name used whenthe receiving equipment does most of the work. In thecase of block codes, the decoder looks for errors and,
once detected, corrects them (according to the capabilityof the code). The technique has become an importantsignal-processing tool used in modern communicationsystems and in a wide variety of other digital applications
such as high-density memoryand recording media. Suchcoding provides system per-formance improvements at
significantly lower cost than through the use of othermethods that increase signal-to-noise ratio (SNR) suchas increased power or antenna gain.
In this article we first develop the ideas behind sim-ple binary codes. We then treat cyclic and nonbinary
An intuitive treatment of error detection and correction
codes. Finally, we review the astounding strides thathave been made in this decade toward approaching thetheoretical limitations of what is possible. This articlecan serve as a precursor for serious students who intendto further pursue the subject and a quick overview forexperienced scientists or mathematicians who have notyet investigated this area.
Channel CodingChannel coding involves data transformations that areused for improving a system’s error performance byenabling a transmitted message to better withstandthe effects of channel impairments such as noise,interference, and fading. For applications that usesimplex channels (one-way channels such as compactdisk recordings), the coding techniques must supportFEC since the receiver must detect and correct errorswithout the use of a reverse channel (for retransmis-sion requests). Such FEC techniques can be thoughtof as vehicles for accomplishing desirable tradeoffsthat can reduce bit error rate (BER) at a fixed powerlevel or allow a specified error rate at a reduced powerlevel at the cost of increased bandwidth (or transmis-sion delay) and a processing burden. Figure 1 illus-trates such coding applied to a typical digital radio. Adata or message vector m = m1, m2, . . . , mk contain-ing k message elements from an alphabet is trans-formed by the block code into a longer code vector orcode word U = u1, u2, . . . , un containing n code ele-ments constructed from the same alphabet. The ele-ments in the alphabet have a one to onecorrespondence with elements drawn from a finitefield. Finite fields are referred to as Galois fields, afterthe French mathematician Evariste Galois(1811–1832). A Galois field containing q elements isdenoted GF(q), with thesimplest such finite fieldbeing GF(2), the binaryfield with elements (1, 0),which have the obviousconnection to the logicalsymbols (1, 0) called bits.When we deal with fieldsthat contain more thantwo elements, these nonbi-nary elements are encodedas binary m-tuples (m-bitsequences). Then the ele-ments are processed asbinary words according tothe rules of the field inmuch the same way thatdecimal integers wereencoded as binary-codeddecimal (BCD) symbols inearly computers (such asENIAC or the IBM 650)
and in contemporary calculators. The number of out-put elements n (code bits) and input elements k (databits) characterizing a block code are denoted by theordered pair (n, k). Often, the designation (n, k, t) isused to indicate that the code is capable of correctingt-errors in the n-element code word.
For transmitting the code bits (comprising U) withwaveforms, a common practice is to use bipolar pulseswith values (+1,−1) to represent the binary logic lev-els (1, 0), respectively. For a radio system, such pulsesare modulated on to a carrier wave, typically denoteds (t ), as shown in Figure 1. Channel impairments areresponsible for transforming a transmitted waveforms (t ) into a corrupted waveform r(t ) = s (t ) + n(t ) ,which is received and processed by a demodulator/detector. The demodulator recovers samples of the cor-rupted waveform, and the detector interprets the digi-tal meaning of that waveform. A commonly usedmodel for n(t) is that of an additive white Gaussiannoise (AWGN) process [1]. Noise, interference, andchannel distortion mechanisms account for the detec-tor making errors. Consequently, instead of accuratelyreproducing the bipolar pulse values or logic levels(representing U), the detector might instead output acorrupted version r, written as
r = U + e (1)
where r = r1, r2, . . . , rn represents a received block of nelements, and e = e1, e2, . . . , en represents the corrup-tion, referred to as the error sequence or error pattern.
Hard Decisions and Soft Decisions In Figure 1, each detected element ri of the receivedvector r can be described as a quantized-amplitude
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 15
▲ 1. Simplified diagram of channel coding applied to a typical digital radio.
MessageVector
Transmitter
Receiver
m
Channel Encoder
U = mG
(n,k) Code
Codeword
U
Modulator
Antenna
TransmittedWaveform
s(t)
ReceivedWaveformDemodulator/
Detectorr = U + e
ReceivedVector
r
Channel Decoder
U = r + er(t) = s(t) + n(t)
m
m = G–1U
decision. The decision may simply answer the question“Is the amplitude greater or less than zero?” yielding abinary decision of 1 or 0. Such a decision is called ahard decision because the detector firmly selects one oftwo levels. Sometimes the detector’s decision mayanswer multiple questions such as “Is the amplitudegreater or less than zero, and is it greater or less thansome reference level?” For binary signaling, such multi-part decisions are called soft decisions; they offer thedecoder side information about the SNR of the cor-rupted analog waveform. A soft decision might tell thedecoder, “this signal has a positive amplitude, but it isnot very far from the zero amplitude” or “this signalhas a positive amplitude, and it is quite far from thezero amplitude.” The most popular soft-decision for-mat entails eight-level signal quantization, which canbe interpreted as a hard decision plus a measure of con-fidence. The figure-of-merit for the error performanceof a digital communication system is usually expressedas a normalized SNR, known as the ratio of bit energyto noise power spectral density, Eb /N0. The codinggain or benefit provided by an error-correcting code toa system can be defined as the “relief” or reduction inrequired Eb /N0 that can be realized due to the code.For an AWGN channel, when the detector presents thedecoder with such soft decisions, the system can typi-cally manifest an improvement in coding gain of about2 dB compared to hard-decision processing [1]. Forthe majority of block-code applications, hard-decisiondecoding is used. For most of this tutorial, a receivedvector r out of the detector is made up of hard-decisioncomponents, designated by pulses (+1, −1) or by logiclevels (1, 0). However, later we show that soft decisionsare of great value for systems using iterative decodingtechniques that operate close to theoretical limitations.Examples of such techniques are turbo codes and low-density parity check (LDPC) codes.
Simple Parity CodesAt the transmitter, the encoder adds redundancy with aset of constraints that must be satisfied by the set of allcode words. Error detection occurs when a receivedvector does not satisfy the constraints. The simplestapproach to error detection modifies a binary datasequence into a code word by appending an extra bitcalled a parity bit. When using the constraint that acode word must contain an even number of ones, thescheme is referred to as even parity (the constraint of anodd number of ones is called odd parity). To establish
the even-parity condition, the parity bit p is formed, asthe modulo-2 sum of the message bits, as
p = m1 ⊕ m2 ⊕ m3 ⊕ · · · ⊕ mk (2)
where the symbol ⊕ indicates modulo-2 addition. Thetest (even-parity check) conducted by the receiver veri-fies that the modulo-2 sum of the parity plus messagebits in the received sequence r is zero. If the sequencefails the test, an error has been detected. We refer tothe test result as the syndrome S, written as
S = r1 ⊕ r2 ⊕ · · · ⊕ rk ⊕ rk+1. (3)
The syndrome in (3) can be modeled as the modulo-2sum of the transmitted sequence and the errorsequence, as
S = (m1 ⊕ e1) ⊕ (m2 ⊕ e2)
⊕ · · · ⊕ (mk ⊕ ek) ⊕ (p ⊕ ek+1) (4)
S = (m1 ⊕ m2 ⊕ · · · ⊕ mk ⊕ p)
⊕ (e1 ⊕ e2 ⊕ · · · ⊕ ek ⊕ ek+1)
= 0 ⊕ (e1 ⊕ e2 ⊕ · · · ⊕ ek ⊕ ek+1). (5)
When factored into separate message and errorsequences as seen in (5), we recognize that the syn-drome tests both the transmitted sequence and theerror sequence, but since the syndrome of the transmit-ted sequence is zero, the syndrome is only respondingto the error sequence. For the case of a single paritybit, as in (5), only an odd number of errors can bedetected, since an even number of errors will yield thesyndrome S = 0.
A single parity bit can only be used for error detec-tion. To perform error correction, we require addition-al information to locate the error positions; the codeword needs to be embedded with more than a singleparity bit. A simple example of a code that appendsadditional parity bits to the message sequence is shownin Table 1. Here, a set of eight message elements ispacked into a two-dimensional array from which weform parity for each row and parity for each column.
The appended array can be rearranged into a codeword sequence U as
When U is received, it can be mapped back to thesame two-dimensional array, and a set of syndromes canbe calculated corresponding to each row and each col-umn. A single error located anywhere in the messagepositions will cause a nonzero syndrome in a row andin a column, and thus the intersection of the row andcolumn corresponding to the parity failure contains thesingle error. One should conclude that a block codecapable of detecting and correcting error sequences
IEEE SIGNAL PROCESSING MAGAZINE16 JULY 2004
m1 m2 m3 m4 p1
m5 m6 m7 m8 p2
p3 p4 p5 p6
Table 1. Row and column parity for a two-dimensional data set.
needs to have multiple parity symbols appended to thedata message and multiple syndromes generated duringthe parity checks at the receiver.
The Generator Matrix and Systematic CodesThe most general form of the parity generationprocess, in which each code element ui of the codeword U is a weighted sum of message elements, can bewritten in the form of a vector matrix equation as
where the entries of the matrix G, called the generatormatrix, represent weights (field-element coefficients),and the multiplication operation follows the usual rulesof matrix multiplication. The product of a messagerow-vector m with the ith column-vector of G formsui a weighted sum of message elements representingthe ith element of the code word row-vector U. For abinary code, the data elements as well as the matrixweights are 1s and 0s, but for a nonbinary code thedata and weights are general field elements (of the non-binary field) with arithmetic performed in accordancewith the field structure [1], [2].
A useful variant of the code word U is one in whichthe vector of message elements is embedded, withoutchange, in the code word along with an appended vectorof parity elements. When the code word is constrained inthis manner, the code is called a systematic code. To forma systematic code the generator matrix G can be modi-fied in terms of submatrices P and Ik as follows:
where P is the parity portion of G, and Ik is a k-by-kidentity submatrix (ones on the main diagonal, andzeros elsewhere).
Vector Spaces and SubspacesFor now, we restrict our discussion to binary codes andn-bit sequences called n-tuples. The space of all binaryn-tuples is called a vector space Vn over the GF(2).This field is characterized by two operations, modulo-2addition and scalar multiplication [2], [3]. Within thespace Vn , one can form as many as 2n distinctsequences or n-tuples. A block of k message bits,referred to as a k-tuple, can form any one of 2k possiblemessage sequences. The encoding process involves aone-to-one assignment, whereby the 2k message k-tuples are uniquely mapped into a new set of 2k codewords chosen from the set of 2n n-tuples. When thecode is in systematic form, one can view the procedureas the addition of n – k parity bits (or redundant bits,since they carry no new information) to each k-bit mes-sage sequence. Even though such a code word is com-prised of k data bits and n – k parity bits, we typicallyrefer to each element in a binary code word as a codebit. For linear codes, the mapping transformation is, ofcourse, linear. Within a block, the ratio of data bits tototal code bits, denoted k/n, is called the code rate; itrepresents the portion of a code bit that constitutesinformation. For example, in a rate 1/2 code, eachcode bit carries 1/2 b of information. Note that whenthe coding tradeoff involves expanded bandwidth toachieve improved performance, then use of a rate 1/2code will require twice as much transmission band-width as that of an uncoded system. Similarly, the useof a rate 3/4 code will require a bandwidth expansionby the factor 4/3.
To construct a valid linear block code, the 2k n-tuples, making up the (n, k) code, must be confined toa subspace of the Vn space; all the code words of thesystem must satisfy this constraint. A subset of n-tuplevectors can be viewed as a subspace of Vn if the follow-ing two conditions are met: 1) the all-zeros vector isone of the vectors in the subset, and 2) the sum of anytwo vectors in the subset is also a member of the subset(known as the closure property). Note that condition 2requires condition 1, because any vector added to itselfmodulo-2 is equal to an all-zeros vector. Figure 2 illus-trates the framework of a linear block code using a sim-ple pictorial. The entire vector space Vn comprises 2n
n-tuples (shown as points). Within this space thereexists a subset of 2k n-tuples (shown as darker points)comprising a subspace or code system. This subset of2k points, “sprinkled” among the more numerous 2n
points, represent the code words {U} assigned to mes-sages. At the receiver, a corrupted version r of a trans-mitted U may be detected. If the corrupted vector r isnot too unlike (not too distant from) the transmittedU, the decoder can decode the message correctly. Acorrectable error is one that takes the transmitted codeword outside the subspace, but doesn’t take it “too faraway.” A noncorrectable error is one that takes thetransmitted code word to the vicinity of another code
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 17
word. A nondetectable error is one that takes the trans-mitted code word to exactly another code word. Tohave a sense of what is “too far away” we next examinethe concept of distance between vectors.
Weight and Distance PropertiesThe Hamming weight w(U) of a code word U isdefined as the number of nonzero elements in U. For abinary vector (or a nonbinary vector with field elementsrepresented in binary form), this is equivalent to thenumber of ones in the vector. For example, if U =1 0 0 1 0 1 1 0 1, then w(U) = 5. The Hamming dis-tance d(U, V) between two binary code words U andV is defined as the number of bit positions in whichthey differ. For example
if U = 1 0 0 1 0 1 1 0 1and V = 0 1 1 1 1 0 1 0 0
then d(U, V) = 6. (9)
By the properties of subspaces, we note that the sum oftwo code words U + V is another code word W in thesubspace, with binary ones in those positions in whichthe code words U and V differ. For example
W = U + V = 1 1 1 0 1 1 0 0 1. (10)
Thus, we observe that the Hamming distance betweentwo code words is equal to the Hamming weight of thesummed vectors: that is, d(U, V) = w(U + V). Also,note that the Hamming weight of a code word is equalto its Hamming distance from the all-zeros vector.
Decoding TaskThe decoding task can be stated as follows: Havingreceived the vector r, find the best estimate of the
particular code word Ui that was transmitted. Theoptimal decoder strategy is to minimize the decodererror probability, which is the same as maximizingthe probability P (U = Ui |r). If all code words areequally likely and the channel is memoryless, this isequivalent to maximizing P (r|Ui ), the conditionalprobability density function (pdf) of r, expressed as
p(r|Ui ) = max p(r|Uj )
over all Uj (11)
where the pdf, conditioned on having sent Ui , is calledthe likelihood of Ui . Equation (11), known as the max-imum likelihood (ML) criterion [1], can be used forfinding the “most likely” Ui that was transmitted. Foralgorithms using Hamming distances, the likelihood ofUi with respect to r is inversely proportional to the dis-tance between r and Ui , denoted d(r, Ui ). Therefore,we can express the decoder decision rule as: Decide infavor of Ui if
d(r, Ui ) = min d(r, Uj )
over all Uj . (12)
In the context of Figure 2, (12) means that for any n-tuple r, received in the Vn space of 2n possible values,the decoder shall choose the closest code word (heavyblack point) as the best estimate U of the transmittedcode word.
Error-Detecting and Error-Correcting CapabilityConsider the set of distances between all pairs of codewords in the space Vn. The smallest member of the setis called the minimum distance of the code and isdenoted dmin. To find dmin, we need not search the setof code words in a pairwise fashion. Because of the clo-sure property, we need only find the nonzero codeword having the minimum weight. The minimum dis-tance, like the weakest link in a chain, gives us a meas-ure of the code’s capability (indicates the smallestnumber of channel errors that can lead to decodingerrors). Figure 3 illustrates the distance between twocode words U and V using a number line calibrated inHamming distance, where each black dot represents acorrupted code word. In this example, let the distanced(U, V) be the minimum distance dmin = 5. Figure 3(a)illustrates the reception of a vector r1, which is distance1 from U and distance 4 from V. An error-correctingdecoder, following the ML strategy, will select U uponreceiving r1. If r1 had been the result of a 1-b corrup-tion to the transmitted code word U, the decoder hassuccessfully corrected the error. But if r1 had been theresult of a 4-b corruption to the transmitted code wordV, the result is a decoding error. Similarly a doubleerror in transmission of U might result in the receivedvector r2, which is distance 2 from U and distance 3
IEEE SIGNAL PROCESSING MAGAZINE18 JULY 2004
▲ 2. Linear block code framework.
2n n-Tuples Constitutethe Entire Space Vn
2k n-Tuples Constitutethe Subspace of Code Words
from V, as shown in Figure 3(b).Here too, the decoder will select Uupon receiving r2. A triple error intransmission of U might result in areceived vector r3 that is distance 3from U and distance 2 from V, asshown in Figure 3(c). Here thedecoder will select V upon receivingr3 and, given that U was transmit-ted, will have made a decoding error.From Figure 3, one can see that ifthe task is error detection (and notcorrection), then as many as 4-berrors can be detected. But, if thetask is error-correction, the decision to choose U if r fallsin region 1, and V if r falls in region 2, illustrates that thiscode (with dmin = 5) can correct as many as 2-b errors.We can generalize a linear block code’s error-detectioncapability ε and error-correction capability t as [1]
ε = dmin − 1 (13a)
t =⌊
dmin − 12
⌋(13b)
where the notation �x�, called the floor of x, means thelargest integer not to exceed x (in other words, rounddown if not an integer).
A (6, 3) Linear Block Code ExampleTable 2 describes a code word-to-message assignmentfor a (6, 3) code, where the rightmost bit representsthe earliest (and most-significant) bit. For each codeword, the rightmost k = 3 bits represent the message(hence, the code is in systematic form). Since k = 3,there are 2k = 23 = 8 message vectors, and thereforethere are eight code words. Since n = 6, then withinthe vector space Vn = V6 there are a total of 2n = 26 =64 6-tuples.
It is easy to verify that the eight code words shownin Table 2 form a subspace of V6 (the all-zeros vector isone of the code words, and the sum of any two codewords is also a code word). Note that for a particular(n, k) code, a unique assignment does not exist; how-ever, neither is there complete freedom of choice.
A Generator Matrix for the (6, 3) CodeFor short codes, the message-to-code-word mapping inTable 2 can be accomplished via a lookup table, but if k islarge, such an implementation would require a prohibitiveamount of memory. Fortunately, by using a generatormatrix G, described in (7) and (8), it is possible to reducecomplexity by generating the required code words asneeded instead of storing them. Since the set of codewords is a k-dimensional subspace of the n-dimensionalvector space, it is always possible to find a set of n-tuples(row-vectors of the matrix G), fewer than 2k that can gen-erate all the 2k code words of the subspace. The generat-
ing set of vectors is said to span thesubspace. The smallest linearly inde-pendent set that spans the subspace iscalled a basis of the subspace, and thenumber of vectors in this basis set isthe dimension of the subspace. Anybasis set of k linearly independent n-tuples V1, V2, . . . , Vk (that spans thesubspace) can be used to form a gen-erator matrix G. This matrix can thenbe used to generate the required codewords, since each code word is a linearcombination of V1, V2, . . . , Vk .That is, each code word U within the
set of 2k code words can be described by
U = m1V1 + m2V2 + · · · + mkVk (14)
where each mi = (1 or 0) is a message bit and theindex i = 1, . . . , k represents its position. In general,we describe this code generation in terms of multiply-ing a message vector m by a generator matrix G. Forthe (6, 3) code introduced earlier, we can fashion agenerator matrix G in systematic form, as
Table 2. Assignment of messages to code words for the (6, 3) code.
where P and Ik represent the parity and identity sub-matrices, respectively, and V1, V2, and V3 are three lin-early independent vectors (a subset of the eight codevectors) that can generate all the code words, made upof the weights {gi, j } as described in (8). Note also thatthe sum of any two generating vectors does not yieldany of the other generating vectors since linear inde-pendence is, in effect, the opposite of closure. The gen-erator matrix G completely defines the code andrepresents a compact way of describing a block code. Ifthe encoding operation utilizes storage, then theencoder only needs to store the k rows of G instead ofall 2k code words of the code. For systematic codes, theencoder only stores the P submatrix; it doesn’t need tostore the identity portion of G.
Visualization of a 6-Tuple SpaceFigure 4 is a visualization of the eight code words pre-sented in the (6, 3) example of Table 2. Since k = 3and the code words are generated from linear combina-tions of the k = 3 independent 6-tuples in (14), thecode words form a three-dimensional subspace. Figure4 shows such a subspace completely occupied by theeight code words (large black circles); the coordinatesof the subspace are purposely drawn to emphasize theirnonorthogonality. Figure 4 is an attempt to illustratethe entire space, containing 64 6-tuples, even thoughthere is no precise way to draw or construct such amodel. Spherical layers or shells are shown around eachcode word. Each of the nonintersecting inner layers is aHamming distance of one from its associated codeword; each outer layer is a Hamming distance of twofrom its code word. Larger distances are not useful inthis example. The two shells surrounding each code
word are occupied by corrupted code words. There aresix such points on each inner sphere because, for a (6,3) code word, there are six ways to make a 1-b error.Thus there are a total of 48 such 6-tuples on the innerspheres (eight code words × six ways for each to makean error). These 48 6-tuples are distinct in the sensethat each one can best be associated with only one codeword, and therefore for the case of a single bit beingreceived in error, the code word can be corrected.Later, we show that there is also one 2-b error patternthat can be corrected for this (6, 3) code. There is atotal of
( 62
) = 15 different 2-b error patterns that canbe inflicted on each code word, but only one of them,in our example the 010001 error pattern, can be cor-rected. The other 14 2-b error patterns yield vectorsthat cannot be uniquely identified with just one codeword. In the figure, all correctable (56) 1- and 2-berror-corrupted code words are shown as small blackcircles. Corrupted code words that cannot be correctedare shown as small clear circles. Figure 4 is useful forunderstanding desirable goals for a code system. Wewould like for the space to be filled with as many codewords as possible (yielding efficient utilization of theadded redundancy), and we would also like these codewords to be as far away from one another as possible(yielding reduced vulnerability to noise). Obviously,these goals conflict.
Error Detection and the Parity-Check MatrixAt the decoder, a method of verifying the correctness ofa received vector is needed. Let us define a matrix H,called the parity-check matrix, that will help us decodethe received vectors. For each (k × n) generator matrixG, one can construct an (n − k) × n matrix H, suchthat the rows of G are orthogonal to the rows of H.Another way to express this orthogonality is to say thatGHT = 0, where HT is the transpose of H, and 0 is ak × (n − k) all-zeros matrix [1]. HT is an n × (n − k)
matrix (whose rows are the columns of H). To fulfill theorthogonality requirements of a systematic code, the Hmatrix can be written as H = [In−k|PT ], where In−k rep-resents an (n − k) × (n − k) identity submatrix and Prepresents the parity submatrix defined in (8). Since bythis definition of H, we see that GHT = 0, and sinceeach U is a linear combination of the rows of G, thenany vector r is a code word generated by the matrix G, ifand only if
rHT = 0. (16)
Equation (16) is the basis for verifying whether areceived vector r is a valid code word.
Towards Error Correction: Syndrome TestingIn (1), the received vector r was expressed as the addi-tion of a transmitted code word U and an error pattern
IEEE SIGNAL PROCESSING MAGAZINE20 JULY 2004
▲ 4. Visualization of eight code words in a 6-tuple space.
101001
011101
000000
110011
011010
101110
000111
110100
e. Following (16), we define a syndrome vector S of r as
S = rHT . (17)
The syndrome (like the symptom of an ailment) is theresult of a parity check (like a diagnostic test) per-formed on r to determine whether r is a member of thecode word set. If, in fact, r is a valid code word, thenfrom (16), its syndrome S must be an all-zeros vector;in other words, r = U must have been generated bythe matrix G. If r contains detectable errors, its syn-drome will have some nonzero value. If the detectederrors are correctable, the syndrome will have a nonze-ro value that can uniquely earmark the particular errorpattern. A forward error-correcting decoder will thentake action to correct the errors.
The Standard Array and Error Correction The syndrome test gives us the ability to detect errorsand to correct some of them. Let us arrange the 2n
n-tuples that represent possible received vectors in anarray, called the standard array. This array can bethought of as an organizational tool or a filing cabinetthat contains all of the possible vectors in the space,nothing missing, and nothing replicated. The first rowcontains the set of all the 2k code wordsU1, U2, . . . , U2k starting with the all-zeros code worddesignated U1. In this array, each row, called a coset,consists of an error pattern in the leftmost position,called a coset leader, followed by corrupted code words(corrupted by that error pattern). Thus the first column,made up of coset leaders, displays all of the correctableerror patterns. The structure of the standard array for an(n, k) code, is
Note that code word U1 plays two roles. It is one ofthe code words (the all-zeros code word), as well as theerror pattern e1, that is the pattern that introduces noerrors so that r = U + e1 = U. Since the array containsall the 2n n-tuples in the space, each n-tuple appearingonly once, and each coset or row contains 2k n-tuples,we can compute the number of rows in the array bydividing the total number of entries by the number ofcolumns. Thus, in any standard array, there are2n/2k = 2n−k cosets. At first glance, the benefits of thistool seem limited to small block codes, because forcode lengths beyond n = 20 there are millions of n-tuples in Vn . Even for large codes, however, the stan-
dard array concept allows visualization of importantperformance issues, such as bounds on error-correctioncapability, as well as possible tradeoffs between errorcorrection and detection.
In the sections that follow, we show how the decod-ing algorithm replaces a received corrupted code wordr = U + e with an estimate U of the valid code wordU. If code word Ui is transmitted over a noisy channel,and the corrupting error pattern is a coset leader, thenthe received vector will be decoded correctly into thetransmitted code word Ui . If the error pattern is not acoset leader, an erroneous decoding will result [4], [5].
The Syndrome of a CosetThe name coset is short for “a set of numbers having acommon feature.” What do the members of a cosethave in common? Each member has the same syn-drome. We confirm this as follows: If ej is the cosetleader or error pattern of the jth coset, then Ui + ej isan n-tuple in this coset. From (17), the syndrome ofthis n-tuple can be written as
S = rHT = (Ui + ej )HT = Ui HT + ej HT . (19)
Since Ui is a valid transmitted code word, thenUi HT = 0, since the parity check matrix H was con-structed with this feature in mind. We can thereforeexpress (19) as
S = rHT = ej HT . (20)
Thus, the syndrome test, performed on either a cor-rupted code vector or on the error pattern that causedit, yields the same syndrome. Equation (20) establishesthat the syndrome is in fact only responding to the errorpattern, which was similarly shown in (5) for a simpleparity code. An important property of linear blockcodes, fundamental to the decoding process, is that themapping between correctable error patterns and syn-dromes is one to one. The syndrome for each coset isdifferent from that of any other coset in the code; it isthe syndrome that is used to estimate the error pattern,which then allows for the errors to be corrected.
Locating the Error PatternReturning to the (6, 3) code example, we arrange the
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 21
The process of decoding acorrupted code word by firstdetecting and then correctingan error can be compared to afamiliar medical analogy.
26 = 64 6-tuples in a standard array as shown in Figure5. The valid code words are the eight vectors in the firstrow, and the correctable error patterns are the sevennonzero coset leaders in the firstcolumn. Note that all 1-b error pat-terns are correctable. Also note thatafter exhausting all 1-b error pat-terns, there remains some error-cor-recting capability since we have notyet accounted for all 64 6-tuples.There is still one unassigned cosetleader; therefore, there remains thecapability of correcting one addi-tional error pattern. We have theflexibility of choosing this error pat-tern to be any of the n-tuples in theremaining coset. In Figure 5, thisfinal correctable error pattern waschosen, somewhat arbitrarily, to be the 2-b error pat-tern 010001. The error-correcting task performed bythe decoder can be implemented to yield correct mes-sages if, and only if, the error pattern caused by thechannel is one of the coset leaders. For the (6, 3) codeexample, we now use (20) to determine the syndrome(symptom) corresponding to each correctable errorpattern (ailment), by computing ej HT for each cosetleader, as follows:
S = ej
1 0 00 1 00 0 11 1 00 1 11 0 1
. (21)
The results are listed in Table 3. Since each syndromein the table has a one-to-one relationship with the list-ed error patterns, solving for a syndrome earmarks theparticular error pattern corresponding to that syn-drome [6].
Error Correction DecodingGiven a received vector r at the input of the decoder inFigure 1, we summarize the procedure for deciding on
U and finally on m as follows: 1) calculate the syn-drome of r using S = rHT and 2) use Table 3 tolocate the coset leader (error pattern) ej , whose syn-
drome equals rHT . This error pat-tern is assumed to be thecorruption caused by the channeland will be our estimate e of theerror (3) An estimate of the codeword U is identified as U = r + e.We can say that the decoderobtains an estimate of the transmit-ted code word by removing an esti-mate of the error e (in modulo-2arithmetic, the act of removal iseffected via addition). This step canbe written as
U = r + e = (U + e) + e = U + (e + e
). (22)
If the estimated error pattern is the same as the actualerror pattern, that is, if e = e, then the estimate U isequal to the transmitted code word U. However, if theerror estimate is incorrect, the decoder will choose acode word that was not transmitted, resulting in adecoding error.
As an example from the (6, 3) code, assume thatcode word U = 1 0 1 1 1 0 corresponding to m = 1 1 0(see Table 2) is transmitted and that the vectorr = 0 0 1 1 1 0 is received. From (17) we compute thesyndrome as
S = [0 0 1 1 1 0] HT = 1 0 0. (23)
From Table 3, we can verify that the error pattern ise = 1 0 0 0 0 0. Then, using (22), the corrected vectoris estimated as
U = r + e = 0 0 1 1 1 0 + 1 0 0 0 0 0 = 1 0 1 1 1 0. (24)
Since in this example, the estimated error pattern is theactual error pattern, the error correction procedureyields U = U, which means that the output m will cor-respond to the actual message 1 1 0. Note that the
process of decoding a corrupted codeword by first detecting and then cor-recting an error can be compared to afamiliar medical analogy. A patient r(potentially corrupted code word)enters a medical facility (decoder). Theexamining physician performs a diag-nostic test (multiplies r by HT ) to finda symptom (syndrome). Imagine thatthe physician finds characteristic spotson the patient’s X rays. An experiencedphysician would immediately recognizethe correspondence between the symp-
IEEE SIGNAL PROCESSING MAGAZINE22 JULY 2004
▲ 5. Example of a standard array for a (6, 3) code.
tom and the ailment, for example tuberculosis. Anovice physician might have to refer to a medical hand-book (Table 3) to associate the symptom (syndrome)with the ailment (error pattern). The final step is toprovide medication e. If e is the proper medication (ife = e), then the ailment is removed, as seen in (22). Inthe context of binary codes and the medical analogy,(22) and (24) reveal an unusual type of medicine prac-ticed here. The patient is cured by reapplying the origi-nal ailment, a process that works because in the binaryfield 1 + 1 = 0.
Decoder ImplementationWhen the code is short as in the case of the (6, 3) code,the decoder can be implemented with simple circuitry.The steps that such a circuit must take are: 1) calculatethe syndrome, 2) locate the error pattern correspondingto that syndrome, and 3) modulo-2 add the estimatederror pattern to the received vector to yield an estimateof the corrected vector. Consider the circuit in Figure 6,made up of exclusive-OR gates and AND gates that canaccomplish these decoder steps for any single-error pat-tern in the (6, 3) code. From Table 3 and (17), we canwrite an expression for the syndrome bits s1, s2, s3 interms of the received code word bits r1, . . . , r6 as
S = rHT (25a)
S = [s1 s2 s3] = [r1 r2 r3 r4 r5 r6]
1 0 00 1 00 0 11 1 00 1 11 0 1
(25b)
and s1 = r1 + r4 + r6
s2 = r2 + r4 + r5
s3 = r3 + r5 + r6. (26)
We use these syndrome expressions for wiring up thecircuit in Figure 6. The exclusive-OR gate provides thesame operation as modulo-2 arithmetic and hence usesthe same symbol. A small circle at the termination ofany line entering the AND gate indicates the logiccomplement of the signal.
The corrupted signal r enters the decoder at twoplaces simultaneously. At the upper part of the circuit,the syndrome S is computed, and at the lower partthat syndrome is transformed to its correspondingerror pattern e. The error is removed by adding it back
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 23
▲ 6. Implementation of the (6, 3) decoder.
Exclusive-ORGates
AND gates
e1 e2 e3 e4 e5 e6
r1 r2 r3 r4 r5 r6
u1 u2 u3 u4 u5 u6
r1 r2 r3 r4 r5 r6
s1 s2 s3
ReceivedVector r
ReceivedVector r
CorrectedOutput U
Syndrome S
ErrorPattern e
to the received vector yielding the corrected codeword U. Note that, for tutorial reasons, Figure 6 hasbeen drawn to emphasize the algebraic decoding steps,calculation of syndrome, error pattern, and correctedoutput. For real circuitry, the decoder would not needto deliver the entire code word; its output would con-sist of the message bits only. Hence, the Figure 6 cir-cuitry becomes simplified by eliminating the gates thatare shown with shading. For longer codes such animplementation is very complex, and the preferreddecoding techniques conserve circuitry by using asequential approach instead of this parallel method[2]. It is important to emphasize that Figure 6 hasbeen configured to detect and correct only single-errorpatterns for the (6, 3) code. Error control for a dou-ble-error pattern would require additional circuitry.
Cyclic CodesCyclic codes are a subset of linear block codes based onan algebraic structure that leads to strong error correct-ing capabilities and to computationally efficient encod-ing and decoding algorithms. Important classes ofcyclic codes are the binary Bose, Chaudhuri, andHocquenghem (BCH) codes and the nonbinary Reed-Solomon (R-S) codes. The cyclic nature of such codescan be stated as follows: If U is a code word (which ismade up of n elements) expressed as
U = u0, u1, u2, . . . , un−1 (27)
then a single cyclic (or end around) shift of the codeelements, written as
U(1) = un−1, u0, u1, . . . , un−2 (28)
is also a code word. A rich set of properties and struc-ture can be extracted from this simple relationship. Thecyclic code properties are best understood by mappingdata sequences and code words to polynomials in theindeterminate X, in a manner similar to the Z-trans-form (or delay-transform) [7] for sampled datasequences. For this article, the indexing scheme thatwas used for elements and vectors (data, code bits, pari-ty, and errors) up until this section, started with theindex 1, as seen in (6)–(8). Counting in this naturalway generally facilitates the intuitive understanding oferror-correction coding. When treating the subject ofcyclic codes, where polynomials are used to representsequences of elements, a different indexing scheme ispreferred. Whenever polynomials are used, we typicallystart the counting of elements with the index 0, as seenin (27). With this scheme, a code word of the formshown in (27) can be more naturally mapped to its cor-responding polynomial as
The coefficients of the polynomial are the correspon-ding elements of the sequence. In this mapping, theindeterminate X uses positive-integer exponents (asopposed to the negative integers used in the Z-trans-form). In this notation, all code words can be repre-sented by polynomials of degree n − 1 or less, withcoefficients drawn from the binary field (the degreeindicates the value of the highest-order exponent). Fornonbinary codes, the definition must include nonbina-ry coefficients. The polynomial corresponding to (28)is written as
We note that the cyclically shifted form shown in (30)can be obtained by using X as a delay operator followedby a set of trivial manipulations. We first form the poly-nomial XU(X) to obtain a linear shift of the elementsand then add the terms un−1 + un−1 . In the binaryfield, the added terms sum to zero so this addition doesnot alter the shifted polynomial. We write this as
X U(X ) = u0X + u1X 2 + · · · + un−2X n−1 + un−1X n
= un−1 + u0X + u1X 2 + · · · + un−2X n−1
+ un−1X n + un−1
= un−1 + u0X + u1X 2 + · · · + un−2X n−1
+ un−1(X n + 1). (31)
We now point out that we can form the cyclicallyshifted polynomial in (30) from the linearly shiftedversion in (31), as the remainder of X U(X) modulo-(X n + 1). We write this as
U(1)(X ) = X U(X )modulo-(X n + 1)
= un−1 + u0X + u1X 2 + · · · + un−2 X n−1.
(32)
Note that the modulo-(X n + 1) expression is some-times seen in the literature as modulo-(X n − 1). In thebinary field, either one is identical because +1 = −1.The expression in (32), often called the residue reduc-tion operation, consists of dividing the right side of(31) by X n + 1, then discarding the quotient polyno-mial and retaining the remainder polynomial. Byrepeated application of the cyclic shift using this residuereduction operation, we can similarly obtain the poly-nomial corresponding to all the cyclic shifts of any codeword in the subspace.
The Generator PolynomialIn an (n, k) cyclic code there is one code word polyno-mial g(X ) having minimum degree equal to n − k .Without proof, we state that g(X ) must be a factor ofthe polynomial X n + 1 [2], [8]. Because g(X ) can be
IEEE SIGNAL PROCESSING MAGAZINE24 JULY 2004
used to generate all code words in the code subspace(in a manner similar to the operation of the generatormatrix G described earlier), this polynomial is calledthe generator polynomial. The form of g(X ) is
For this polynomial to be of minimum degree, thecoefficient of X 0 and X n−k must each be one while thecoefficient of any term having a power higher thanX n−k must be zero. These zero-valued coefficients ofthe higher-powered terms represent the zeros in a codesequence needed to ensure that each code word ismade up of exactly n elements.
There is a one-to-one correspondence betweenthe basis vectors comprising a generator matrix Gand the polynomials formed by cyclically shifted ver-sions of g(X ). Here is how we illustrate the corre-spondence. By the properties of cyclic codes, sinceg(X ) is a proper code word polynomial, then anycyclic shift of g(X) is also a code word. In this dis-cussion, we limit the number of shifts to k − 1, thevalue that brings the degree of the shifted g(X ) upto n − 1. Each cyclic shift of g(X ) is performed byfirst multiplying g(X ) by X (linear shift) and thenend-around shifting the zero coefficient of the X n
term so that it becomes the new coefficient of theX 0 term. Consequently we can say that for thesek − 1 shifts, the cyclic shift formed by the residuereduction in (32) yields the same result as the linearshift indicated as
X i g(X ) modulo-(X n + 1) = X i g(X )
where i = 0, 1, 2, . . . , k − 1. (34)
These i-shifted versions of the generator polynomialg(X ) in (34) constitute a basis set of the code in thesame manner as the row space of the generator matrixG provides a basis set in (7). In a linear code, theweighted sum of proper code words yields anothercode word. Hence, since any X i g(X ) of the form citedin (34), is a proper code word polynomial, then we canrepresent it in terms of a message sequence as
Note the similarity between the formation of a codeword polynomial using the generator polynomial g(X )
in (35) and the formation of a code word using thegenerator matrix G in (7). Since for a binary code,there are 2k possible polynomials described by (35),and knowing that an (n, k) code contains precisely 2k
code words, we can conclude that (35) describes allproper code word polynomials in the cyclic code. Thatis, every code word polynomial can be formed as aproduct of a data polynomial m(X ) and the generatorpolynomial g(X ). One might recall from knowledge oflinear systems, that the product of polynomials is equiv-alent to the convolution of their corresponding coeffi-cients. Thus a tapped delay line convolver, with weightsequal to the coefficients of the generator polynomial,can form the code words of a cyclic code [9].
Systematic Cyclic CodeThe generator polynomial can also be used to form asystematic form of the code word polynomial. Fromthe expression derived in (35), we know that a propercode word is a multiple of g(X). Thus, dividing a prop-er code word by g(X ) yields a quotient (message poly-nomial) and a zero remainder. In general, if we divideany polynomial by g(X ), the remainder must be a poly-nomial of degree less than the divisor g(X ); that is, itmust be less than degree (n − k). We can encode a datapolynomial m(X ) into a systematic code word polyno-mial by first multiplying m(X ) by X n−k (representingn − k shifts) and then performing polynomial division,written as
X n−k m(X )
g(X )= q(X ) + p(X )
g(X )(36)
where q(X) and p(X) are the quotient and remainderpolynomials, respectively. When each side of thisexpression is multiplied by the divisor polynomial weobtain an equivalent form known as the Euclidiandivision algorithm, written as
X n−km(X ) = q(X )g(X ) + p(X ). (37)
We next add the remainder polynomial p(X) to both sidesof (37), yielding U(X) in systematic form, as follows:
p(X ) +X n−k m(X ) = q(X ) g(X ) + p(X ) + p(X )
= q(X ) g(X ) = U(X ). (38)
Since the p(X) + p(X) sums to zero, then U(X) in(38) is seen to be a multiple of g(X); hence U(X) is aproper code word polynomial. We note that the poly-nomial p(X) contains powers of X from zero throughn − k − 1 (thus there are n − k coefficients), while theshifted polynomial X n−k m(X ) contains the same coef-ficients as m(X), but positioned at powers of X fromn − k through n − 1. When we add the prefix (parity)polynomial p(X) to the shifted message polynomialX n−k m(X ), there is no overlap in the powers of X. Wecan think of the k coefficients of the polynomialX n−k m(X ) as the message that has been right shiftedn − k positions to make room for parity bits, as then − k coefficients of the polynomial p(X).
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 25
Encoding with a Linear Feedback Shift RegisterThe linear feedback shift register (LFSR) shown inFigure 7 performs the division operation as well as the(n − k)-bit shift presented in (36) and (37). This LFSRencoding process operates in the following manner. Forthe first k shifts of the input data sequence, the feedbackpath in the LFSR is enabled (switch 1 is closed); also,switch 2 is down. During this interval, the LFSR dividesby g(X), while the output switch simultaneously passesthe input data sequence to the output port. At the endof the kth input shift, the register contains the paritybits of the code word polynomial. For the next n − kshifts, the feedback path is disabled (switch 1 is opened)during which time the n − k parity bits from the regis-ter are transferred to the output port through switch 2,now moved to the up position. Note that the generatorpolynomial g(X ) (analogous to the generator matrix G)completely defines the cyclic code. g(X) is the mini-mum degree code word polynomial and thus representsthe most compact description of a cyclic code.
One may recognize Figure 7 as the dual form of therecursive filter structure used in standard digital signalprocessing applications. The traditional recursive filterforms a feedback term as a weighted sum of states
stored in a tapped delay line. This dual form is moredesirable for high-speed implementation because of thesimultaneous operation of the distributed summationsbetween the registers, as opposed to the adder treerequired in the tapped-delay feedback path.
Figure 8 demonstrates the use of an LFSR to encodethe message m = 1 0 1 1 for a (7, 4) cyclic code (in sys-tematic form) using the generator polynomialg(X ) = 1 +X +X 3 . The output code word isU = 1 0 0 1 0 1 1, where the rightmost four bits repre-sent the message. Table 4 tabulates the state of theLFSR, the input and output queues, and the switch set-tings, for seven successive operating cycles.
One should note that the wiring of the LFSR inFigures 7 and 8 correspond to the coefficients of thegenerator polynomial g(X), where a coefficient 1 or 0corresponds to the presence or absence of a wire,respectively. Sometimes a schematic such as the one inFigure 8 is used to specify a particular cyclic code,though a more compact way is to represent the codewith its generator polynomial.
Error Detection with an (nnn − kkk) Stage LFSRA code word that has been altered by AWGN can bedescribed as a received polynomial r(X), which is thesum of the transmitted polynomial U(X) and an errorpolynomial e(X), written as
r(X ) = U(X ) + e(X ). (39)
The decoder must perform an error-detection test onthe received polynomial r(X) to determine if it is aproper code word. This proceeds as follows: First, thedecoder forms a syndrome polynomial S(X) as theremainder of r(X) modulo-g(X). Note that S(X) servesthe same function as the syndrome vector S in (19) and(20). Since the remainder of the transmitted polynomi-al U(X) modulo-g(X) is zero, then S(X) represents atest on the error polynomial e(X). In the context of the
IEEE SIGNAL PROCESSING MAGAZINE26 JULY 2004
▲ 7. Linear feedback (n– k stage) shift register.
g1 g2 g3 gn-k-1
Switch 1
Input m0,m1,m2,...,mk–2,mk–1
Up at Shift k
Up at Shift kSwitch 2
u0,u1,u2,...,un–2,un–1
Output
Shift Input Register Number S1 S2 Queue Contents Feedback Output0 D D 1 0 1 1 0 0 0 0 11 D D 1 0 1 1 1 0 1 12 D D 1 0 1 0 1 1 03 D D 1 1 0 0 1 14 U U — 1 0 0 — 05 U U — 0 1 0 — 06 U U — 0 0 1 — 1
Table 4. Description of LFSR encoding of the message (1 0 1 1).
S1 = Switch 1, S2 = Switch 2, U = Up, D = Down
medical analogy proposed earlier, this test is the cyclic-code “diagnostic procedure” for finding the symptomof an error. We represent this procedure as
Whenever a channel-induced error results in areceived polynomial r(X) being a noncode word, thenthe syndrome polynomial S(X) will be nonzero and itsnonzero Hamming weight is an indication of an error.The same LFSR that formed the parity sequence at theencoder can be used to generate the syndrome polyno-mial S(X) at the decoder. The n bits of the receivedr(X) are processed by such an n − k stage LFSR (simi-lar to the one in Figure 8), and after the n-shift opera-tions, the contents of the register will be thecoefficients of S(X). The syndrome along with thestructure (embedded in the generator polynomial) canbe used to locate the error pattern.
Hamming Codes and BCH CodesHamming codes are single-error correcting codes thatexist for the (n, k, 1) set described by
(n, k, 1) = (2m − 1, 2m − 1 − m, 1) (41)
where m is the number of parity bits in the code. Theencoder and decoder for a Hamming code can beimplemented with an m-stage LFSR where m is thedegree of the generator polynomial (a factor of thepolynomial X n + 1) and n = 2m − 1.
A generalization of the Hamming codes leads to theBCH codes [10]–[12], which represent a class of cycliccodes that offer a large range of block length, coderates, and error correcting strength. A t-error correct-ing BCH code exists for the (n, k, t) set describedapproximately by
(n, k, t ) ≈ (2m − 1, 2m − 1 − mt , t ) (42)
where the degree of the generating polynomial as wellas number of parity bits in the code is less than or equalto mt. We use (42) as an upper bound for the numberof parity bits required to correct t binary errors in acode word polynomial of length 2m − 1 bits. Since theBCH code is a cyclic code, the encoding process is per-formed by a standard LFSR in the same manner thatwe form any cyclic code.
The operational difference for a BCH code is in thedecoding segment performed at the receiver. BCHdecoders, except for the simplest codes, do not use par-ity check matrices and syndrome tables. These earlier-described techniques are only implemented for shortsimple codes. For each received code word, typicalBCH decoders perform a sequence of algebraic opera-tions that yield the number of errors and the errorlocations. Rather than deriving the underlying mathe-matics for BCH codes, we will simply state the proper-ties of the polynomials and show how they enable thedesign of g(X) as well as the implementation of theerror-detection and correction algorithms. Bear inmind that the processing mechanism used to select thegenerating polynomial and implement the decodingalgorithms perform Galois field arithmetic, a detail thatis not needed to understand the overall process. Oneconcept that we do need to examine to understand theencoding and decoding process is the representation of
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 27
▲ 8. LFSR for g(X) = 1 + X + X3.
X0 X1 X2 X3
Switch 1
Up at Shift 4
Up at Shift 4
Switch 2
Input Queue 1011
OutputQueue
For R-S codes, one can say thatthe codes “prefer” to see thenoise occur in bursts. There is akind of “economy of scale”at work.
the elements in a Galois field. The nonzero elements ofGF(2n) are the roots of the polynomial X n + 1. In thecomplex field, the roots of X n − 1 form the periodicsequence exp ( j 2π/n), exp ( j 4π/n), exp ( j 6π/n),and so forth, which we write compactly asw1, w2, w3, . . . , where w = exp ( j 2π/n). Recall that inthe binary field, X n − 1 is equal to X n + 1. We oftenrefer to the roots of unity by their exponent, as we dofor instance when referring to the kth bin of a fast-Fourier transform (FFT). In like manner the roots ofX n + 1 in a GF form a periodic sequence written asα, α2, α3, . . . that are also referred to by their expo-nent. In the error location process described later, theexponents of a set of roots will be the indicator of theerror locations.
Decoding of BCH CodesThe generator polynomial g(X) is selected as a productof factors of X n + 1, with n = 2m − 1, such that thecomposite polynomial has a set of 2t known roots,α, α2, α3, . . . , α2t denoted here as β1, β2, . . . , β2t . Thebinary polynomials containing these roots are tabulatedin many textbooks [1], [2]. We have already describedin (40) how the remainder modulo-g(X) of thereceived code word polynomial is in fact the syndromeof the error polynomial. The relationship between theerror polynomial e(X) and the syndrome polynomialS(X) is shown as
e(X ) = q(X ) g(X ) + S(X ). (43)
Evaluating both sides of (43) at the known rootsX = βi of the generator g(X), we get
Since g(βi ) = 0, when we evaluate the syndrome poly-nomial at values of X equal to the roots βi , we obtainthe values that are the same as those we would obtainby evaluating the error polynomial. Hence, these valuescontain information (or clues) about the error polyno-mial. We compute the 2t scalar syndromes by firstforming S(X) from the received vector and then evalu-ating it at the roots βi as follows:
Si = S(X )|X =βi= S (βi ) : i = 1, 2, . . . , 2t . (45)
In general, the error polynomial e(X) is of the form:
e(X ) = {e i } =ν∑
j=1
X i( j ) (46)
where the jth error is located at indicator-index i(j)carried by the exponent of X, and ν represents thenumber of errors in the code word. Note that thedecoder has no knowledge of either the number oferrors ν or their location i( j ). A syndrome Si , com-puted by evaluating the error polynomial at the ith
power of α, is related to the error sequence by therelationship written as
Si =ν∑
j=1
(αi )i( j ) =ν∑
j=1
(αi( j ))i
=ν∑
j=1
(βj )i , i = 1, 2, . . . , 2t . (47)
In (47) we have interchanged the ith power with theerror location index i( j ) and simplified notation byreplacing αi( j ) by βj (similar to an earlier simplification).Suppose we have a double error correcting code, say a(15, 7, 2) code, and an error polynomial with nonzeroerrors at the two positions i(1) and i(2), written as
e(X ) = X i(1) +X i(2). (48)
We form the scalar values Si , for i = 1, 2, . . . , 2t , fromthe syndrome polynomial S(X) that for this example isan eighth-order polynomial. Then for this double-errorcorrecting example (t = 2), S = S1, S2, S3, S4. When weevaluate the eighth-order polynomial S(X) at the fourknown root values βi we obtain a set of four simultane-ous nonlinear equations in two unknowns written as
S1 = βi(1)1 + β
i(2)1
S2 = βi(1)2 + β
i(2)2
S3 = βi(1)3 + β
i(2)3
S4 = βi(1)4 + β
i(2)4 . (49)
In the general case, the decoder does not know thenumber of errors or the number of exponents to bedetermined. Note that for this double-error example,(49) indicates that there are two unknowns, the expo-nents i(1) and i(2). Why are there two unknowns(errors) but four equations? The answer is that in abinary code two of the equations are dependent andcan be discarded. But, in a nonbinary code with twoerrors, there are four unknowns—the two error posi-tions and the two error values. A related question is,suppose we only had one error but four equations? Theanswer is the same; some of the equations are depend-ent and are discarded.
A decoding algorithm that solves for the unknownexponents is termed an error locating process. Oncethe number of exponents and their values has beendetermined, error correction with a binary code is amatter of adding a one to the identified location(s) inthe received vector. One technique for solving the non-linear set of relationships in (49) is to define an auxil-iary polynomial known as the error locating polynomialσσσ (X) and use the syndrome values to solve for the coef-ficients σσσi . The error locating polynomial is defined tohave roots equal to the inverse of the error locationnumbers λi. The σσσ (X) polynomial is represented as
IEEE SIGNAL PROCESSING MAGAZINE28 JULY 2004
σσσ(X ) = (1 − λ1X )(1 − λ2).....(1 − λνX )
= σ0 + σ1X + · · · + σν−1X ν−1 + σνX ν. (50)
To determine the coefficients of the σσσ (X), we firstform an infinite series S∞(X) from an extended list ofsyndrome coefficients, shown as
S∞(X ) =∞∑
i=1
SiX (i−1). (51)
Substituting (47), the expression for the syndromes Siinto (51), and rearranging the sum, we can write
S∞(X ) =∞∑
i=1
X (i−1)ν∑
j=1
(βj )i
=∞∑
i=1
X (i−1)ν∑
j=1
(βj )(i−1)βj (52)
=ν∑
j=1
βj
∞∑i=0
(βiX )i =ν∑
j=1
βj1
1 − βjX. (53)
In (53) we have replaced each of the geometric series(connected to each error) with its closed form and rec-ognize the resulting sum as the partial fraction expan-sion of the infinite series. If we now form the productof S∞(X) and σσσ (X), we find that the factors of σσσ (X)cancel the denominator terms of (53)
S∞(X ) σσσ(X ) =ν∑
j=1
βj1
1 − βjX
ν∏�=1
(1 − β�X )
=ν∑
j=1
βj
ν∏�=1��= j
(1 − β�X ). (54)
From (54) we concluded thatz, due to the denominatorcanceling, the product of the infinite polynomial S∞(X)and the finite polynomial σσσ (X) is a finite polynomial ofdegree ν − 1, where ν is the number of errors in thesequence. Consequently, the coefficients formed by theproduct in (54) for the powers of X greater than or equalto ν are precisely zero. The coefficients of any power of Xformed by the polynomial product are seen to be aweighted sum of the syndrome coefficients. Setting thecoefficients of the powers of X from ν to 2t to zero, leadsto the collection of equations relating the error locatingcoefficients σσσi and the scalar syndromes Si shown as
S1 S2 · · · StS2 S3 · · · St+1...
.... . .
...
St St+1 · · · S2t−1
σtσt−1
...
σ1
=
−St+1−St+2
...
−S2t
.
(55)
One may recognize (55) as the set of normal equationsthat describe a one-step linear predictor used in manysignal-processing applications. The Berlekamp-Massey
algorithm [2], [13], [14] iteratively forms the solution,but any standard technique to invert a matrix can beapplied to the task. If (55) has more equations thanunknowns its determinant will be zero and we cansequentially peel off one row and column and attemptto invert the reduced size matrix. Once the coefficientsσσσi of the error location polynomial have been deter-mined, the polynomial is tested to find its roots λi . TheChien algorithm [8], [15] is a method of exhaustivelytesting the polynomial to find these roots, which wheninverted, identify the error locations. Then, adding aone to their identified locations easily repairs the errors.
Reed Solomon CodesIn 1960, Irving Reed and Gus Solomon described anew class of error correcting codes [16] that are nowcalled R-S codes. The R-S codes are nonbinary cycliccodes in which the code symbols are binary m-tuples.They are used in many applications, from the compactdisk to spacecraft traveling beyond the orbit of Pluto[17]. In 1964, Singleton showed that R-S codes havethe best possible error-correction capability for anycode of the same length and dimension [18]. Codesthat achieve this “optimal” error correction capabilityare called maximum distance separable (MDS). As thename implies, their key property is that they providemaximum distances between code words [17], [18].These codes are extensions of the BCH codes in muchthe same manner that the BCH codes are extensions ofthe Hamming codes. There exist t-error correcting R-Scodes for the (n, k, t) set described as
(n, k, t ) = (2m − 1, 2m − 1 − 2t , t ) (56)
where 2t is the degree of the generating polynomial aswell as the number of parity symbols in the code[19], [20]. The coefficients of the generating polynomi-al g(X ) are no longer binary but rather they are m-bitcoefficients having the same meaning as the m-bit parti-tion of the input data. The LFSR, which performs bothencoding and decoding, stores m-bit words in its regis-ters and performs products of the m-bit coefficientsusing arithmetic rules defined by an extension fieldreferred to as GF(q), where the number of elements q inthe field must be of the form pm where p is a prime inte-ger and m is positive integer [2], [21]. Elements fromthe extension field GF(2m) are used in the constructionof R-S codes. The rules for doing the arithmetic are notneeded to understand the overall algorithms. Cyclic R-Scodes with code words from GF(q) have lengthm = q − 1, as is seen in (56). Popular R-S code choicesfor many applications are chosen from the field GF(28)= GF(256), because each of the 256 field elements canbe represented as an 8-b sequence, or byte [17].
Earlier we showed that simple block codes can bedecoded by using a syndrome lookup table (or circuit).The syndrome was computed and its correspondingerror pattern was subtracted from the received vector.
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 29
Unfortunately, this approach is out of the question forall but the most trivial R-S codes. For example, the (63,53) five-error correcting R-S code has approximately1020 syndromes. The construction of such a table issomewhat questionable. In [16], Reed and Solomonproposed a decoding algorithm based on solving sets ofsimultaneous equations. Though much more efficientthan a lookup table, this algorithm is only useful for thesmallest R-S codes. The decoding of longer R-S codesbecame viable in 1967 when Berlekamp demonstratedhis efficient decoding algorithm for both R-S and non-binary BCH codes [14], [22], [23].
Since the symbols in the R-S code are m-tuples, theerror symbols introduced by the channel are also m-tuples. A typical two-symbol error polynomial mayhave the form shown as
e(X ) = e1X i(1) + e2X i(2) (57)
where m-bit errors e1 and e2 have been added at posi-tions i(1) and i(2). A channel error is detected at thedecoder if the syndrome polynomial has nonzero coef-ficients. As with the BCH codes, the syndrome polyno-mial is evaluated at the roots of the generatorpolynomial to determine the received syndrome valuesSi . These values are used in (55) to find the coefficientsof the error location polynomial. The degree of theerror location polynomial tells us the number of errorsin the received code vector. The final task is to solve forthe error values {e i ,} by solving the following matrix(shown for the general case of ν errors)
λ1 λ2 · · · λvλ2
1 λ22 · · · λ2
v...
.... . .
...
λv1 λv
2 · · · λvv
e1e2...
ev
=
S1S2...
Sv
. (58)
Equation (58) expresses (57) evaluated at the rootsX = λi of the error-locating polynomial σσσ(X ), wherethe indeterminate X is replaced with the known errorlocations. Once the error values are determined, the m-tuple error values are added to the received m-tuples atthe previously determined error locations.
Channel Erasure Decisions In one form of soft-decision making, the detectorforms one of three decisions regarding the receivedprocessed waveform: binary 1, binary 0, or E, called anerasure. The 1 and 0 denote the sign of a processed sig-nal if its magnitude is greater than some threshold(hard decision), and E denotes the condition that thesignal magnitude is less than the threshold. The erasurereflects the information that the signal level is “tooclose to call” and might yield an unreliable decision.
A decoder can correct dmin − 1 erasures (treated asunknown errors at known error locations). Normallyhalf the information residing in the syndrome values is
used to locate the error positions and half is used tocorrect the errors. Since erasure locations are known,only their values have to be determined by the decod-ing algorithm. Thus for the erasure correction task, thetotal syndrome information can be directed to repair-ing up to dmin − 1 erasures (rather than about half thatmany �dmin − 1�/2 for the error correction task). Thedecoding technique involves setting the erasured sym-bols to zero value and then proceeding with standardR-S decoding (omitting application of the error locat-ing polynomial). The ability to recover from twice asmany erasures as errors is one of the reasons that thecompact disk player uses erasure detection and correc-tion in its concatenated and interleaved R-S error-con-trol procedures [24], [25].
Performance of R-S Codes in Bursty NoiseA key benefit of nonbinary codes, such as R-S codes, istheir performance in the presence of bursty noise [26].Even when a channel is characterized by random noise,R-S codes are often concatenated with convolutionalcodes to repair any convolutional decoding errors(which typically occur in bursts) [27], [28]. Consideras an example, the R-S (255, 247) code. In the contextof (56), this code can be described using the (n, k)notation of (28 − 1, 28 − 1 − 2t ). Hence, n − k = 2t ,or the code’s error-correcting capability ist = (n − k)/2. In terms of its 8-b symbols (or bytes),t = (255 − 247)/2 = 4 B within each code wordsequence of 255 code bytes. Imagine a burst of con-tiguous noise events, having a duration of 25 b. Howmany code bytes within the code word would be affect-ed? No matter where these contiguous noise eventsmight appear within the 255 code bytes, they mustaffect exactly 4 B. However, the R-S (255, 247) code iscapable of correcting 4 B. The decoding is based onreplacing an incorrect byte regardless how many bitswithin the byte have been corrupted. An incorrect byteis replaced with a correct one, whether the corruptionwas caused by a 1-b-duration noise event or an 8-b-duration noise event. Now compare this to the casewhere the noise does not appear as a contiguous burst,but as 25 random events. How many of the 255 R-Scode bytes might be “hit” in such case? Twenty-fivebytes might be hit, and the code would then be over-whelmed. For R-S codes, one can say that the codes“prefer” to see the noise occur in bursts. For nonbinarycodes such as R-S, there is an kind of “economy ofscale” at work whenever the degrading events occur inbursts rather than as random events.
Approaching the Shannon LimitShannon [29], [30] developed many of the fundamen-tal mathematical relationships for communicating infor-mation in the presence of noise. He showed that everychannel has a channel capacity C, and for transmissionrates less than C, there exists codes capable of achieving
IEEE SIGNAL PROCESSING MAGAZINE30 JULY 2004
arbitrarily small decoded-error probabilities. Anoutgrowth of his capacitytheorem shows that withoptimum coding, it ispossible to approach anerror probability of zerowith an Eb /N0 as low as−1.6 dB. This limitingvalue is known as theShannon limit; it is oftenshown as a discontinuouscurve (vertical line) onthe bit-error probabilityPB versus Eb /N0 plot,extending from PB = 1/2 to PB = 0. Shannon’s workproves the existence of codes that can approach thislimiting performance but does not show how to con-struct them. It does show that to achieve very low errorprobabilities, long block lengths are needed. In effect,Shannon predicted that coding techniques would befound for achieving coding gains of about 11–12 dB.Ever since 1948, research in error-correction codinghas focused on constructing good codes and findingeasy-to-implement encoding/decoding methodstoward fulfilling Shannon’s prediction. The Shannonlimit represents an interesting theoretical bound, but itis not a practical goal because it was derived for a rate 0code. Several authors use an Eb /N0 of 0.2 dB as apragmatic Shannon limit for a rate 1/2 code over abinary-input AWGN channel [31].
For about 30 years after Shannon’s fundamentalwork, coding research produced a rich assortment ofpractical techniques (some of which are listed in thereferences), resulting in an achievable coding gain ofabout 6–7 dB at PB = 10−5. During the 1980s, thereweren’t any major developments to allow for largergains. In 1993 Berrou et al. reported on a concatenat-ed rate 1/2 coding scheme that used iterative decodingto achieve a PB = 10−5 at an Eb /N0 within about 0.5dB away from the “pragmatic” Shannon limit [32].That development, known as turbo coding, enabled theimplementation of real systems with coding gains ofabout 8–10 dB.
The next revolution, known as LDPC codes, wasactually a resurgence of a scheme that had been pro-posed by Gallager in his 1961 doctoral dissertationand in [33] and [34]. These codes were largely forgot-ten for several decades due to the lack of computation-al capability in those years. LDPC codes, which alsouse iterative decoding, were rediscovered by McKayand Neal [35] in 1996. Today the field of LDPC cod-ing appears to represent the main competition toturbo coding. An LDPC rate 1/2 code using a blocklength of 107 holds the distinction of achieving a BERof 10−6 within 0.04 dB of the “pragmatic” Shannonlimit, as reported by Chung et al. [36] in 2001.
Likelihood Functions and the ML CriterionLinear block codes, including cyclic codes and R-Scodes, lend themselves to hard-decision decoding giv-ing rise to the type of syndrome calculation and deci-sion making described earlier. However, for iterativedecoding (as exemplified by both turbo and LDPCcodes), the improvements offered by soft-decisiondecoding (about 2 dB for eight-level quantization, inAWGN) are an essential part of the operation.
Figure 9 shows the conditional pdfs, referred to aslikelihood functions, that depict the binary decision-making process associated with pulses (+1,−1) dis-turbed by AWGN. Since AWGN is the most commonlyused noise model in communication systems, we use ithere and therefore show the likelihood functions inFigure 9 drawn with the familiar Gaussian shapes. Therightmost function p(x |d = +1), called the likelihoodof d = +1, shows the pdf of the random variable x(noisy signal sample) conditioned on the data d = +1being sent. The leftmost function p(x |d = −1), calledthe likelihood of d = −1, illustrates a similar pdf condi-tioned on d = −1 being sent. In Figure 9, one sucharbitrary received sample value xj is shown. A line sub-tended from xj intercepts the two likelihood functionsyielding the likelihood values �1 = p(xj |d = +1) and�2 = p(xj |d = −1). Stemming from Bayes’ theorem[1], one can make a data decision by forming a ratio oflikelihoods and deciding that d = +1 if
�1
�2= p(xj |d = +1)
p(xj |d = −1)>
P (d = −1)
P (d = +1)
orp(xj |d = +1)P (d = +1)
p(xj |d = −1)P (d = −1)> 1. (59)
Equation (59), known as the maximum a posteriori (MAP)criterion, corresponds to making a decision based on com-paring �1/�2 to the threshold P (d = −1)/P (d = +1), aratio of a priori probabilities. That is, decide d = +1 if thelikelihood ratio �1/�2 is greater than the threshold, andotherwise decide d = −1. When knowledge about the a
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 31
▲ 9. Likelihood functions.
Likelihood of d = –1p(x|d = –1)
Likelihood of d = +1p(x|d = +1)
–1
Threshold
+1xj
xj2
xj3
x
priori probabilities is not available, decisions are usuallymade by assuming equally likely a priori probabilities.When this is done, the MAP criterion in (59) is thenknown as the ML criterion.
Turbo CodesInitially, turbo codes were implemented with two ormore convolutional encoders configured as concate-nated component codes; but block encoders are equal-ly applicable. By taking the logarithm of a likelihoodratio, we obtain a useful metric called the log-likeli-hood ratio (LLR). In the context of (59), we designateL ′(d) to represent the sum of two LLRs (one for thedetector’s soft-decision output and one for the ratio ofa priori probabilities of the data). We show this as
L ′(d) = log[
P (x |d = +1)
P (x |d = −1)
]+ log
[P (d = +1)
P (d = −1)
]
(60a)= L (x |d) + L (d) = Lc (x ) + L (d) (60b)
where L (x |d ), designated Lc (x ), is the LLR of therandom variable x representing the detector’s measure-ment of the received signal x under the alternate condi-tions that d = +1 or d = −1 may have been sent, andL (d) is the a priori LLR of the data bit d. The notationLc (x ) emphasizes that this LLR term is the result of achannel measurement made at a receiver. Note that forequally likely a priori probabilities, the ratio of priorprobabilities is equal to one; hence, its logarithm, seenas the last term in (60), is equal to zero. This corre-sponds to a threshold that is located at the crossover ofthe two likelihoods as seen in Figure 9. So far we havenot considered the presence of any coding. The intro-duction of a code will typically yield decision-makingbenefits. For a systematic code, it can be shown [32]that the LLR (soft output) L (d) out of the decoder canbe expressed as
L (d) = L ′(d) + Le (d) = Lc (x ) + L (d) + Le (d) (61)
where L ′(d) stems from (60) and Le (d), called theextrinsic LLR, represents extra knowledge that isgleaned from the decoding process. Thus, (61) express-es the decoder output in terms of three LLR elements—a channel measurement, a priori knowledge of the data,and an extrinsic LLR stemming solely from the decoder.Each of these individual LLRs can be added (for a sys-tematic code) because these three terms are statisticallyindependent [1], [33], [37]. This decoder output L (d)
is a real number that provides a hard decision plus aconfidence metric regarding that decision. The sign ofL (d) denotes the hard decision; that is, for positive val-ues of L (d), we decide that d = +1, and for negativevalues decide that d = −1. The magnitude of L (d)
denotes the confidence of that decision. Often, thevalue of Le (d) has the same sign as Lc (x ) + L (d), andtherefore acts to improve the reliability of L (d).
Decoding of Turbo CodesTurbo decoding involves feeding outputs from onedecoder to the inputs of other decoders in an iterativefashion. The phenomenal performance of these tech-niques requires a novel type of decoder—one thataccepts soft decisions at its input and delivers soft deci-sions at its output. For the first decoding iteration, illus-trated in Figure 10, one generally starts with theassumption that the binary data is equally likely, yieldingan initial a priori LLR value of L (d) = 0 for the right-most term in (60). The channel LLR value, Lc (x ), ismeasured by forming the logarithm of the ratio of thevalues of �1 and �2 for a particular observation of x (seeFigure 9). As illustrated in Figure 10, where thedecoder block represents two or more decoders, theextrinsic likelihood Le (d) out of the decoder is fedback to the decoder input, to serve as a refinement ofthe a priori probability of the data for the next iteration.Additional iterations yield values of L (d) that asymptot-ically manifest further refinements and hence bettererror performance. A fundamental principle for feedingback information to another decoder is that a decodershould never be supplied with information that stemsfrom its own input (because the input and output cor-ruption will be highly correlated). Such a soft-input,soft-output decoder is sometimes referred to as a “SNRamplifier.” Real systems are typically implemented witha fixed number of about four to eight such decodingiterations. Figure 9 offers some insight into the iterativeprocess. As a result of a single pass through the decoder,the sample xj yields a soft number, the LLR L (d). Thisis illustrated by a vertical line passing through xj andthe likelihood intercepts �1 and �2. Imagine a secondand third decoding iteration of the same sample, nowlabeled xj2 and xj3, respectively, that may result in amore confident L (d) output, as shown in Figure 9 bythe rightward movement of the vertical line. Such
movement can come about (with increased iterations)because the initial a priori probabilities become“refined” during each round, resulting in an apparentmovement of the sample point yielding a more confi-dent L (d). We can interpret this to mean that the turbodecoder iterations give us an apparent SNR enhance-ment. An extensive treatment of turbo encoding anddecoding can be found in [32] and [38]–[51].
LDPC Codes An LDPC code is traditionally defined by its M ×Nparity-check matrix H. The M rows in H specify theM constraints in the code. For example, if one constraintdictates that bit numbers 1, 4, and 6 must sum to zerofor each code word, then there must be one row in Hthat contains a one in positions 1, 4, and 6, and zeroselsewhere. The N columns in H correspond to the totalnumber of code bits within a code word. A regularLDPC code has a sparse H matrix having exactly wc onesper column and exactly wr = wc (N /M ) ones per row,where wc and wr are both small compared to N . Thus,each parity-check equation involves exactly wr bits, andevery bit in a code word is involved in exactly wc paritycheck equations. The term “low density” indicates thatthese codes are characterized by sparse H matrices. Thefraction of ones in a regular LDPC matrix is wr/N ,
which can be quite small for long codes. In contrast, theaverage fraction of ones in a random binary matrix (withindependent components equally likely to be one orzero) is 1/2. If H is sparse, but the number of ones percolumn or row is not constant, the code is called anirregular LDPC code. Such irregular codes often outper-form regular codes with similar dimensions [52], [53].
Any parity-check code (including an LDPC code)may be specified by a Tanner graph [54], as shown inFigure 11. The (6, 3) code example (described earlier)does not have a sparse H matrix; nevertheless, we use ithere as a simple way to illustrate this graph. For LDPCcodes, we shall use the index m to indicate a particularparity check location (row in H), and index n to indicatea particular bit position (column in H). The Tannergraph contains two kinds of nodes, check nodes (desig-nated with squares) and bit nodes (designated with cir-cles). There are M check nodes, one for each paritycheck C1, . . . ,Cm, . . . ,CM , and N bit nodes, one foreach code bit v1, . . . , vn, . . . , vN . The check nodes areconnected to the bit nodes that they check. Specifically, abranch (also called an edge) connects check-node m tobit-node n if and only if the mth parity check involvesthe nth bit (i.e., only if Hm,n = 1). Thus, the graph isanalogous to the H matrix, since the entries in H are 1 ifand only if the mth check node is connected to the nthbit. The branches between nodes can be thought of asinformation-flow pathways for the iterative computationof probabilistic quantities. During the decoding process,likelihoods obtained from soft-decision components of areceived vector r initialize the bit nodes. Each bit node
determines the probability of a bit having value one (alsozero), given the parity check information from all thecheck nodes involving that bit. Each check node deter-mines the probability that its parity check is satisfied,given that a particular bit is set to one (also zero) andthe other participating bits have separable distributions.
In Figure 11, we have repeated properties of the(6, 3) code described earlier—code word assignments,matrix G, and matrix H. Only bits and checks that arerelated by having a one in the corresponding locationof Hm,n need to be considered. For the (6, 3) code,the rows of H, describing the parity checks that mustbe met by each proper code word, can be written as
For the (6, 3) code, the set of relationships in (62) cor-roborates what was described in (26). That is, each rela-tionship yields a syndrome, and the code constraints aremet when each syndrome component has a zero value.
Decoding of LDPC CodesFor decoding an LDPC code, we want to find theprobability that each bit vn of a received vector r equalsone (also zero), knowing that the estimated code wordU stemming from r must satisfy the constraintU HT = 0. Given a received vector r, solving directlyfor the probability P (vn = b |r), that the nth bit equalseither one or zero is very complex. Gallager [34] pro-vided an iterative technique, known as the sum-productalgorithm, where the probability µmn(b ) that the mthcheck is satisfied by a received vector r (given vn = b ),is passed from check node Cm to bit node vn . This sat-isfied-check probability µmn(b ) is gleaned from all bitsparticipating in the m th check (other than vn ).Likewise, the bit probability qmn(b ) that the nth bit hasvalue vn = b , is passed from bit node vn to check nodeCm . This bit probability qmn(b ) is gleaned from all thechecks that the nth bit participates in (other than Cm).
IEEE SIGNAL PROCESSING MAGAZINEJULY 2004 33
▲ 11. Representing the (6, 3) code with a Tanner Graph.
The message-passing procedure is indicated on theTanner graph of Figure 11 in the context of the (6, 3)code described earlier. Messages containing satisfied-check probabilities µmn(b ) are shown moving fromcheck nodes to bit nodes, and messages containing bitprobabilities qmn(b ) are shown moving from bit nodes tocheck nodes. The process is repeated until it converges toa code word solution or until a time limit is reached.With the aid of Figure 11, consider the following exam-ple of message-passing iterations, focusing particularly onbit-node v4 and check-nodes C1 and C2. At the start, bitnodes are initialized with likelihood values stemmingfrom a detector. Suppose that bit node v4 passes theprobability q24(b ) that v4 = 1 (also 0) to check-node C2.(It passes similar information to check-node C1.) Check-node C2 collects the incoming probabilities from allother bits involved in check 2, namely v2 and v5, com-putes a probability µ24(b ) that parity-check C2 is satisfiedgiven that v4 = 1 (also 0), and passes this message to bit-node v4. Check-node C2 passes similar information to v2,given v2 = 1 (also 0), and to v5, given v5 = 1 (also 0).When bit-node v4 receives such satisfied-check informa-tion from all the check nodes involving v4, namely C1and C2, it recomputes the probability that v4 = 1 (also0) gleaned from the connected check nodes (apart fromcheck-node C2) and passes this message back to nodeC2, and similar information to check-node C1, and soforth. Note that when sending µmn(b ) probabilities fromCm to vn , this algorithm excludes information passedfrom vn to Cm in the prior round. Similarly when send-ing qmn(b ) probabilities from vn to Cm , the algorithmexcludes information passed from Cm to vn in the priorround. These rules resemble those used in the iterativeturbo-decoding process, where information is never fedback to a decoder that stems from itself.
After decoder initialization, we can think of theprocess of updating the probabilities between nodes asproviding extrinsic information that yields improve-ments in coding performance. Each iteration typicallyprovides more confident likelihood values of the codebits. The algorithm is typically implemented in the loga-rithmic domain using log-likelihood ratios similar to theturbo-decoding process. The iterative decoding tech-niques used in both turbo and LDPC codes are thetools that have brought about the astounding error per-formance that was promised by Shannon in 1948.Extensive treatments of LDPC coding can be found in[33], [34], [52]–[68]. An important coding task, yet tobe accomplished, is to determine for implementing realsystems, which codes and decoding techniques can yieldoptimum performance with minimum complexity.
ConclusionBasic principles of block codes were presented withillustrations to visualize the concepts of vector spacesand subspaces. We offered intuitive explanations of
goals, capabilities, and limitations of codes. We exam-ined an important subclass of block codes called cycliccodes, described their algebraic structure, and lookedat the very popular BCH and R-S cyclic codes. Also, welooked at the newest techniques, turbo codes andLDPC codes, that use iterative decoding to obtain per-formance exceedingly close to theoretical limitations.
Bernard Sklar has 50 years of electrical engineeringexperience at companies that include Hughes Aircraft,Litton Industries, and The Aerospace Corp. He ishead of advanced systems at CommunicationsEngineering Services. He received the 1984 PrizePaper Award from the IEEE Communications Societyand is the author of Digital Communications:Fundamentals and Applications, 2nd edition (Prentice-Hall, 2001).
Fredric J. Harris is a professor of electrical and comput-er engineering at San Diego State University where heholds the CUBIC Signal Processing Chair. He holdsseveral patents and is the author of Multirate SignalProcessing for Communication Systems (Prentice Hall2004). He was chair of the Asilomar Conference onSignals, Systems, and Computers, and the SoftwareDefined Radio Conference.
References[1] B. Sklar, Digital Communications: Fundamentals and Applications, 2nd ed.
Englewood Cliffs, NJ: Prentice-Hall Inc., 2001.[2] S. Lin and D.J. Costello, Jr., Error Control Coding: Fundamentals and
Applications. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1983.[3] G.C. Clark and J.B. Cain, Error Correction Coding for Digital
Communications. New York: Plenum, 1988.[4] J. Wolf, A. Michelson, and A. Levesque, “On the probability of undetected
error for linear block codes,” IEEE Trans. Commun., vol. COM-30, pp. 317–325, Feb. 1982.
[5] T. Kasami, T. Klove, and S. Lin, “Linear block codes for error detection,”IEEE Trans. Inform. Theory, vol. IT-29, pp. 131–136, Jan. 1983.
[6] D. Hertz and Y. Azenkot, “Memory/speed tradeoffs for look-up tabledecoding of systematic linear block codes,” IEEE Trans. Commun., vol. 38,pp. 109–111, Jan. 1990.
[7] R.G. Gallager, Information Theory and Reliable Communication. NewYork: Wiley, 1968.
[19] S.B. Wicker, Error Control Systems for Digital Communication andStorage. Englewood Cliffs, NJ: Prentice Hall, 1994.
[20] B. Cipra, “The ubiquitous Reed-Solomon codes,” SIAM News, vol. 26,no.1, Jan. 1993.
[21] R.J. McEliece, Finite Fields for Computer Scientists and Engineers. Boston,MA: Kluwer Academic, 1987.
[22] E.R. Berlekamp, “Nonbinary BCH decoding,” presented at the 1967 Int.Symp. Information Theory, San Remo, Italy.
[23] E.R. Berlekamp, Algebraic Coding Theory. New York: McGraw Hill, 1968(rev. ed., Aegean Park Press: Laguna Hills, CA, 1984.)
[24] K.C. Pohlmann, The Compact Disk Handbook, 2nd ed. Madison, WI: A-REditions, 1992.
[25] M.J. Riley and I.E.G. Richardson, Digital Video Communications.Boston, MA: Artech House, 1997.
[26] G.D. Forney, “Burst-correcting codes for the classic bursty channel,”IEEE Trans. Commun., vol. COM-19, pp. 772–781, Oct. 1971.
[27] G.D. Forney, Concatenated Codes. Cambridge, MA: MIT Press, 1967.
[28] J.L Massey, “Deep space communications and coding: A match made inheaven,” in Advanced Methods for Satellite and Deep SpaceCommunications, J. Hagenauer, Ed. (Lecture Notes in Control andInformation Sciences, vol. 182). Berlin: Springer-Verlag, 1992.
[29] C.E. Shannon, “A mathematical theory of communication,” BSTJ, vol. 27, pp. 379–423, 623–657, 1948.
[30] C.E. Shannon, “Communication in the presence of noise,” Proc. IRE,vol. 37, no. 1, pp. 10–21, Jan. 1949.
[31] S.A. Butman and R.J. McEliece, “The ultimate limits of binary coding fora wideband gaussian channel,” JPL Deep Space Network Progress Rep.42–22, May–June 1974, pp. 78–80, Aug. 15, 1974.
[32] C. Berrou, A. Glavieux, and P. Thitimajshima, “Near Shannon limit error-correcting coding and decoding: Turbo codes,” in IEEE Proc. Int. Conf.Communications, Geneva, Switzerland, May 1993, pp. 1064–1070.
[35] D.J.C. Mackay and R.M. Neal “Near Shannon-limit performance of low-den-sity parity-check codes,” Electron. Lett., vol. 32, pp. 1645–1646, Aug. 1996.
[36] S.-Y. Chung, G.D. Forney, Jr., T.J. Richardson, and R. Urbanke, “On thedesign of low-density parity-check codes within 0.0045 dB of the Shannonlimit,” IEEE Commun. Letters, vol. 5, pp. 58–60, Feb. 2001.
[37] J. Hagenauer, “Iterative decoding of binary block and convolutional codes,”IEEE Trans. Inform. Theory, vol. 42, no. 2, pp. 429–445, Mar. 1996.
[38] D. Divsalar and F. Pollara, “On the design of turbo codes,” JetPropulsion Laboratory, Pasadena, CA, TDA Progress Rep. 42–123, pp.99–121, Nov. 15, 1995.
[39] D. Divsalar and R.J. McEliece, “Effective free distance of turbo codes,”Electron. Lett., vol. 32, no. 5, pp. 445–446, Feb. 29, 1996.
[40] S. Dolinar and D. Divsalar, “Weight distributions for turbo codes usingrandom and nonrandom permutations,” Jet Propulsion Laboratory,Pasadena, CA, TDA Progress Rep. 42–122, pp. 56–65. Aug. 15, 1995.
[41] D. Divsalar and F. Pollara, “Turbo codes for deep-space sommunica-tions,” Jet Propulsion Lab, Pasadena, CA, TDA Progress Rep. 42–120, pp.29–39, Feb. 15, 1995.
[42] D. Divsalar and F. Pollara, “Multiple turbo codes for deep-space commu-nications,” JPL, Pasadena, CA, TDA Progress Rep. 42–121, pp. 66–77,May 15, 1995.
[43] D. Divsalar and F. Pollara, “Turbo codes for PCS applications,” in Proc.ICC ‘95, Seattle, Washington, June 18–22, 1995, vol. 1, pp. 54–59.
[44] L.R. Bahl, J. Cocke, F. Jeinek, and J. Raviv, “Optimal decoding of linearcodes for minimizing symbol error rate,” IEEE Trans. Inform. Theory, vol.IT-20, pp. 248–287, Mar. 1974.
[45] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “Soft output
decoding algorithm in iterative decoding of turbo codes,” Jet PropulsionLab, Pasadena, CA, TDA Progress Rep. 42-124, pp. 63–87, Feb. 15, 1996.
[46] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “A soft-input soft-output maximum a posteriori (MAP) module to decode parallel and serialconcatenated codes,” Jet Propulsion Lab, Pasadena, CA, TDA ProgressRep. 42-127, pp. 63–87, Nov. 15, 1996.
[47] S. Benedetto, D. Divsalar, G. Montorsi, and F. Pollara, “A soft-input soft-output APP module for iterative decoding of concatenated codes,” IEEECommun. Lett., vol. 1, no. 1, pp. 22–24, Jan. 1997.
[48] S. Pietrobon, “Implementation and performance of a turbo/MAPdecoder,” Int. J. Satellite Commun., vol. 16, pp. 23–46, Jan.–Feb. 1998.
[49] P. Robertson, E. Villebrun, and P. Hoeher, “A comparison of optimal andsub-optimal MAP decoding algorithms operating in the log domain,” inProc. ICC’95, Seattle, WA, June 1995, pp. 1009–1013.
[50] F. Burkert and J. Hagenauer, “A serial concatenated coding scheme withiterative turbo and feedback decoding,” in Proc. Int. Symp. Turbo Codes,Sept. 1997, pp. 227–230.
[51] R. Pyndiah, “Near optimum decoding of product codes: Block turbocodes,” IEEE Trans. Commun., vol. 46, no. 8, pp. 1003–1010, Aug. 1998.
[52] T.J. Richardson, M.A. Shokrollahi, and R.L. Urbanke, “Design of capaci-ty-approaching irregular low-density parity-check codes,” IEEE Trans.Inform. Theory, vol. 47, pp. 619–637, Feb. 2001.
[53] Tanner, “A recursive approach to low complexity codes,” IEEE Trans.Inform. Theory, vol. IT-27, pp. 533–547, Sept. 1981.
[54] X.-Y. Hu, E. Eleftheriou, D.-M. Arnold, and A. Dholakia, “Efficientimplementations of the sum product algorithm for decoding LDPCcodes,” in Proc. Globecom, 2001, vol. 2, pp. 25–29.
[55] D. MacKay, “Good error correcting codes based on very sparsematrices,” IEEE Trans. Inform. Theory, vol. 45, pp. 399–431, Mar.1999.
[56] M.P.C. Fossorier, M. Mihaljevic, and H. Imai, “Reduced complexity itera-tive decoding of low-density parity-check codes based on belief propaga-tion,” IEEE Trans. Commun., vol. 47, pp. 673–680, May 1999.
[57] T.J. Richardson and R.L. Urbanke, “Efficient encoding of low-densityparity-check codes,” IEEE Trans. Inform. Theory, vol. 47, pp. 638–656,Feb. 2001.
[58] T.J. Richardson and R. Urbanke, “The capacity of low-density parity-check codes under message-passing decoding,” IEEE Trans. Inform.Theory, vol. 47, no. 2, pp. 599–618, Feb. 2001.
[59] D.J.C. MacKay, S.T. Wilson, and M.C. Davey, “Comparison of construc-tions of irregular Gallager codes,” IEEE Trans. Commun., vol. 47, pp.1449–1454, Oct. 1999.
[60] M.G. Luby, M. Mitzenmacher, M.A Shokrollahi, and D.A. Spielman,“Improved low-density parity check codes using irregular graphs,” IEEETrans. Inform. Theory, vol. 47, pp. 585–598, Feb. 2001.
[61] J. Fan, E. Kurtas, A. Friedman, and S. McLaughlin, “Low-density parity-checkcodes for magnetic recording,” in Proc. 1999 Allerton Conf, pp. 1314–1323.
[62] R. Lucas, M. Fossorier, et al., “Iterative decoding of one-step majority-logic decodable codes based on belief propagation,” IEEE Trans.Commun., vol. 48, pp. 931–937, June 2000.
[64] B. Levine, R. Taylor, and H. Schmit, “Implementation of near Shannonlimit error-correcting codes using reconfigurable hardware,” in Proc. 2000IEEE Symp. Field-Programmable Custom Computing Machines (FCCM),Napa, CA, Apr., 2000, pp. 217–226.
[65] A. Shokrollahi, “LDPC codes: An introduction,” Digital Fountain, Inc.,Fremont, CA, Tech. Rep., Apr. 2, 2003.
[66] F.R. Kschischang, “Codes defined on graphs,” IEEE Commun. Mag., vol.41, no. 8, pp. 118–125, Aug. 2003.
[67] T. Richardson and R. Urbanke, “The renaissance of Gallager’s low-densityparity-check codes,” IEEE Commun. Mag., vol. 41, no. 8, pp. 126–131,Aug. 2003.
[68] E. Yeo and V. Anantharam, “Iterative decoder architectures,” IEEECommun. Mag., vol. 41, no. 8, pp. 132–140, Aug. 2003.
![Scientology Sklar Case 2009 SupremeCt Writ S.Ct.08-9180-Sklar-Cert_Petition[1]](https://static.documents.pub/doc/80x56/577dabcd1a28ab223f8cfa99/scientology-sklar-case-2009-supremect-writ-sct08-9180-sklar-certpetition1.jpg)
