arX

ivc

ond-

mat

010

4079

v1

4 A

pr 2

001

The glassy phase of Gallager codes

Andrea MontanariLaboratoire de Physique Theorique de lrsquoEcole Normale Superieurelowast

24 rue Lhomond 75231 Paris CEDEX 05 FRANCEInternet AndreaMontanarilptensfr

October 1 2006

Abstract

Gallager codes are the best error-correcting codes to-date In this paper we study themby using the tools of statistical mechanics The corresponding statistical mechanics model isa spin model on a sparse random graph The model can be solved by elementary methods(ie without replicas) in a large connectivity limit For low enough temperatures it presentsa completely frozen glassy phase (qEA = 1) The same scenario is shown to hold for finiteconnectivities In this case we adopt the replica approach and exhibit a one-step replicasymmetry breaking order parameter We argue that our ansatz yields the exact solution ofthe model This allows us to determine the whole phase diagram and to understand theperformances of Gallager codes

LPTENS 0119

lowastUMR 8549 Unite Mixte de Recherche du Centre National de la Recherche Scientifique et de lrsquo Ecole NormaleSuperieure

1 Introduction

Information theory [12] deals with the problem of reliable communication through an imper-fect (noisy) communication channel This can be done by properly encoding the informationmessage in such a way to increase its redundancy If a transmission error occurs due to thenoise the correct message can be restored by exploiting this redundancy

The price to pay for error-correction to be possible is to increase the length of the transmit-ted message ie to decrease the information rate through the channel In 1948 C E Shan-non [3] computed the maximal achievable rate at which information can be transmitted througha given communication channel (the so-called capacity of the channel) Since then a lot of workhas been spent for constructing practical error-correcting codes that could realize Shannon pre-diction ie that could saturate the channel capacity

In the past few years it has become progressively clear that such an objective is not un-reachable It has become possible to construct error-correcting codes which remain effectiveextremely near to the Shannon capacity [4] The reasons of this revolution have been theinvention of ldquoturbo codesrdquo [5] and the re-invention of ldquolow-density parity check codesrdquo (LD-PCC) [6] The last ones [7] were proposed for the first time by R Gallager in 1962 but weresoon forgotten afterwards probably because of the lack of computational resources at thattime

As it has been shown by N Sourlas [8ndash10] error-correcting codes can be mapped ontodisordered spin models This mapping allows to employ statistical mechanics techniques toinvestigate the behavior of the former Both turbo codes [11 12] and LDPCC [13ndash19] havebeen already studied using this approach However all previous studies were restricted toparticular regions of the phase diagram The principal technical reason was the difficulty ofimplementing replica symmetry breaking in finite connectivity systems

In this work we focus on regular Gallager codes (a particular family of LDPCC) and weaddress the fundamental problem of determining the corresponding phase diagram There aretwo type of motivations for such a task to be undertaken First the spin model correspondingto Gallager codes is a disordered spin model on a diluted graph The study of such systemshas greatly improved our understanding of glassy systems over the last few years Secondit is of great practical importance to have a complete quantitative picture of the behavior ofGallager codes For instance the existence of a glassy phase can have important effects on thedecoding algorithms and the knowledge of the phase diagram can be used to improve them

The model is presented Sec 2 In Sec 3 we prove some exact properties which hold atinverse temperature β = 1 The line β = 1 can be regarded as the Nishimori line [20] of thephase diagram In Sec 4 we solve the model in the large connectivity limit We show thatit becomes identical to a simplified model which we call the random codeword model (RCM)The RCM is shown to have a freezing phase transition analogous to the one of the randomenergy model (REM) [21] In Sec 5 we adopt the replica approach [22] and prove that thesame scenario applies for finite connectivities In particular we construct a replica symmetrybreaking solution of the saddle point equations The proposed solution is much simpler thanthe generic one-step replica symmetry breaking solution Rather than being parametrized by afunctional over a probability space [23] it depends simply upon the probability distribution ofa local field Such a probability distribution can be easily computed numerically It can be alsoobtained from a large connectivity expansion see Sec 6 In Sec 7 we compute the finite-sizecorrections of the free energy for the RCM and compare the result with exact enumerationsFinally in Sec 8 we discuss the validity of our replica symmetry breaking ansatz

2

2 The model

Let us suppose we want to transmit an information message consisting of L bits There are2L such messages Each of them is encoded in a string of N gt L bits (codewords)

This motivates the following model There are 2L possible configurations of the system (thecodewords) each one corresponding to a distinct sequence of N gt L bits We shall denote the

codewords as x(α) = (x(α)1 x

(α)N ) with α = 1 2L The set of codewords C is a linear

space This means that 0 equiv (0 0) isin C and that if x(α) x(β) isin C then x(α) + x(β) isin C(where the sum has to be carried modulo 2)

Like any linear space the set of codewords C can be specified as the kernel of a linearoperator In other words we can find an M by N matrix C = Ciji=1M j=1N withCij = 0 1 and M = N minus L such that

C = x(α) α = 1 2L = x isin 0 1N Cx = 0 (mod2) (21)

The condition Cx = 0 (mod2) can be regarded as a set of M linear equations (called constraints

or parity checks) of the form

Ci1x1 + Ci2x2 + + CiNxN = 0 (mod2) (22)

with i = 1 M To each bit xi i = 1 N we assign an a priori probability distribution pi(xi) In the

information-theory context the a priori distributions pi(xi) are induced by the observation ofthe channel output and by the knowledge of the statistical properties of the channel We areinterested in studying the induced probability distribution over the codewords x(α) In otherwords we want to consider the following probability distribution over the strings x of N bits

P (x) =1

Zδ[Cx]

Nprod

i=1

pi(xi) (23)

where Z is a normalization constant δ[z] = 1 if z = 0 (mod2) and δ[z] = 0 otherwiseThere are several graphical representations of the above model The most used in the

coding theory community makes use of the so-called Tanner graph [24] cf Fig 1 This is abipartite graph which is constructed as follows A node on the left is associated to each binaryvariable xj and a node on the right to each constraint ie to each linear equation (22) withi = 1 M There are therefore N left nodes (variable nodes) and M right nodes (checknodes) A given check i is connected to the variables xj which appear with nonzero coefficientin the corresponding equation (22)

The model (23) has a spin-wise formulation [13ndash19] which we shall employ hereafter Wereplace any bit sequence x = (x1 xN ) with a spin configuration σ = (σ1 σN ) whereσi = (minus1)xi The constraints (22) on the sums of bits xi get translated into constraints onthe product of spins σi These have the form

σωi equivprod

jisinωi

σj = +1 (24)

where ωi = j isin 1 N Cij = 1 The other ingredient of the model are the a priori

probability distributions pi(xi) They can be encoded into properly chosen magnetic fieldspi(xi) = eβhiσi(2 cosh βhi) with 2βhi = log(pi(0)pi(1)) where we introduced the inverse

3

Figure 1 Two Tanner graphs a regular one with (k l) = (6 3) on the left and an irregular one onthe right In both cases N = 8 M = 4 (and therefore the rate is R = 12)

temperature β for later convenience With these building blocks we can write down the spinmodel equivalent of Eq (23)

P (σ) =1

Z(β)

Mprod

j=1

δ[σωj +1] exp

(

β

Nsum

i=1

hiσi

)

(25)

where δ[a b] is the Kronecker delta function This can be regarded as a spin model with infinitestrength multi-spin interactions (which enforce σωj = +1) and a random magnetic field

Instead of insisting on the motivations for the probabilistic model (25) coming from codingtheory we shall remark that as it stands it is remarkably general Any spin-model hamiltonianH(σ) = minus

sumi1ip

Jiiipσi1 σip can be written in the form (25) This can be done by

introducing the auxiliary spin variables σi1ip The Kronecker delta functions in Eq (25) canbe used to enforce σi1ip = σi1 σip The couplings Jiiip become magnetic fields acting onthe variables σi1ip

Untill now we have been pretty generic in the presentation of the model In order to bemore precise we have to choose the constraint matrix C and the magnetic fields hii=1N

Following Gallager [7] we shall take C to be random and sparse More precisely C will beconstrained to have k non-zero elements for each row and l non-zero elements for each column(with l lt k) and not to have two identical rows1 This choice corresponds to taking the Tannergraph (cf Fig 1) as a random bipartite graph with variable (left) nodes of fixed degree land check (right) nodes of degree k We shall choose among the matrices of this ensemble

with flat probability distribution We shall use the pair (k l) to denote the spin model (or theerror-correcting code) defined by this ensemble of matrices An important characteristic of thecode is its rate R = 1 minus lk which measures the redundancy of the encoded message (infactR = LN)

The magnetic fields hi will be random iid variables with probability distribution ph(hi)We consider ph(hi) to be biased towards positive values of hi (ie

intdhi ph(hi)hi gt 0) We

1Remark that with this choice some of the parity check equations (22) may be linearly dependent Howeversuch an event is rare for k gt l [7]

4

shall refer often to two simple examples the two-peak distribution

ph(hi) = (1 minus p)δ(hi minus h0) + pδ(hi + h0) (26)

with p lt 12 and h0 gt 0 and the gaussian distribution

ph(hi) =1radic2πh2

exp

minus

(hi minus h0)2

2h2

(27)

with h0 gt 0 It can be shown that if the model describe communication through a noisyldquosymmetricrdquo channel the condition

ph(minushi) = eminus2hiph(hi) (28)

follows This implies h0 = (12) log(1 minus p)p for the example (26) (which corresponds to abinary symmetric channel) and h0 = h2 for the example (27) (corresponding to a gaussianchannel) Hereafter we shall denote with 〈middot〉h and 〈middot〉C the averages with respect to the magneticfields hi and the ensemble of matrices C

More details on the model introduced in this Section and on analogous examples can befound in Refs [11ndash19]

3 The Nishimori line

Nishimori [20 25] showed that the physics of disordered spin models simplifies considerablyon a particular line in the phase diagram In particular it has been recently shown [26] thatreplica symmetry breaking is absent on this line The Nishimori line plays a distinguishedrole in the correspondence between error-correcting codes and disordered spin models Asshown in Refs [27 28] maximum a posteriori symbol probability (MAP) decoding for a givenerror-correcting code is equivalent to computing expectation values on the Nishimori line ofthe corresponding spin model

In this Section we extend the results concerning the Nishimori line to the model (25) Weshall consider a generic magnetic field distribution ph(hi) satisfying Eq (28) In this casethe Nishimori line is simply given by β = 1 Although the proofs are very similar to the onesof Refs [25 26] we present them for sake of completeness Some consequences of the exactresults of this Section will be outlined in Sec 5

Let us start with some convention Notice that there are two sources of disorder in ourmodel (23) the magnetic field hi (which is determined by the channel output) and the checkmatrix C Different C correspond to different error-correcting codes In this Section we keepthe parity check matrix C fixed and average uniquely over the random magnetic fields hiwith distribution ph(hi) Our results will remain valid after averaging with respect to anyensemble of check matrices C (ie to any ensemble of codes) It is convenient to introduce thenotation δC[σ] to denote the product of Kronecker delta functions in Eq (25) In other wordsδC[σ] = 1 if and only if σ satisfies all the parity checks encoded in C ie if the correspondingstring of bits x is a codeword We assume that the parity check matrix C selects 2L = 2NR

codewords This means that there are 2L distinct configurations σ such that δC[σ] = 1 Finallywe shall take the distribution of the random fields to satisfy the identity (28)

We start by writing down the definition of the (field averaged) free energy density fC(β)for a given parity check matrix C

minus βNfC(β) =

int +infin

minusinfin

Nprod

i=1

dhi ph(hi) log

sum

σ

δC[σ] eβsum

i hiσi

(31)

5

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

1 Introduction

Information theory [12] deals with the problem of reliable communication through an imper-fect (noisy) communication channel This can be done by properly encoding the informationmessage in such a way to increase its redundancy If a transmission error occurs due to thenoise the correct message can be restored by exploiting this redundancy

The price to pay for error-correction to be possible is to increase the length of the transmit-ted message ie to decrease the information rate through the channel In 1948 C E Shan-non [3] computed the maximal achievable rate at which information can be transmitted througha given communication channel (the so-called capacity of the channel) Since then a lot of workhas been spent for constructing practical error-correcting codes that could realize Shannon pre-diction ie that could saturate the channel capacity

In the past few years it has become progressively clear that such an objective is not un-reachable It has become possible to construct error-correcting codes which remain effectiveextremely near to the Shannon capacity [4] The reasons of this revolution have been theinvention of ldquoturbo codesrdquo [5] and the re-invention of ldquolow-density parity check codesrdquo (LD-PCC) [6] The last ones [7] were proposed for the first time by R Gallager in 1962 but weresoon forgotten afterwards probably because of the lack of computational resources at thattime

As it has been shown by N Sourlas [8ndash10] error-correcting codes can be mapped ontodisordered spin models This mapping allows to employ statistical mechanics techniques toinvestigate the behavior of the former Both turbo codes [11 12] and LDPCC [13ndash19] havebeen already studied using this approach However all previous studies were restricted toparticular regions of the phase diagram The principal technical reason was the difficulty ofimplementing replica symmetry breaking in finite connectivity systems

In this work we focus on regular Gallager codes (a particular family of LDPCC) and weaddress the fundamental problem of determining the corresponding phase diagram There aretwo type of motivations for such a task to be undertaken First the spin model correspondingto Gallager codes is a disordered spin model on a diluted graph The study of such systemshas greatly improved our understanding of glassy systems over the last few years Secondit is of great practical importance to have a complete quantitative picture of the behavior ofGallager codes For instance the existence of a glassy phase can have important effects on thedecoding algorithms and the knowledge of the phase diagram can be used to improve them

The model is presented Sec 2 In Sec 3 we prove some exact properties which hold atinverse temperature β = 1 The line β = 1 can be regarded as the Nishimori line [20] of thephase diagram In Sec 4 we solve the model in the large connectivity limit We show thatit becomes identical to a simplified model which we call the random codeword model (RCM)The RCM is shown to have a freezing phase transition analogous to the one of the randomenergy model (REM) [21] In Sec 5 we adopt the replica approach [22] and prove that thesame scenario applies for finite connectivities In particular we construct a replica symmetrybreaking solution of the saddle point equations The proposed solution is much simpler thanthe generic one-step replica symmetry breaking solution Rather than being parametrized by afunctional over a probability space [23] it depends simply upon the probability distribution ofa local field Such a probability distribution can be easily computed numerically It can be alsoobtained from a large connectivity expansion see Sec 6 In Sec 7 we compute the finite-sizecorrections of the free energy for the RCM and compare the result with exact enumerationsFinally in Sec 8 we discuss the validity of our replica symmetry breaking ansatz

2

2 The model

Let us suppose we want to transmit an information message consisting of L bits There are2L such messages Each of them is encoded in a string of N gt L bits (codewords)

This motivates the following model There are 2L possible configurations of the system (thecodewords) each one corresponding to a distinct sequence of N gt L bits We shall denote the

codewords as x(α) = (x(α)1 x

(α)N ) with α = 1 2L The set of codewords C is a linear

space This means that 0 equiv (0 0) isin C and that if x(α) x(β) isin C then x(α) + x(β) isin C(where the sum has to be carried modulo 2)

Like any linear space the set of codewords C can be specified as the kernel of a linearoperator In other words we can find an M by N matrix C = Ciji=1M j=1N withCij = 0 1 and M = N minus L such that

C = x(α) α = 1 2L = x isin 0 1N Cx = 0 (mod2) (21)

The condition Cx = 0 (mod2) can be regarded as a set of M linear equations (called constraints

or parity checks) of the form

Ci1x1 + Ci2x2 + + CiNxN = 0 (mod2) (22)

with i = 1 M To each bit xi i = 1 N we assign an a priori probability distribution pi(xi) In the

information-theory context the a priori distributions pi(xi) are induced by the observation ofthe channel output and by the knowledge of the statistical properties of the channel We areinterested in studying the induced probability distribution over the codewords x(α) In otherwords we want to consider the following probability distribution over the strings x of N bits

P (x) =1

Zδ[Cx]

Nprod

i=1

pi(xi) (23)

where Z is a normalization constant δ[z] = 1 if z = 0 (mod2) and δ[z] = 0 otherwiseThere are several graphical representations of the above model The most used in the

coding theory community makes use of the so-called Tanner graph [24] cf Fig 1 This is abipartite graph which is constructed as follows A node on the left is associated to each binaryvariable xj and a node on the right to each constraint ie to each linear equation (22) withi = 1 M There are therefore N left nodes (variable nodes) and M right nodes (checknodes) A given check i is connected to the variables xj which appear with nonzero coefficientin the corresponding equation (22)

The model (23) has a spin-wise formulation [13ndash19] which we shall employ hereafter Wereplace any bit sequence x = (x1 xN ) with a spin configuration σ = (σ1 σN ) whereσi = (minus1)xi The constraints (22) on the sums of bits xi get translated into constraints onthe product of spins σi These have the form

σωi equivprod

jisinωi

σj = +1 (24)

where ωi = j isin 1 N Cij = 1 The other ingredient of the model are the a priori

probability distributions pi(xi) They can be encoded into properly chosen magnetic fieldspi(xi) = eβhiσi(2 cosh βhi) with 2βhi = log(pi(0)pi(1)) where we introduced the inverse

3

Figure 1 Two Tanner graphs a regular one with (k l) = (6 3) on the left and an irregular one onthe right In both cases N = 8 M = 4 (and therefore the rate is R = 12)

temperature β for later convenience With these building blocks we can write down the spinmodel equivalent of Eq (23)

P (σ) =1

Z(β)

Mprod

j=1

δ[σωj +1] exp

(

β

Nsum

i=1

hiσi

)

(25)

where δ[a b] is the Kronecker delta function This can be regarded as a spin model with infinitestrength multi-spin interactions (which enforce σωj = +1) and a random magnetic field

Instead of insisting on the motivations for the probabilistic model (25) coming from codingtheory we shall remark that as it stands it is remarkably general Any spin-model hamiltonianH(σ) = minus

sumi1ip

Jiiipσi1 σip can be written in the form (25) This can be done by

introducing the auxiliary spin variables σi1ip The Kronecker delta functions in Eq (25) canbe used to enforce σi1ip = σi1 σip The couplings Jiiip become magnetic fields acting onthe variables σi1ip

Untill now we have been pretty generic in the presentation of the model In order to bemore precise we have to choose the constraint matrix C and the magnetic fields hii=1N

Following Gallager [7] we shall take C to be random and sparse More precisely C will beconstrained to have k non-zero elements for each row and l non-zero elements for each column(with l lt k) and not to have two identical rows1 This choice corresponds to taking the Tannergraph (cf Fig 1) as a random bipartite graph with variable (left) nodes of fixed degree land check (right) nodes of degree k We shall choose among the matrices of this ensemble

with flat probability distribution We shall use the pair (k l) to denote the spin model (or theerror-correcting code) defined by this ensemble of matrices An important characteristic of thecode is its rate R = 1 minus lk which measures the redundancy of the encoded message (infactR = LN)

The magnetic fields hi will be random iid variables with probability distribution ph(hi)We consider ph(hi) to be biased towards positive values of hi (ie

intdhi ph(hi)hi gt 0) We

1Remark that with this choice some of the parity check equations (22) may be linearly dependent Howeversuch an event is rare for k gt l [7]

4

shall refer often to two simple examples the two-peak distribution

ph(hi) = (1 minus p)δ(hi minus h0) + pδ(hi + h0) (26)

with p lt 12 and h0 gt 0 and the gaussian distribution

ph(hi) =1radic2πh2

exp

minus

(hi minus h0)2

2h2

(27)

with h0 gt 0 It can be shown that if the model describe communication through a noisyldquosymmetricrdquo channel the condition

ph(minushi) = eminus2hiph(hi) (28)

follows This implies h0 = (12) log(1 minus p)p for the example (26) (which corresponds to abinary symmetric channel) and h0 = h2 for the example (27) (corresponding to a gaussianchannel) Hereafter we shall denote with 〈middot〉h and 〈middot〉C the averages with respect to the magneticfields hi and the ensemble of matrices C

More details on the model introduced in this Section and on analogous examples can befound in Refs [11ndash19]

3 The Nishimori line

Nishimori [20 25] showed that the physics of disordered spin models simplifies considerablyon a particular line in the phase diagram In particular it has been recently shown [26] thatreplica symmetry breaking is absent on this line The Nishimori line plays a distinguishedrole in the correspondence between error-correcting codes and disordered spin models Asshown in Refs [27 28] maximum a posteriori symbol probability (MAP) decoding for a givenerror-correcting code is equivalent to computing expectation values on the Nishimori line ofthe corresponding spin model

In this Section we extend the results concerning the Nishimori line to the model (25) Weshall consider a generic magnetic field distribution ph(hi) satisfying Eq (28) In this casethe Nishimori line is simply given by β = 1 Although the proofs are very similar to the onesof Refs [25 26] we present them for sake of completeness Some consequences of the exactresults of this Section will be outlined in Sec 5

Let us start with some convention Notice that there are two sources of disorder in ourmodel (23) the magnetic field hi (which is determined by the channel output) and the checkmatrix C Different C correspond to different error-correcting codes In this Section we keepthe parity check matrix C fixed and average uniquely over the random magnetic fields hiwith distribution ph(hi) Our results will remain valid after averaging with respect to anyensemble of check matrices C (ie to any ensemble of codes) It is convenient to introduce thenotation δC[σ] to denote the product of Kronecker delta functions in Eq (25) In other wordsδC[σ] = 1 if and only if σ satisfies all the parity checks encoded in C ie if the correspondingstring of bits x is a codeword We assume that the parity check matrix C selects 2L = 2NR

codewords This means that there are 2L distinct configurations σ such that δC[σ] = 1 Finallywe shall take the distribution of the random fields to satisfy the identity (28)

We start by writing down the definition of the (field averaged) free energy density fC(β)for a given parity check matrix C

minus βNfC(β) =

int +infin

minusinfin

Nprod

i=1

dhi ph(hi) log

sum

σ

δC[σ] eβsum

i hiσi

(31)

5

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

2 The model

Let us suppose we want to transmit an information message consisting of L bits There are2L such messages Each of them is encoded in a string of N gt L bits (codewords)

This motivates the following model There are 2L possible configurations of the system (thecodewords) each one corresponding to a distinct sequence of N gt L bits We shall denote the

codewords as x(α) = (x(α)1 x

(α)N ) with α = 1 2L The set of codewords C is a linear

space This means that 0 equiv (0 0) isin C and that if x(α) x(β) isin C then x(α) + x(β) isin C(where the sum has to be carried modulo 2)

Like any linear space the set of codewords C can be specified as the kernel of a linearoperator In other words we can find an M by N matrix C = Ciji=1M j=1N withCij = 0 1 and M = N minus L such that

C = x(α) α = 1 2L = x isin 0 1N Cx = 0 (mod2) (21)

The condition Cx = 0 (mod2) can be regarded as a set of M linear equations (called constraints

or parity checks) of the form

Ci1x1 + Ci2x2 + + CiNxN = 0 (mod2) (22)

with i = 1 M To each bit xi i = 1 N we assign an a priori probability distribution pi(xi) In the

information-theory context the a priori distributions pi(xi) are induced by the observation ofthe channel output and by the knowledge of the statistical properties of the channel We areinterested in studying the induced probability distribution over the codewords x(α) In otherwords we want to consider the following probability distribution over the strings x of N bits

P (x) =1

Zδ[Cx]

Nprod

i=1

pi(xi) (23)

where Z is a normalization constant δ[z] = 1 if z = 0 (mod2) and δ[z] = 0 otherwiseThere are several graphical representations of the above model The most used in the

coding theory community makes use of the so-called Tanner graph [24] cf Fig 1 This is abipartite graph which is constructed as follows A node on the left is associated to each binaryvariable xj and a node on the right to each constraint ie to each linear equation (22) withi = 1 M There are therefore N left nodes (variable nodes) and M right nodes (checknodes) A given check i is connected to the variables xj which appear with nonzero coefficientin the corresponding equation (22)

The model (23) has a spin-wise formulation [13ndash19] which we shall employ hereafter Wereplace any bit sequence x = (x1 xN ) with a spin configuration σ = (σ1 σN ) whereσi = (minus1)xi The constraints (22) on the sums of bits xi get translated into constraints onthe product of spins σi These have the form

σωi equivprod

jisinωi

σj = +1 (24)

where ωi = j isin 1 N Cij = 1 The other ingredient of the model are the a priori

probability distributions pi(xi) They can be encoded into properly chosen magnetic fieldspi(xi) = eβhiσi(2 cosh βhi) with 2βhi = log(pi(0)pi(1)) where we introduced the inverse

3

Figure 1 Two Tanner graphs a regular one with (k l) = (6 3) on the left and an irregular one onthe right In both cases N = 8 M = 4 (and therefore the rate is R = 12)

temperature β for later convenience With these building blocks we can write down the spinmodel equivalent of Eq (23)

P (σ) =1

Z(β)

Mprod

j=1

δ[σωj +1] exp

(

β

Nsum

i=1

hiσi

)

(25)

where δ[a b] is the Kronecker delta function This can be regarded as a spin model with infinitestrength multi-spin interactions (which enforce σωj = +1) and a random magnetic field

Instead of insisting on the motivations for the probabilistic model (25) coming from codingtheory we shall remark that as it stands it is remarkably general Any spin-model hamiltonianH(σ) = minus

sumi1ip

Jiiipσi1 σip can be written in the form (25) This can be done by

introducing the auxiliary spin variables σi1ip The Kronecker delta functions in Eq (25) canbe used to enforce σi1ip = σi1 σip The couplings Jiiip become magnetic fields acting onthe variables σi1ip

Untill now we have been pretty generic in the presentation of the model In order to bemore precise we have to choose the constraint matrix C and the magnetic fields hii=1N

Following Gallager [7] we shall take C to be random and sparse More precisely C will beconstrained to have k non-zero elements for each row and l non-zero elements for each column(with l lt k) and not to have two identical rows1 This choice corresponds to taking the Tannergraph (cf Fig 1) as a random bipartite graph with variable (left) nodes of fixed degree land check (right) nodes of degree k We shall choose among the matrices of this ensemble

with flat probability distribution We shall use the pair (k l) to denote the spin model (or theerror-correcting code) defined by this ensemble of matrices An important characteristic of thecode is its rate R = 1 minus lk which measures the redundancy of the encoded message (infactR = LN)

The magnetic fields hi will be random iid variables with probability distribution ph(hi)We consider ph(hi) to be biased towards positive values of hi (ie

intdhi ph(hi)hi gt 0) We

1Remark that with this choice some of the parity check equations (22) may be linearly dependent Howeversuch an event is rare for k gt l [7]

4

shall refer often to two simple examples the two-peak distribution

ph(hi) = (1 minus p)δ(hi minus h0) + pδ(hi + h0) (26)

with p lt 12 and h0 gt 0 and the gaussian distribution

ph(hi) =1radic2πh2

exp

minus

(hi minus h0)2

2h2

(27)

with h0 gt 0 It can be shown that if the model describe communication through a noisyldquosymmetricrdquo channel the condition

ph(minushi) = eminus2hiph(hi) (28)

follows This implies h0 = (12) log(1 minus p)p for the example (26) (which corresponds to abinary symmetric channel) and h0 = h2 for the example (27) (corresponding to a gaussianchannel) Hereafter we shall denote with 〈middot〉h and 〈middot〉C the averages with respect to the magneticfields hi and the ensemble of matrices C

More details on the model introduced in this Section and on analogous examples can befound in Refs [11ndash19]

3 The Nishimori line

Nishimori [20 25] showed that the physics of disordered spin models simplifies considerablyon a particular line in the phase diagram In particular it has been recently shown [26] thatreplica symmetry breaking is absent on this line The Nishimori line plays a distinguishedrole in the correspondence between error-correcting codes and disordered spin models Asshown in Refs [27 28] maximum a posteriori symbol probability (MAP) decoding for a givenerror-correcting code is equivalent to computing expectation values on the Nishimori line ofthe corresponding spin model

In this Section we extend the results concerning the Nishimori line to the model (25) Weshall consider a generic magnetic field distribution ph(hi) satisfying Eq (28) In this casethe Nishimori line is simply given by β = 1 Although the proofs are very similar to the onesof Refs [25 26] we present them for sake of completeness Some consequences of the exactresults of this Section will be outlined in Sec 5

Let us start with some convention Notice that there are two sources of disorder in ourmodel (23) the magnetic field hi (which is determined by the channel output) and the checkmatrix C Different C correspond to different error-correcting codes In this Section we keepthe parity check matrix C fixed and average uniquely over the random magnetic fields hiwith distribution ph(hi) Our results will remain valid after averaging with respect to anyensemble of check matrices C (ie to any ensemble of codes) It is convenient to introduce thenotation δC[σ] to denote the product of Kronecker delta functions in Eq (25) In other wordsδC[σ] = 1 if and only if σ satisfies all the parity checks encoded in C ie if the correspondingstring of bits x is a codeword We assume that the parity check matrix C selects 2L = 2NR

codewords This means that there are 2L distinct configurations σ such that δC[σ] = 1 Finallywe shall take the distribution of the random fields to satisfy the identity (28)

We start by writing down the definition of the (field averaged) free energy density fC(β)for a given parity check matrix C

minus βNfC(β) =

int +infin

minusinfin

Nprod

i=1

dhi ph(hi) log

sum

σ

δC[σ] eβsum

i hiσi

(31)

5

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

Figure 1 Two Tanner graphs a regular one with (k l) = (6 3) on the left and an irregular one onthe right In both cases N = 8 M = 4 (and therefore the rate is R = 12)

temperature β for later convenience With these building blocks we can write down the spinmodel equivalent of Eq (23)

P (σ) =1

Z(β)

Mprod

j=1

δ[σωj +1] exp

(

β

Nsum

i=1

hiσi

)

(25)

where δ[a b] is the Kronecker delta function This can be regarded as a spin model with infinitestrength multi-spin interactions (which enforce σωj = +1) and a random magnetic field

Instead of insisting on the motivations for the probabilistic model (25) coming from codingtheory we shall remark that as it stands it is remarkably general Any spin-model hamiltonianH(σ) = minus

sumi1ip

Jiiipσi1 σip can be written in the form (25) This can be done by

introducing the auxiliary spin variables σi1ip The Kronecker delta functions in Eq (25) canbe used to enforce σi1ip = σi1 σip The couplings Jiiip become magnetic fields acting onthe variables σi1ip

Untill now we have been pretty generic in the presentation of the model In order to bemore precise we have to choose the constraint matrix C and the magnetic fields hii=1N

Following Gallager [7] we shall take C to be random and sparse More precisely C will beconstrained to have k non-zero elements for each row and l non-zero elements for each column(with l lt k) and not to have two identical rows1 This choice corresponds to taking the Tannergraph (cf Fig 1) as a random bipartite graph with variable (left) nodes of fixed degree land check (right) nodes of degree k We shall choose among the matrices of this ensemble

with flat probability distribution We shall use the pair (k l) to denote the spin model (or theerror-correcting code) defined by this ensemble of matrices An important characteristic of thecode is its rate R = 1 minus lk which measures the redundancy of the encoded message (infactR = LN)

The magnetic fields hi will be random iid variables with probability distribution ph(hi)We consider ph(hi) to be biased towards positive values of hi (ie

intdhi ph(hi)hi gt 0) We

1Remark that with this choice some of the parity check equations (22) may be linearly dependent Howeversuch an event is rare for k gt l [7]

4

shall refer often to two simple examples the two-peak distribution

ph(hi) = (1 minus p)δ(hi minus h0) + pδ(hi + h0) (26)

with p lt 12 and h0 gt 0 and the gaussian distribution

ph(hi) =1radic2πh2

exp

minus

(hi minus h0)2

2h2

(27)

with h0 gt 0 It can be shown that if the model describe communication through a noisyldquosymmetricrdquo channel the condition

ph(minushi) = eminus2hiph(hi) (28)

follows This implies h0 = (12) log(1 minus p)p for the example (26) (which corresponds to abinary symmetric channel) and h0 = h2 for the example (27) (corresponding to a gaussianchannel) Hereafter we shall denote with 〈middot〉h and 〈middot〉C the averages with respect to the magneticfields hi and the ensemble of matrices C

More details on the model introduced in this Section and on analogous examples can befound in Refs [11ndash19]

3 The Nishimori line

Nishimori [20 25] showed that the physics of disordered spin models simplifies considerablyon a particular line in the phase diagram In particular it has been recently shown [26] thatreplica symmetry breaking is absent on this line The Nishimori line plays a distinguishedrole in the correspondence between error-correcting codes and disordered spin models Asshown in Refs [27 28] maximum a posteriori symbol probability (MAP) decoding for a givenerror-correcting code is equivalent to computing expectation values on the Nishimori line ofthe corresponding spin model

In this Section we extend the results concerning the Nishimori line to the model (25) Weshall consider a generic magnetic field distribution ph(hi) satisfying Eq (28) In this casethe Nishimori line is simply given by β = 1 Although the proofs are very similar to the onesof Refs [25 26] we present them for sake of completeness Some consequences of the exactresults of this Section will be outlined in Sec 5

Let us start with some convention Notice that there are two sources of disorder in ourmodel (23) the magnetic field hi (which is determined by the channel output) and the checkmatrix C Different C correspond to different error-correcting codes In this Section we keepthe parity check matrix C fixed and average uniquely over the random magnetic fields hiwith distribution ph(hi) Our results will remain valid after averaging with respect to anyensemble of check matrices C (ie to any ensemble of codes) It is convenient to introduce thenotation δC[σ] to denote the product of Kronecker delta functions in Eq (25) In other wordsδC[σ] = 1 if and only if σ satisfies all the parity checks encoded in C ie if the correspondingstring of bits x is a codeword We assume that the parity check matrix C selects 2L = 2NR

codewords This means that there are 2L distinct configurations σ such that δC[σ] = 1 Finallywe shall take the distribution of the random fields to satisfy the identity (28)

We start by writing down the definition of the (field averaged) free energy density fC(β)for a given parity check matrix C

minus βNfC(β) =

int +infin

minusinfin

Nprod

i=1

dhi ph(hi) log

sum

σ

δC[σ] eβsum

i hiσi

(31)

5

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

shall refer often to two simple examples the two-peak distribution

ph(hi) = (1 minus p)δ(hi minus h0) + pδ(hi + h0) (26)

with p lt 12 and h0 gt 0 and the gaussian distribution

ph(hi) =1radic2πh2

exp

minus

(hi minus h0)2

2h2

(27)

with h0 gt 0 It can be shown that if the model describe communication through a noisyldquosymmetricrdquo channel the condition

ph(minushi) = eminus2hiph(hi) (28)

follows This implies h0 = (12) log(1 minus p)p for the example (26) (which corresponds to abinary symmetric channel) and h0 = h2 for the example (27) (corresponding to a gaussianchannel) Hereafter we shall denote with 〈middot〉h and 〈middot〉C the averages with respect to the magneticfields hi and the ensemble of matrices C

More details on the model introduced in this Section and on analogous examples can befound in Refs [11ndash19]

3 The Nishimori line

Nishimori [20 25] showed that the physics of disordered spin models simplifies considerablyon a particular line in the phase diagram In particular it has been recently shown [26] thatreplica symmetry breaking is absent on this line The Nishimori line plays a distinguishedrole in the correspondence between error-correcting codes and disordered spin models Asshown in Refs [27 28] maximum a posteriori symbol probability (MAP) decoding for a givenerror-correcting code is equivalent to computing expectation values on the Nishimori line ofthe corresponding spin model

In this Section we extend the results concerning the Nishimori line to the model (25) Weshall consider a generic magnetic field distribution ph(hi) satisfying Eq (28) In this casethe Nishimori line is simply given by β = 1 Although the proofs are very similar to the onesof Refs [25 26] we present them for sake of completeness Some consequences of the exactresults of this Section will be outlined in Sec 5

Let us start with some convention Notice that there are two sources of disorder in ourmodel (23) the magnetic field hi (which is determined by the channel output) and the checkmatrix C Different C correspond to different error-correcting codes In this Section we keepthe parity check matrix C fixed and average uniquely over the random magnetic fields hiwith distribution ph(hi) Our results will remain valid after averaging with respect to anyensemble of check matrices C (ie to any ensemble of codes) It is convenient to introduce thenotation δC[σ] to denote the product of Kronecker delta functions in Eq (25) In other wordsδC[σ] = 1 if and only if σ satisfies all the parity checks encoded in C ie if the correspondingstring of bits x is a codeword We assume that the parity check matrix C selects 2L = 2NR

codewords This means that there are 2L distinct configurations σ such that δC[σ] = 1 Finallywe shall take the distribution of the random fields to satisfy the identity (28)

We start by writing down the definition of the (field averaged) free energy density fC(β)for a given parity check matrix C

minus βNfC(β) =

int +infin

minusinfin

Nprod

i=1

dhi ph(hi) log

sum

σ

δC[σ] eβsum

i hiσi

(31)

5

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

Then we notice following Ref [25] that the integral over the field hi can be decomposed intoan integral over its absolute value and a sum over its sign Using Eq (28) we get for anyfunction O(hi)

int +infin

minusinfindhi ph(hi)O(hi) =

int +infin

0dhi ρ(hi)

sum

τi

ehiτiO(hiτi) (32)

where ρ(hi) is given by

ρ(hi) =ph(hi) + ph(minushi)

2 cosh hi (33)

By using the decomposition (32) into the definition (31) we get

minus βNfC(β) =

int +infin

0

Nprod

i=1

dhi ρ(hi)sum

τ

esum

i hiτi log

sum

σ

δC[σ] eβsum

i hiτiσi

(34)

To be more compact we shall use hereafter the shorthand 〈middot〉ρ equivint +infin0

prodNi=1 dhi ρ(hi) (middot) for the

average over the absolute values of the fields hiThe next step consists in performing a gauge transformation τi rarr σprime

iτi σi rarr σprimeiσi Because

of the constraint term δC[σ] the free energy (34) is not invariant with respect to such atransformation for a generic choice of σprime

i However if δC[σprime] = 1 ie if σprime is a codewordthen the gauge transformation leaves invariant the free energy We can sum over all suchldquoallowedrdquo transformations and divide by their number namely 2NR obtaining

minus βNfC(β) =

lang1

2NR

sum

τ

sum

σprime

δC[σprime]esum

i hiτiσprime

i log

sum

σ

δC[σ] eβsum

i hiτiσi

rang

ρ

(35)

where the constraint δC[σprime] force the gauge transformation σprime to be an allowed oneIn Eq (35) we wrote the sums over quenched and dynamical variables in a symmetric

form This allows to derive several exact identities for β = 1 where the symmetry is completeIn particular let us consider the internal energy per spin ǫC(β) = partβ(βfC(β)) From Eq (35)we get

ǫC(β = 1) = minus

lang1

2NR

sum

τ

sum

σ

δC[σ]

(1

N

Nsum

i=1

hiτiσi

)

esum

i hiτiσi

rang

ρ

(36)

We can now perform a second gauge transformation τi rarr τiσi sum over the σi using theconstraint and finally sum over the τi We obtain ǫC(β = 1) = minus〈h tanh h〉h Analogously toRef [25] we can further simplify this result obtaining

ǫC(β = 1) = minus〈h〉h (37)

which is the first important result of this SectionWe want now to prove the absence of replica symmetry breaking on the Nishimori line of

our model (23) ie for β = 1 As in Ref [26] we consider the magnetization distribution

P(1)βC(m) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσ δC[σ] eβ

sumi hiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiσi

(38)

6

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

and the overlap distribution

P(2)βC(q) equiv

int +infin

minusinfin

Nprod

i=1

dhi ph(hi)

sumσσprime δC[σ] δC[σprime] eβ

sumi hiσi+β

sumi hiσ

prime

i δ(q minus Nminus1sum

i σiσprimei)sum

σσprime δC[σ] δC[σprime] eβsum

i hiσi+βsum

i hiσprime

i

(39)

As before we keep the parity check matrix C fixed We shall prove that the two probability

distributions defined above are indeed identical on the Nishimori line β = 1 ie P(1)1C(x) =

P(2)1C(x) Since the probability distribution of the magnetization is expected to be a single delta

function2 [22] this implies the absence of replica symmetry breaking for β = 1We begin by using the decomposition (32) in Eq (38) This yields

P(1)βC(m) =

langsum

τ

esum

i hiτi

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σi)

sumσ δC[σ] eβ

sumi hiτiσi

rang

ρ

(310)

Then we notice that the above distribution is invariant under an ldquoallowedrdquo gauge transfor-mation τi rarr σprime

iτi σi rarr σprimeiσi As before ldquoallowedrdquo means that δC[σprime] = 1 We can therefore

average over these transformations obtaining

P(1)βC(m) =

langsum

τ σprime

δC[σprime]esum

i hiτiσprime

i

sumσ δC[σ] eβ

sumi hiτiσi δ(m minus Nminus1

sumi σiσ

primei)

2NRsum

σ δC[σ] eβsum

i hiτiσi

rang

ρ

(311)

We then insert 1 = (sum

σ δC[σ]esum

i hiτiσi)(sum

σprime δC[σprime]esum

i hiτiσprime

i) perform a second gauge trans-formation τi rarr σiτi σi rarr σiσi σprime

i rarr σiσprimei and sum over σ Finally we set β = 1 obtaining

P(1)1C(m) = P

(2)1C(m) as anticipated above

4 The random codeword limit

The limiting case k l rarr infin with lk = 1 minus R fixed plays an important role We shall call itthe random codeword limit for reasons which will be clear later It is a non-trivial limit sincethe redundancy of the error-correcting code is kept fixed From a theoretical point of view itallows a simple solution of the model without changing its qualitative features Our methodswill be similar to the ones used by Derrida to solve the REM [21] Finally we will show thatthe corrections for finite values of k and l are exponentially small in k Therefore this limit isinteresting also from a quantitative point of view

41 The limit k l rarr infin

Let us consider the probability for a given sequence of bits x = (x1 xN ) to be a codewordwith respect to the ensemble of parity check matrices C This coincides with the probabilityPσ for a given spin configuration σ to satisfy the constraints (24) In other words

Pσ equiv1

NC

sum

C

Mprod

j=1

δ[σωj +1] (41)

2Notice that our model (23) has no spin-reversal symmetry

7

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

where the sum over C runs over all the matrices of the (k l)-ensemble and NC is their numberClearly Pσ depend upon σ uniquely through the magnetization mσ equiv (1N)

sumi σi In

general it has the form

Pσ sim exp[NΣ

(kl)1 (mσ)

] (42)

The function Σ(kl)1 (m) is computed in Appendix A for general values of k and l and is not

particularly illuminating However in the limit k l rarr infin lk = 1 minus R fixed we have

Σ(kl)(m) rarr minus(1 minus R) log 2 (43)

for any minus1 lt m lt 1 In other words any spin configuration σ has the same probabilityPσ sim 2minus(1minusR)N of being a codeword In addition we must keep track of the completely orderedconfigurations σi = +1 for i = 1 N and σi = minus1 for i = 1 N The positive onesatisfies the all constraints for any k and l and for any matrix C (this configuration is quiteimportant for the thermodynamics of the model) The negative one satisfies the constraintsfor k even but it is irrelevant for the thermodynamics

Let us now turn to a slightly more complicated quantity We consider the joint probabilityPστ for two different spin configurations τ and σ to satisfy the same set of constraints (24)corresponding to some matrix C taken from the (k l)-ensemble In formulae

Pστ =1

NC

sum

C

Mprod

j=1

δ[σωj +1]δ[τωj +1] (44)

As before we can argue that Pστ depends upon σ and τ only through their magnetizationsmσ mτ and their overlap q equiv (1N)

sumi σiτi The form of Pστ in the thermodynamic limit is

Pστ sim exp[NΣ(kl)2 (mσmτ q)] (45)

The function Σ(kl)2 (m1m2 q) is computed in Appendix A Again we shall not report here

the result but we remark that in the k l rarr infin limit

Σ(kl)2 (m1m2 q) rarr minus2(1 minus R) log 2 (46)

for any minus1 lt m1m2 q lt 1 In other words the probability for two configurations σ and τto satisfy the same set of constraints is Pστ sim PσPτ sim 2minus2(1minusR)N the two configurations canbe regarded as independent ones

42 The random codeword model

The previous considerations allow us to replace (in the k l rarr infin limit) the original model(25) with the following random codeword model (RCM) The model has 2NR possible stateswhich we shall index with the letter α = 1 2NR To each of these states we associate a

random spin configuration σ(α) = (σ(α)1 σ

(α)N ) By random we mean that each spin σ

(α)i is

chosen independently from the others and that σ(α)i = +1 or minus1 with equal probability Let us

underline that in the random codeword model the σ(α)i are quenched variables the dynamical

one being the index α There is a second set of quenched variables the magnetic fields hi

8

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

minus2 minus1 0 1 2e

00

02

04

s(e)

Figure 2 The microcanonical entropy density of the RCM with binary field distribution cf Eq(26) Here we set R = 12 p = 0025 h0 = arctanh(1 minus 2p) Notice the continuous contributioncoming from the random configurations (solid line) and the isolated ordered configuration (filledcircle)

with i = 1 N As in the original model we take them to be random iid variables withdistribution ph(hi) The energy of the state α reads

E(α) = minusNsum

i=1

hiσ(α)i (47)

To the 2NR ldquodisorderedrdquo states described above we add the ordered state α = 0 and the

corresponding spin configuration σ(0) with σ(0)i = +1 for i = 1 N This corresponds to

the ldquoall zerosrdquo codeword 0 Its energy is obviously E(0) = minussum

i hiThe random codeword model can be solved through elementary methods Here we shall

solve it for the plusmnh0 distribution of fields see Eq (26) At the end of this Section we shallquote the result for a general distribution ph(hi) For sake of clarity we shall report thecalculation for this case which is slightly less straightforward in the Appendix B

We begin by taking into account the ldquorandomrdquo states α = 1 2NR Later we shallconsider the contribution coming from the ordered state α = 0 Let us consider a fixed

configuration of the magnetic fields hi Since the probability distribution of the σ(α)i is

flat P (σ(α)i ) = 2minusN2R we can apply a gauge transformation σ

(α)i rarr εiσ

(α)i with εi = plusmn1

without changing their statistical properties If we choose εi = sign(hi) the energy (47)

becomes E(α) = minush0sum

i σ(α)i We conclude that for what concerns the ldquorandomrdquo states the

plusmnh0 field distribution is equivalent to an uniform field hi = h0Now we would like to compute the typical number Ntyp(ǫ) of states having a given energy

density E(α)N = ǫ This is equal to the typical number of states having magnetization

9

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

m(α) = minusǫh0 This is a very simple problem Define the function

H(x) = minus1 + x

2log(1 + x) minus

1 minus x

2log(1 minus x) (48)

Then Ntyp(ǫ) sim expNR log 2 + NH(ǫh0) when |ǫ| lt ǫc and Ntyp(ǫ) = 0 otherwise Thecritical energy ǫc = h0ǫ(R) is the positive solution of R log 2 + H(ǫh0) = 0 The entropydensity of the system s(ǫ) = logNtyp(ǫ)N is depicted in Fig 2 Since sprime(minusǫc) gt 0 the(sub)system of the random codewords undergoes a freezing phase transition at the criticaltemperature βc = sprime(minusǫc) This phase transition is analogous to the one of the REM [21] itseparates an highndashtemperature paramagnetic phase from a lowndashtemperature frozen one

Let us now consider the ordered state α = 0 whose energy is given by E(0) = minussum

i hi Inthis case we can apply the central limit theorem For N rarr infin the energy density of the stateα = 0 is ǫ(0) = minus(1minus2p)h0 with probability one We have therefore the following picture of theenergy spectrum of the model a single ordered state at ǫ(0) = minus(1minus 2p)h0 plus a bell-shapedcontinuum between minusǫc(h0) and ǫc(h0) The ordered state is thermodynamically relevant aslong as it is separated by a gap from the continuum This happens if p lt pc(R) where pc(R)is the unique solution between 0 and 12 of the equation

R log 2 + H(1 minus 2p) = 0 (49)

Notice that Eq (49) coincide with the equation determining the capacity of the binarysymmetric channel [1] This means that in the k l rarr infin limit Gallager codes saturateShannon capacity

The free energy is easily determined from the entropy

f(β) = minǫ

ǫ minus

1

βs(ǫ)

(410)

The phase diagram includes three different phases a paramagnetic (P) and a spin-glass (SG)phases associated with the continuum part of the energy spectrum a ferromagnetic (F) phaseassociated with the ordered state The free energy of the paramagnetic phase is given by

fP (β) = minusR

βlog 2 minus

1

βlog cosh βh0 (411)

The paramagnetic-spin glass phase boundary is given by the zero-entropy condition partfP partβ =0 We obtain the curve βh0 = arctanh(1 minus 2pc(R)) equiv hlowast(R) At the transition the systemfreezes and the free energy in the spin-glass phase is

fSG(β) = fP (β = hlowast(R)h0) = minush0(1 minus 2pc(R)) (412)

The ferromagnetic free energy is nothing but the energy of the ferromagnetic state

fF (β) = minush0(1 minus 2p) (413)

The ferromagnetic-spin glass phase boundary has therefore the simple form p = pc(R)For sake of clarity let us consider the magnetic field distribution which describes a binary

symmetric channel ie let us fix h0 = h0(p) equiv arctanh(1 minus 2p) cf Eq (28) The resultingphase diagram is reported in Fig 3 The ferromagnetic-spin glass phase boundary is at

10

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

00 05 10 15 20 25 301β

1

10

w

PARA

FERRO

SG

Figure 3 The phase diagram for binary (left see Eq (26)) and gaussian (right see Eq (27))field distribution In both cases the field distribution was chosen to satisfy Eq (28)

p = pc(R) The paramagnetic-spin glass boundary is β arctanh(1minus 2p) = arctanh(1minus 2pc(R))Finally the ferromagnetic-paramagnetic phase boundary is given by

R log 2 + log cosh βh0(p) minus βh0(p) tanh h0(p) = 0 (414)

The triple point is at β = 1 p = pc(R) and lies on the Nishimori lineUntill now we treated the simple case of a two-peak distribution of the magnetic fields

ph(hi) = (1 minus p) δ(hi minus h0) + p δ(hi + h0) What does it happen for a generic ph(hi) InAppendix B it is shown that the same scenario applies with some slight modification The freeenergy in the paramagnetic phase becomes

fP (β) = minusR

βlog 2 minus

1

β〈log cosh βh〉h (415)

The system undergoes a freezing transition at a critical temperature βc determined from thecondition partfpartβ|βc

= 0 For β gt βc the system is in a glassy phase with free energyfSG(β) = fP (βc) Finally the ferromagnetic phase coincides with the ordered state α = 0and has free energy fF (β) = minus〈h〉h

To be specific we report in Fig 3 the phase diagram for the gaussian distribution

ph(h) =

radicw2

2πexp

minusw2

2

[h minus

1

w2

]2

(416)

which describes a gaussian channel with noise variance w The triple point is located at β = 1and w = wc(R) wc(R) being the solution of the equation below

R log 2 + 〈log cosh h〉h minus 〈h tanh h〉h = 0 (417)

It is easy to show that the solution R(w) of the above equation correspond to the capacity ofa gaussian channel with constrained binary inputs [2]

11

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

5 The replica calculation

As always [22] we compute the integer moments 〈Zn〉hC of the partition function by replicatingthe system n times To the leading exponential order we get

〈Zn〉hC sim

int prod

~σ

dλ(~σ)dλ(~σ) eminusNS[λλ] (51)

where

S[λ λ] = lsum

~σ

λ(~σ)λ(~σ) minusl

k

sum

~σ1~σk

λ(~σ1) middot middot λ(~σk)

nprod

a=1

δ[σa1 σa

k +1] minus

minus log

sum

~σ

λ(~σ)l〈eβhsum

a σa〉h

minus l +l

k (52)

and ~σ = (σ1 σn) is the replicated spin variable The calculations which lead to Eq (52)are completely analogous to the ones of Refs [1719] To be self-contained we shall sketch themin Appendix C The free energy f(β) is obtained by taking the saddle point of the integral(51) (let say λ = λlowast

n λ = λlowastn) and evaluating the n rarr 0 limit βf(β) = limnrarr0 partnS[λlowast

n λlowastn]

The saddle point equations are

λ(~σ) =sum

~σ1~σkminus1

λ(~σ1) middot middot λ(~σkminus1)

nprod

a=1

δ[σaσa1 σa

kminus1+1] (53)

λ(~σ) =λ(~σ)lminus1〈eβh

suma σa

〉hsum~σ λ(~σ)l〈eβh

suma σa

〉h (54)

The above equations are satisfied by the totally ordered solution λ0(~σ) = λ0(~σ) = δ~σ~σ0

where ~σ0 = (+1 +1) The corresponding free energy is fF (β) = minus〈h〉h Such a solution isis possible because of the infinite-strength ferromagnetic interactions in our model (23) Phys-ically it is related to the configuration σi = +1i=1N which satisfies all the constraints3

51 Stability of the ferromagnetic phase

In the ferromagnetic solution found above (as in the ferromagnetic phase found in Sec 4) thesystem is completely ordered (ie the magnetization is m = 1) This correspond to no-errorcommunication in the coding language Knowing the boundaries of the ferromagnetic phaseis therefore of great practical relevance Here we shall investigate the issue of local stabilityThe calculation is similar (although much simpler) to the one carried out for turbo codes inRef [12]

We start by computing the replicated action (52) for λ(~σ) λ(~σ) ldquonearrdquo the ferromagneticsaddle point namely λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) We first consider the casel gt 2

δS[λ0 λ0] = lsum

σ

δ(σ)δ(σ) minus1

2l(k minus 1)

sum

σ

δ(σ)2 +1

2l δ(σ0)

2 + O(δ3) (55)

3Notice that for k even there are 2n solutions of the type λ(~σ) = λ(~σ) = δ~σ~τ The ldquospuriousrdquo solutions with~τ 6= ~σ0 are related to the σi = minus1i=1N configuration Since we took 〈h〉h gt 0 these solutions do not havethermodynamical relevance

12

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

where δS[λ0 λ0] equiv S[λ0 + δ λ0 + δ] minus S[λ0 λ0] It is convenient to integrate over λ(σ) usingthe saddle point equation (53) which for λ(~σ) = λ0(~σ) + δ(~σ) λ(~σ) = λ0(~σ) + δ(~σ) givesδ(~σ) = δ(~σ)(k minus 1) + O(δ2) We finally get

δS[λ0] =1

2

sum

~σ

ζ~σδ(~σ)2 + O(δ2) (56)

where ζ~σ0= lk(k minus 1) and ζ~σ = l(k minus 1) for ~σ 6= ~σ0 We conclude that for l gt 2 the

ferromagnetic phase is always locally stable and its boundaries must correspond to first orderphase transitions

For l = 2 the situation is physically different Equation (56) is still valid with ζ~σ0=

2k(k minus 1) and

ζ~σ = 2

[1

k minus 1minus

〈eβhsum

a σa〉h

〈eβhn〉h

](57)

for ~σ 6= ~σ0 We have therefore n different eigenvalues ζnω with degeneracies

(nω

) where

ω equiv nminussum

a σa The first instability occurs for ω = 1 The corresponding critical line is givenby (k minus 1)〈eminusβch〉h = 1 This local stability condition is already known [29] in the codingcommunity although it has been obtained by completely different methods

Hereafter we shall focus on the case l ge 3

52 Replica symmetric approximation

The simplest approximation for treating the n rarr 0 limit consists in choosing λ(~σ) and λ(~σ)to be replica symmetric ie to depend upon ~σ uniquely through the symmetric combinationsum

a σa A commonly adopted parametrization [30] is the following

λ(~σ) =

intdxπ(x)

eβxsum

a σa

(2 cosh βx)n (58)

and the analogous one for λ(~σ) (with a different distribution π(y)) The replica symmetricorder parameters π(x) and π(y) have the physical meaning of probability distributions of cavityfields In particular

P (H) =

intdxπ(x)

intdy π(y) δ(H minus x minus y) (59)

is the probability distribution of the effective fields Hi equiv (1β)arctanh〈σi〉Using the ansatz (58) we easily obtain the replica symmetric free energy

βfP [π π] =l

klog 2 minus 〈log cosh βh〉h + l

intdxπ(x)

intdy π(y) log[1 + tβ(x)tβ(y)] minus

minusl

k

intdx1 π(x1)

intdxk π(xk) log[1 + tβ(x1) tβ(xk)] minus

minus

intdy1 π(y1)

intdyl π(yl)〈log Fl(h y1 ylβ)〉h (510)

13

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

where we defined tβ(x) equiv tanh βx and

Fl(y0 y1 ylβ) equivlprod

i=0

(1 + tβ(yi)) +lprod

i=0

(1 minus tβ(yi)) (511)

The field distributions π(x) and π(y) are determined by the saddle point equations

π(y) =

intdx1 π(x1)

intdxkminus1 π(xkminus1) δ

[y minus

1

βarctanh(tβ(x1) tβ(xkminus1))

]

(512)

π(x) =

intdy1 π(y1)

intdylminus1 π(ylminus1)〈δ(x minus h minus y1 minus minus ylminus1)〉h (513)

The above equations can be solved either numerically or in some particular limit In the nextSection we will see that the expansion around the random codeword limit provides ratheraccurate results

53 One step replica symmetry breaking

To go beyond replica symmetric approximation one has to divide the n replicas into nmsubgroups of m replicas (with 1 le m le n) The order parameters λ(~σ) and λ(~σ) depend upon~σ through the nm variables σα equiv

summαa=m(αminus1)+1 σa As discussed clearly in Refs [23 31] in

the n rarr 0 limit the order parameter becomes a functional over a probability space and thecalculations becomes rather cumbersome (see Refs [3132] for two viable approaches)

In our case there exists a very simple solution to the saddle point equations (53) (54)incorporating one step replica symmetry breaking

λ(~σ) =sum

sα

intdxπm(x)

eβxsumnm

α=1sα

(2 cosh βx)nm

nmprod

α=1

αmprod

a=(αminus1)m+1

δ[σa sα] (514)

and the analogous one for λ(~σ) (with a different distribution πm(y)) It is easy to see thatthe above ansatz satisfies the saddle point equations as soon as πm(x) πm(y) are solutionof the replica symmetric equations (512) (513) with the substitution h rarr mh The phasedescribed by the solution (514) is completely analogous to the spin-glass phase found inthe random codeword model The system is frozen in a large number of ldquooptimalrdquo con-figurations (with self-overlap qEA = 1) The overlap between two such configurations isq0 =

intdxπm(x)

intdy πm(y) t2β(x + y)

Such a simple scenario (and the simple solution (514)) is possible because the multi-spininteractions of the model (25) have infinite-strength The existence of other replica-symmetry-breaking solutions is an open issue see Sec 8 In the next Section we will show that our ansatzgives back the RCM solution see Sec 4 in the k l rarr infin limit

The free energy of the solution (514) is fSGm(β) = fP (βm) see Eq (510) and has to beoptimized over m with 0 le m le 1 This procedure yields the spin-glass free energy fSG(β) =fP (βc) and m = βcβ The critical temperature βc is given by the marginality conditionpartmfSGm(β)|m=1 = 0 which coincides with the zero-entropy condition partβfP (β)|β=βc = 0

Let us now draw some consequences of our solution (514) for the phase diagram of themodel Since both the spin-glass and the ferromagnetic free energies are temperature inde-pendent the ferromagnetic-spin glass phase boundary must stay parallel to the temperature

14

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

axis If for instance we consider the binary field distribution (26) with h0 = arctanh(1minus 2p)this boundary is simply given by p = pc(k l) Moreover we notice that the energy densityon the line β = 1 see Eq (37) is equal to the ferromagnetic free energy This impliesthat the entropy vanishes at the ferromagnetic-paramagnetic boundary for β = 1 Since theparamagnetic-spin glass boundary is determined by the zero entropy condition this point mustbe the triple point In synthesis the main characteristics of the phase diagram depicted inFig 3 remain valid for finite connectivities

6 Large k l expansion

Here we show that the replica solution exhibited in the previous Section goes to the randomcodeword model solution (cf Sec 4) when l k rarr infin at lk = 1 minus R fixed Moreover we wantto stress that this limit can be useful from a quantitative point of view In fact the correctionsfor finite k are exponentially small in k

Notice that the free energy in the spin glass phase fSG(β) is easily obtained from the para-magnetic free energy fP (β) In fact we have fSG(β) = fP (βc) where the freezing temperatureβc is given by the zero-entropy condition partβfP (β) = 0 Moreover the ferromagnetic free energyis fF (β) = minus〈h〉h and does not depend upon k and l It is then sufficient to solve Eqs (512)

(513) for large k l and evaluate Eq (510) on the solution The result is f(exp)P (β) (exp stands

for ldquoexpandedrdquo) and allow to reconstruct the whole phase diagram as explained aboveThe expansion is obtained by noticing that the product tβ(x1) middot middot tβ(xkminus1) which appears

on the right-hand side of Eq (512) is exponentially small in k as long as π(x) is supportedon finite values of x We then expand the the right-hand side of Eq (513) for small values ofy and plug the result in Eq (512)

The calculations are straightforward For sake of simplicity we show some consequencesfor the two-peak field distribution (26) We refer to Appendix D for the general results

In Fig 4 we report the modified phase diagram for k = 6 l = 3 as computed us-ing the expansion of Appendix D (cf Eq (D8)) for the paramagnetic free energy Weconsider the two-peak distribution (26) with h0 = arctanh(1 minus 2p) The paramagneticspin-

glass boundary is obtained by imposing the zero-entropy condition partβf(exp)P (β) = 0 We set

f(exp)SG (β) equiv f

(exp)P (βc) The ferromagnetic spin-glass and ferromagneticparamagnetic bound-

aries are obtained by imposing fF (β) = f(exp)SG (β) and fF (β) = f

(exp)P (β)

The triple point is at β = 1 p = pc(k l) As we stressed in Sec 3 the line β = 1 is of greatpractical importance since it correspond to a widespread decoding procedure (MAP decod-ing) The critical noise pc(k l) has the meaning of the threshold for no-error communicationunder MAP decoding Since the ferromagnetic-spin glass phase boundary stays parallel to thetemperature axis pc(k l) is also the threshold for any ldquofinite-temperaturerdquo decoding [27] forβ ge 1 We get

pc(k l) = p0c minus

1 minus R

4Hprime(1 minus 2p0c)

(1 minus 2p0c)

2k + O((1 minus 2p0c)

4k) (61)

where the function H(x) has been defined in Eq 48 In the k l rarr infin limit we recover thethreshold p0

c equiv pc(R) of the random codeword model given by the solution of Eq (49) Thedeviations from the optimal properties of the random-codeword model are exponentially smallfor large k

Equations (512) and (513) can be solved numerically by a ldquopopulation dynamicsrdquo algo-rithm One represents the distributions π(x) and π(y) by two populations xii=1L and

15

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

00 05 10 15 20 25 301β

00

01

02

03

04

05

p PARA

SG

FERRO

Figure 4 The phase diagram for the (6 3) code as computed from the large k l expansion (contin-uous lines) and the one of the RCM (dashed lines) The vertical dashed line is the Nishimori lineβ = 1

000 005 010 015 020 025p

000

005

010

015

020

025

Figure 5 The error probability per bit (filled circles and upper curves) and the entropy (emptytriangles and lower curves) for the (6 3) model with binary field distribution (26) We set β = 1 andh0 = arctanh(1minus 2p) The symbols are obtained by solving numerically the saddle point equations(512) (513) The dashed lines are the RCM results The continuous lines are the results of thelarge-connectivity expansion

16

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

yjj=1L and then iterates the equations (512) and (513) This method has been alreadyused for instance in Ref [31] In Fig 5 we consider once again the line β = 1 and comparethe results of large k l expansion with the numerical solution of Eqs (512) and (513) Weplot both the entropy and the average error probability per bit 〈Pe〉hC where

Pe =1

N

Nsum

i=1

1

2(1 minus sign〈σi〉) (62)

As conclusion let us consider the problem of calculating the critical noise pc(k l) Thiscan be obtained either by solving numerically Eqs (512) and (513) or from the expansion(61) The numerical solution yields pc(k l) = 00997(2) 01071(2) 01091(2) for respectively(k l) = (6 3) (8 4) (10 5) From the expansion (61) we get pexp

c (k l) asymp 0103965 01077830109195 for the same values of k and l

7 Finite size corrections and numerical results

In this Section we compare the analytical predictions with numerical results in order to confirmthe validity of the former and to investigate the nature of finite size corrections Needless tosay the last one is a point of utmost practical importance in coding theory Indeed it is knownthat the thermodynamic limit is approached exponentially fast in the ferromagnetic phase atzero temperature [2] We expect the same behavior to hold in the whole ferromagnetic phase

Here we focus on the paramagnetic-spin glass phase transition We compute the finite sizecorrections to the free energy of the RCM This calculation is compared with exact enumerationcalculations on small systems Then we switch to the complete model (25) and compare thethe numerical results with the outcome of the replica calculations cf Sec 5

71 The random codeword model

Let us consider for sake of clarity the binary distribution (26) with p gt pc(R) This cor-responds to focusing on the paramagnetic-spin glass phase transition Under this conditionthe ordered state α = 0 belongs to the continuous part of the spectrum and there is no en-ergy gap We shall therefore neglect this state Its contribution is exponentially small in thethermodynamic limit

With this assumption we obtain the following result for the free energy density

f(βN) = f0(β) +1

Nf1(βN) + O(1N2) (71)

The leading term has been already computed in Sec 4 The first correction f1(βN) vanishesin the paramagnetic phase and depends weakly upon N Explicit formulae are given in Ap-pendix E In particular f1(βN) sim (12βc) log N as N rarr infin The leading correction in theparamagnetic phase is exponentially small in N In order to compute it the ferromagneticstate cannot be neglected

It is very easy to compute numerically the finite-N free energy for the random codewordmodel with binary field distribution (26) as long as we neglect the ordered state All we needfor a given sample is the energy spectrum Let us call νk with k = 0 N the number ofstates α such that E(α) = minush0(N minus 2k) The probability distribution of the spectrum νk is

P (νk) =N

prodNk=0 νk

Nprod

k=0

pνkk (72)

17

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

00 05 10 15

1β

00

05

10

15

20

25

30

∆f(β

N)

(a)

00 05 10 15

1β

0

1

2

3

4

5

6

7

8

∆s(β

N)

(b)

Figure 6 Finite size correction to the free energy (a) and to the entropy (b) of the RCM Thecontinuous lines are the results of numerical computations for N = 40 80 120 160 200 (error barsare not visible on this scale) The dashed lines are the analytical results for the leading finite sizecorrection for N = 40 200 (a) and N = 200 (b)

wheresum

k νk = N equiv 2NR and

pk equiv1

2N

(Nk

) (73)

Once the νk have been generated with probability distribution (72) the partition functionis given by Z(β) =

sumk νk expβh0(N minus 2k)

We considered the RCM with rate R = 12 and binary field distribution (26) with h0 =arctanh(1 minus 2p) The phase diagram of this model is depicted in Fig 3 We fixed the flipprobability p = 02 to be greater than the threshold pc(12) asymp 0110025 and computed thetemperature dependence of the free energy by averaging over 105 realizations of the spectrumνk

In Fig 6 graph (a) we plot the quantity ∆f(βN) equiv [f(βN) minus f0(β)]N together withthe theoretical prediction f1(βN) for several values of N In Fig 6 graph (b) we considerthe entropy density s(βN) equiv β2partβf(βN) we plot the difference ∆s(βN) equiv [s(βN) minuss0(β)]N for the same values of N together with s1(βN) equiv β2partβf1(βN) for N = 200 (theN dependence of s1(βN) is rather weak)

Two remarks can be made by looking at Fig 6 First the O(1N2) terms in Eq (71)seems to be rather small If the temperature is not too close to the critical point the finitesize corrections are well described by f1(βN) Second the curves for ∆f(βN) see Fig 6graph (a) seem to cross at the critical point This is expected since ∆f(βN) sim (12βc) log Nfor β gt βc and ∆f(βN) sim eminusκN for β lt βc The crossing point βNN prime between the curves∆f(βN) and ∆f(βN prime) can be used to estimate βc From the data of Fig 6 we get

β4080 = 152(1) β80120 = 151(1) β120160 = 151(1) β160200 = 151(1) (74)

18

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

00 05 10 15 20

1β

minus09

minus08

minus07

minus06

minus05

minus04

f(β)

00 05 10 15 20

1β

00

01

02

03

04

s(β)

Figure 7 The free energy (left) and the entropy (right) of the (6 3) model computed by exact-enumeration (symbols) and the corresponding theoretical predictions (continuous lines) The vari-ous symbols refer to different system sizes N = 20 (triangles) 30 (circles) 40 (stars) and 50 (filleddiamonds)

which is in good agreement with the exact result βc asymp 150794

72 The (6 3) model

In this case we are forced to consider quite small systems since we do not know any simple formfor the probability distribution of the energy spectrum We must enumerate all the codewords(ie the spin configurations which satisfy the constraints in Eq (25)) this takes at leastO(2NR) operations Notice that finding the codewords is a simple task It suffices to solvethe linear system Cx = 0 (mod2) A standard method (we used gaussian elimination) takesO(N3) operations [33]

As in the previous Subsection we fixed considered the binary field distribution (26) withh0 = arctanh(1 minus 2p) and p = 02 In Fig 7 we plot the results for the free energy and theentropy densities for systems of size N = 20 30 40 (averaged over Nstat = 1000 samples) andN = 50 (with Nstat = 20 samples) The numerical results converge quite well to the theoreticalcalculation at high temperature Below the critical temperature the convergence is very slowas expected from the analogy with the RCM example

The sizes considered here are too small to reach any definite conclusion on the glassy phase

8 Discussion

The main result of this paper is the determination of the phase diagram of regular Gallagercodes see Eq (25) This is depicted in Fig 3 for the infinite connectivity limit The phasediagram for finite connectivities has been obtained by resorting to the replica method and looks

19

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

qualitatively similar The most important quantitative difference is the critical noise level forthe ferromagnetic-spin glass phase transition This quantity determines the performances ofthe corresponding code It can be determined either by solving the mean field equationsnumerically see Sec 5 or in a large connectivity expansion see Sec 6 The result of the lastcomputation is reported in Fig 4

The replica computation was made possible by the particularly simple one-step replicasymmetry breaking solution exhibited in Eq (514) We werenrsquot able to prove that the saddlepoint (514) is either unique or the dominant one There are however several independentindications which confirm this conclusion

bull The proposed solution is consistent with the absence of replica symmetry breaking onthe β = 1 line which has been proved in Sec 3

bull It has been shown [1934] that the critical noise level is the same both for zero-temperatureand for temperature one decoding This implies that the ferromagnetic-spin glass phaseboundary must pass through the points (p = pc(k l) 1β = 0) and (p = pc(k l) 1β =1) see Fig 4 (for sake of simplicity we referred to the case of a binary field distribution)This consistent with our phase diagram

bull Our numerical results although we restricted to fairly small systems do not contradictour conclusions

It can be interesting to notice that recently [35] a ldquofactorized ansatzrdquo has been proposed as anexact one-step replica symmetry breaking solution for some diluted spin models The solutionused in this paper is in some sense complementary to the one of Ref [35]

Acknowledgments

I am grateful to B Derrida for an illuminating discussion on the random codeword model andto N Sourlas for his constant support and encouragement I thank M Mezard and G Parisifor their interest in the subject of this paper This work was supported through a EuropeanCommunity Marie Curie Fellowship

A Codewords in the k l rarr infin limit

In this Appendix we compute the one-codeword and two-codeword probabilities see Eqs(41) and (44) for generic values of k and l Then we show that in the k l rarr infin limitdifferent codewords become statistically independent ie Pστ sim PσPτ

The one-codeword probability is to the leading exponential order

Pσ sim

int prod

σ

dλ(σ)dλ(σ) expNA1(λ λ c) (A1)

where

A1(λ λ c) = minuslsum

σ

λ(σ)λ(σ) +l

2k

(sum

σ

λ(σ)

)k

+

(sum

σ

λ(σ)σ

)k

+

+lsum

σ

c(σ) log λ(σ) + l minusl

k (A2)

20

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

and c(σ) = (1N)sum

i δσσi characterizes the configuration σ The above result can be provedby noticing that

sumσ Pσ exp(βh0

sumi σi) = 〈Z(h0)〉C where Z(h0) is the partition function for

the model (25) with uniform magnetic field hi = h0 The average 〈Z(h0)〉C is easily obtainedfrom Eqs (51) and (52) by setting n = 1 and ph(hi) = δ(hi minus h0)

The integral (A1) can be done through the saddle point method Saddle point equationsare more conveniently written by eliminating λ(σ) and using the variables λ+ equiv

sumσ λ(σ) and

λminus equivsum

σ λ(σ)σ We get

λk+ + λk

minus = 2 (A3)

λminusλkminus1+ + λ+λkminus1

minus = 2m (A4)

where m =sum

σ c(σ)σ = (1N)sum

i σi For large k these equations imply λ+ = 21k + O(mk)λminus = 21km + O(mk) as soon as minus1 lt m lt 1 Substituting in Eq (A2) we get the resultanticipated in Sec 4 see Eqs (42) (43)

Let us now consider the two-codeword probability cf Eq (44) Analogously to Eq (A1)we get

Pστ sim

int prod

στ

dλ(σ τ)dλ(σ τ) expNA2(λ λ c) (A5)

The corresponding ldquoactionrdquo is

A2(λ λ c) = minuslsum

στ

λ(σ τ)λ(σ τ) +l

k

sum

σ1σk

primesum

τ1τk

prime

λ(σ1 τ1) λ(σk τk) +

+lsum

στ

c(σ τ) log λ(σ τ) + l minusl

k (A6)

where c(σ τ) = (1N)sum

i δσiσδτiτ and the sumssumprime are restricted to σ1 middot middot middot σk = +1 and

τ1 middot middot middot τk = +1 As before we notice thatsum

στ Pστ exp(βh1sum

i σi+βh2sum

i τi) = 〈Z(h1)Z(h2)〉Ccan be obtained through a standard replica calculation see Sec 5 and App C with n = 2replicas

We now define the variables λ0 equivsum

στ λ(σ τ) λσ equivsum

στ λ(σ τ)σ λτ equivsum

στ λ(σ τ)τ andλστ equiv

sumστ λ(σ τ)στ The saddle point equations can be written in terms of these variables

as follows

λk0 + λk

σ + λkτ + λk

στ = 4 (A7)

λσλkminus10 + λ0λ

kminus1σ + λστλkminus1

τ + λτλkminus1στ = 4mσ (A8)

λτλkminus10 + λστλkminus1

σ + λ0λkminus1τ + λσλkminus1

στ = 4mτ (A9)

λστλkminus10 + λτλ

kminus1σ + λσλkminus1

τ + λ0λkminus1στ = 4q (A10)

where mσ =sum

στ c(σ τ)σ = (1N)sum

i σi mτ =sum

στ c(σ τ)τ = (1N)sum

i τi and q =sum

στ c(σ τ)στ = (1N)sum

i σiτi From Eqs (A7)-(A10) we get for k rarr infin λ0 ≃ 41k

λσ ≃ 4(1minusk)kmσ λτ ≃ 4(1minusk)kmτ λστ ≃ 4(1minusk)kq as soon as minus1 lt mσmτ q lt 1 Thecorrections to this asymptotic behavior are of order O(mk

σmkτ q

k) Substituting this solutionin Eqs (A5) (A6) we get the results (45) (46)

21

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

minus10 minus05 00 05 10m1

minus10

minus05

00

05

10

m2

Ω

βc

Figure 8 The RCM for ph(hi) = (25) δ(hi minus 12) + (35) δ(hi minus 1) The continuous line encirclesthe region Ω (see text) The dashed line is the curve m1 = tanh β2 m2 = tanh β which intersectthe boundary of Ω for β = βc

B The random codeword model for a generic field

distribution

In this Appendix we solve4 the RCM for a generic field distribution ph(hi) The strategy is tostart from a discrete distribution

ph(hi) =

Msum

q=1

pq δ(hi minus h(q)) (B1)

and then approximate a generic ph(hi) by letting M rarr infinLet us consider the distribution (B1) In the typical sample there will be N1 asymp Np1

sites with field hi = h(1) (which we can suppose without loss of generality to be the sitesi = 1 N1) N2 asymp Np2 sites with field hi = h(2) (let us say for i = N1 + 1 N1 + N2)and so on For a given spin configuration σ we define the partial magnetization mq(σ) as themagnetization of the sites whose magnetic field is h(q) With the labeling of the sites chosenabove we get

mq(σ) equiv1

Nq

Nqsum

i=Nqminus1+1

σi (B2)

where Nq = N1 + + Nq We call mq(σ) the magnetization profile of the configuration σWe now consider the 2NR states α = 1 2NR To each of them it is associated a

random codeword σ(α) where the σ(α)i are quenched variables drawn with flat probability

distribution We ask ourselves what is the typical number Ntyp(mq) of states α having a

4I am deeply indebted with B Derrida who explained to me how to treat this general case

22

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

given magnetization profile mq(σ(α)) = mq The answer is quite easy Define the function

G(mq) as follows

G(mq) = R log 2 +Msum

q=1

pqH(mq) (B3)

where H(x) is given in Eq (48) The typical number Ntyp(mq) is obtained from G(mq)through the usual construction Ntyp(mq) sim exp[NG(mq)] if G(mq) gt 0 and Ntyp(mq) =0 otherwise The convex region Ω equiv mq|G(mq) gt 0 is depicted in Fig 8 for the caseM = 2

The energy of a state α can be written in terms of its magnetization profile E(α) =minusN

sumq pqh

(q)mq(σ(α)) The free energy density can therefore computed from Ntyp(mq) as

follows

f(β) = minmq

minus1

βG(mq) minus

Msum

q=1

pqhqmq

(B4)

where G(mq) equiv (1N) log Ntyp(mq) (ie G(mq) = G(mq) inside Ω and G(mq) =minusinfin outside)

If the expression (B3) is used in Eq (B4) one gets the saddle point condition mq =tanh βhq This describes a curve in the mq space which start at mq = 0 for β = 0 and endsat mq = sign hq for β = infin The corresponding free energy reads

fP (β) = minusR

βlog 2 minus

1

β

Msum

q=1

pq log cosh βhq (B5)

At some critical temperature β = βc the curve mq = tanh βhq crosses the boundary of Ω Thesaddle point mq = tanh βhq is no longer valid for β gt βc The critical temperature can becomputed from the zero entropy condition partβfP |β=βc = 0 For β gt βc the entropy vanishesand the free energy is frozen to its value at the critical point fSG(β) = fP (βc) As in Sec 4we must include in our analysis the ordered state α = 0 whose free energy is fF (β) = minus〈h〉h

The solution for a continuous field distribution ph(hi) follows from the above results bytaking the M rarr infin limit in Eq (B5) This yields Eq (415) Alternatively we couldhave started with a continuous magnetization profile m(h) from the very beginning of thisAppendix

C The derivation of Eq (52)

We start by writing down the partition function of the model (25)

Z(β) =sum

σ

Mprod

j=1

δ[σωj +1] esum

i hiσi (C1)

We rewrite the constraint term (ie the product of Kronecker delta functions) by introducingthe quenched variables Dω = 0 1 where ω = (iω1 ωk ) runs over the k-plets of site indices

23

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

The variables Dω are defined by setting Dω = 1 if ω = ωj for some j = 1 M and Dω = 0otherwise With this definition we can write the replicated partition function as follows

〈Zn〉 =1

N

sum

D

sum

~σ

Nprod

i=1

langeβh

suma σa

i

rang

h

prod

ω

1 minus Dω + Dωδn[~σω] (C2)

where ~σω equiv (prodk

r=1 σ1iωr

prodk

r=1 σniωr

) δn[~σ] equivprodn

a=1 δ[σa+1] and N is a normalization con-stant (to be computed later)

According to our choice of the ensemble of check matrices we must imposesum

ωnii Dω = lfor any i = 1 N This can be done by using the identity

δ

[sum

ωnii

Dω l

]

=

∮dzi

2πi

1

zl+1i

zsum

ωnii Dω

i (C3)

where the integration path encircles the origin in the complex zi plane We get

〈Zn〉 =1

N prime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

h

prod

ω

1sum

Dω=0

w(Dω)1 minus Dω + Dωδn[~σω] zDωω

(C4)

where zω equivprod

iisinω zi The weights w(Dω) have been introduced for later convenience and cor-respond to a rescaling of the zi Their contribution can be readsorbed by the normalizationconstant N prime We set w(1) = l(k minus 1)Nkminus1 and w(0) = 1 minus w(1) Now we can sum over theDω obtaining

〈Zn〉 =1

N primeprime

sum

~σ

Nprod

i=1

∮dzi

2πi

1

zl+1i

langeβh

suma σa

i

rang

hmiddot (C5)

middot exp

Nl

k

sum

~σ1~σk

cz(~σ1) cz(~σk)

nprod

a=1

δ[σa1 σa

k +1]

where cz(~σ) equiv (1N)sum

i ziδ~σ~σi Finally we introduce the order parameter λ(~σ) and its complex

conjugate λ(~σ) by using the following identity

expNF [c] =

int prod

~σ

Nl

πdλ(~σ)dλ(~σ) exp

minusNl

sum

~σ

λ(~σ)λ(~σ)+ (C6)

+NF [λ] + Nlsum

~σ

λ(~σ)cz(~σ)

The use of the above identity allows to integrate over the zi obtaining Eqs (51) and (52)The overall normalization constant can be fixed by requiring 〈Zn〉 sim 2Nn(1minuslk) for hi = 0

D Large k l expansion general formulae

Let us define tp equiv 〈tanh βh〉h We assume formally tp = O(tp) where t is ldquosmallrdquo and expandin tk to the order t3k All the observables can be expressed in terms of the order parameters

24

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

π(x) and π(y) The solutions of Eqs (512) (513) admit an expansion of the form

π(x) = ph(x) +

infinsum

m=1

πmβminusmp(m)h (x) π(y) = δ(y) +

infinsum

n=1

πnβminusnδ(n)(y) (D1)

where p(m)h (x) equiv partm

x ph(x) and δ(n)(y) = partny δ(y) Moreover one gets πm πm = O(tmk) The

results for the first few coefficients are listed below

π1 = minus(l minus 1)tkminus1

1minus (k minus 1)(l minus 1)2(1 minus t2)t

2kminus3

1minus (D2)

minus1

3(l minus 1)tkminus1

3minus

1

2(k minus 1)(k minus 2)(l minus 1)3(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)3(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)2(l minus 2)(t1 minus t3)t

3kminus4

1+ O(t4k)

π2 =1

2(l minus 1)tkminus1

2+

1

2(l minus 1)(l minus 2)t2kminus2

1+ (D3)

+(k minus 1)(l minus 1)2(t1 minus t3)tkminus2

2tkminus1

1+ (k minus 1)(l minus 1)2(l minus 2)(1 minus t2)t

3kminus4

1+ O(t4k)

π3 = minus1

6(l minus 1)tkminus1

3minus

1

2(l minus 1)(l minus 2)tkminus1

2tkminus1

1minus

1

6(l minus 1)(l minus 2)(l minus 3)t3kminus3

1+ O(t4k) (D4)

π1 = minustkminus1

1minus (k minus 1)(l minus 1)(1 minus t2)t

2kminus3

1minus (D5)

minus1

2(k minus 1)(k minus 2)(l minus 1)2(1 minus t2)

2t3kminus5

1minus (k minus 1)2(l minus 1)2(1 minus t2)

2t3kminus5

1+

+(k minus 1)(l minus 1)(t1 minus t3)tkminus1

2tkminus2

1+ (k minus 1)(l minus 1)(l minus 2)(t minus t3)t

3kminus4

1minus

1

3tkminus1

3+ O(t4k)

π2 =1

2tkminus1

2+ (k minus 1)(l minus 1)(t1 minus t3)t

kminus2

2tkminus1

1+ O(t4k) (D6)

π3 = minus1

6tkminus1

3+ O(t4k) (D7)

The result for the paramagnetic free energy is

βfP (β) = minusR log 2 minus 〈log coshβh〉h minusl

ktk1 minus

1

2l(l minus 1)(1 minus t2)t

2kminus2

1+

1

2

l

ktk2 minus

minus1

2(k minus 1)l(l minus 1)2(1 minus t2)

2t3kminus4

1+

1

3l(l minus 1)(l minus 2)(t1 minus t3)t

3kminus3

1+ (D8)

+l(l minus 1)(t1 minus t3)tkminus1

1tkminus1

2minus

1

3

l

ktk3 + O(t4k)

E Finite size corrections for the random codeword

model

Let us consider the binary field distribution (26) with h0 = 1 The results for a genericvalue of h0 are obtained after a trivial rescaling of energies and temperatures f(β h0N) =h0f(βh0 1N)

As explained in Sec 7 the finite size corrections at the paramagnetic-spin glass phasetransition can be studied by neglecting the ordered state This introduces exponentially smallerrors The calculation of the free energy can be done along the lines of Ref [21] AppendixB which starts from the identity

〈log Z〉 =

int infin

0

dt

t

(eminust minus eminustZ

) (E1)

25

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

We limit ourselves to quoting the outcome of the calculation For β lt βc we get f(βN) =fP (β) + O(eminusκN )5 For β gt βc we get Eq (71) with

f0(β) = minusǫ(R) f1(βN) =

int infin

0dφ ρ(φ) eminusφ + γβ (E2)

γ asymp 0577216 being the Euler constant The function ρ(φ) is defined as the (unique) solutionof

βcρ + log Ψ(minusNǫ + ρ) = log(φ) +1

2log[π2N(1 minus ǫ2)

] (E3)

where minusǫ(R) is the ground state energy density in the thermodynamic limit see Sec 4 Thefunction Ψ(x) is defined as follows

Ψ(x) =

+infinsum

q=minusinfin

eminusβc(2q+x)[1 minus exp

(minuseβ(2q+x)

)] (E4)

Notice that Ψ(x + 2) = Ψ(x) The log Ψ term in Eq (E3) gives therefore an oscillatingN dependence to f1(βN) Moreover since Ψ(minusNǫ + ρ) remains finite for any N and ρf1(βN) sim (12βc) log N as N rarr infin Finally we remark that the sum in Eq (E4) divergesas β darr βc This gives the singularity of the free energy corrections at the critical pointf1(βN) sim (1βc) log(1 minus βcβ)

References

[1] T M Cover and J A Thomas Elements of Information Theory (Wiley New York1991)

[2] A J Viterbi and J K Omura Principles of Digital Communication and Coding(McGraw-Hill New York 1979)

[3] C E Shannon Bell Syst Tech J 27 379-423 623-656 (1948)

[4] S-Y Chung G D Forney Jr T J Richardson and R Urbanke On the design of

low-density parity-check codes within 00045 dB from the Shannon limit IEEE CommLetters to appear

[5] C Berrou A Glavieux and P Thitimajshima Proc 1993 Int Conf Comm 1064-1070

[6] D J C MacKay IEEE Trans Inform Theory 45 399-431 (1999)

[7] R G Gallager Low Density Parity Check Codes Research Monograph Series Vol 21(MIT Cambridge MA 1963)

[8] N Sourlas Nature 339 693-694 (1989)

[9] N Sourlas Statistical Mechanics of Neural Networks Lecture Notes in Physics 368 editedby L Garrido (Springer Verlag 1990)

[10] N Sourlas From Statistical Physics to Statistical Inference and Back edited by P Grass-berger and J-P Nadal (Kluwer Academic 1994) p 195

5Obviously the ordered state cannot be longer neglected in computing κ

26

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27

[11] A Montanari and N Sourlas Eur Phys J B 18 107-119 (2000)

[12] A Montanari Eur Phys J B 18 121-136 (2000)

[13] I Kanter and D Saad Phys Rev Lett 83 2660-2663 (1999)

[14] I Kanter and D Saad Phys Rev E 61 2137-2140 (1999)

[15] Y Kabashima T Murayama and D Saad Phys Rev Lett 84 1355-1358 (2000)

[16] I Kanter and D Saad Jour Phys A 33 1675-1681 (2000)

[17] R Vicente D Saad and Y Kabashima Phys Rev E 60 5352-5366 (1999)

[18] R Vicente D Saad and Y Kabashima Europhys Lett 51 698-704 (2000)

[19] Y Kabashima N Sazuka K Nakamura and D Saad Tighter Decoding Reliability Bound

for Gallagerrsquos Error-Correcting Code cond-mat0010173

[20] H Nishimori J Phys C 13 4071-4076 (1980)

[21] B Derrida Phys Rev B 24 2613-2626 (1981)

[22] M Mezard G Parisi and M A Virasoro Spin Glass theory and Beyond (World ScientificSingapore 1987)

[23] R Monasson J Phys A 31 (1998) 513-529

[24] R M Tanner IEEE Trans Infor Theory 27 533-547 (1981)

[25] H Nishimori Prog Theor Phys 66 1169-1181 (1981)

[26] H Nishimori and D Sherrington Absence of Replica Symmetry Breaking in a Region of

the Phase Diagram of the Ising Spin Glass cond-mat0008139

[27] P Rujan PhysRevLett 70 2968-2971 (1993)

[28] N Sourlas EurophysLett 25 159-164 (1994)

[29] T Richardson and R Urbanke The Capacity of Low-Density Parity Check Codes under

Message-Passing Decoding IEEE Trans Inform Theory to appear

[30] K Y M Wong and D Sherrington J Phys A 21 L459-L466 (1988)

[31] M Mezard and G Parisi The Bethe lattice spin glass revisited cond-mat0009418 toappear in Eur Phys J B

[32] G Biroli R Monasson M Weigt Eur Phys J B 14 551-568 (2000)

[33] W H Press B P Flannery S A Teukolsky and W T Vetterling Numerical Recipes(Cambridge University Press Cambridge 1986)

[34] D J C MacKay On thresholds of codes available athttpwolraphycamacukmackayabstractstheorems

[35] S Franz M Leone F Ricci-Tersenghi and R Zecchina Exact solutions for diluted spin

glasses and optimization problems cond-mar0103328

27