+ All Categories
Home > Documents > ch1 examples

ch1 examples

Date post: 03-Jun-2018
Category:
Upload: zamanmunoo
View: 253 times
Download: 0 times
Share this document with a friend

of 15

Transcript
  • 8/12/2019 ch1 examples

    1/15

    Chapter1 Discrete ources nd Entropysource oesnot mita characterttimeindex/,wesaythatthesourceproducesanullcharacterat time r. A null character, herefore, s a kind of virtual symbol denotingthe absenceof a real symbol at /. This convention allows many discrete-timeasyn-chronoussourceso be approximatedassynchronous ources.The symbolsemitted by a source n a physicalcommunicationsystemmust berepresentedsomehow. n digital communicationsystems,his is typically done usingbinary representation. or example,a sourcewith M:4 might have ts symbols epre-sentedby apair of binary digits. n this case, omight be representedas00,a, as01,andso on. t is common o refer to symbols epresentedn this fashionassourcedata.Information theory makesan important distinction between data and nforma-tion. T}aewo conceptsare not the sameand,generally,data s not equivalent o infor-mation.To see his, consideran information source hat hasan alphabetwith only onesymbol.The sourcecanemit only this symbol and nothing else.The representationofthis symbol is data but, clearly, his data is completely uninformative.There is nosurprise whatsoever ied to this data.Since nformation carries he connotation ofuncertainty n what will come next, he information contentof this source s zero.The information content of a source s an important attribute and can be mea-sured. In his original paper (which founded the science of information theory),Shannongavea precisemathematicaldefinition of the average mount of informationconveyed er sourcesymbol.This measure s called the entropyof the source and isdefinedas

    H(A) : ) p*log2Q,lp).m:9Equation1.2.5 ellsus that the informationcontentof a sourceindividual symbol probabilities.Note that for the caseof oursource,he probabilityof its symbol s unity and I: 0.

    t.2.5is determinedby theuninformativeM:7

    In the definition givenabove or entropy,we take the logarithm to the base2 ofthe reciprocalof the probabilities.It s alsopossibleo defineentropyusingany otherbase logarithm. For entropy defined as in Equation 7.2.5, he unit of measureofentropy s called a bit. If we were to replace he logarithm to the base2 in Equation1.2.5 y the natural ogarithm log o the basee), he entropy ssaid o be measurednnatural units or nats. The most commonpractice s to measureentropy n bits.

    cwilfi1ffillffltiilttilmt]iliIttllitlWhat is the entropy of a 4-arysourcehaving symbolprobabilities

    P, : {0.5,0.3,0.15,0.05}?Solution:

    H(A): .5Iog,2(2) .3tog2(1013)f .15log2(100/15).051o92(20)1..6477bits.(Recall hat logr(x) :In(x)lln(2)

  • 8/12/2019 ch1 examples

    2/15

    Section|.2 Discretenformation ourcesndEntropy 7The first term on the right-handsideof Equation L.2.7 issimplyH(A).The sec_nd term is caileda conditio;ar entropy.It is the uo""roiniii"noopy) of B givenAndiswrittenH(BlA).Thus,wemay*rit" . 'rurr \v u

    H(C) : H(A,B) : H(A) + H(B]A). 1.2.8(Note thesimilarityof Equation1.2.gwith theJogarithm f p,,',,:OO,p).Itis a simpleatter to prove that H(A, B): H(B) + 77(Ap) ai well. ' "r tNow, f B is statistically ndependentLf a,H(BtA), In,4oo,tosr(rlp1,): )o,\ nltos(t1p):H(B).' lIn this case,he total entropy s simply the sumof the entropiesof the two sources. nhe other hand, f B is depende"t on A, i- "uo be shown inat n1q,

  • 8/12/2019 ch1 examples

    3/15

    I Chapter1 Discrete ources nd Entropy1.2.3 Entropy f SymbolBlocks nd he ChainRuleAnother usefulapplication or entropy s in finding the information contentof a blockf symbols. uppose? s,oTce froJyl a seqye.n:ef n symbolsso, 1, .. , s,_, withachs,drawn rom alphabet . Theentropyof thisblock s denotedasH(A* A.,, .. ,,_.t) where the arguments-4, indicate that trr" ,y-uoi at inoex r was drawn iomlphabetA' ThisFl ii a oint "ttiffiin"e its vatue

    'oepe"ndson the oint probablity ofhesymbolsn theblocl. f we .s'r'r1,*r*;;il5i;;, :,r, .. ,.rn_1)sacompoundymbol,imilarryothepreceanfJis"ussron,wecanuse uithe entropy as r -----o$rvveusrv'' ws rv'lil useour prevlous result to expressH (A0,A1, . ,An_t ) : H (Ao) + H (A1,A2, . ,An_t lAo).il:tJffi',',11Jfiil#n,t1fl:'",X113,::1""','ointntropvonditionednAo.t

    H (Ay A2,. ,An_lA;) : H (AlAs) + H (A2,. ,An_lA0,A1).,Tfi?Jt"lt#is argumentnductivelvndplugginghe esuttsacknto he irstequa-

    H(A0,A1,.. . ,An_,) H(Ao) + H(AlA0) + H(A21A0,A)+ . . . H ( A , - 1 1 A , , . . . , A , _ 2 ) . 1 . 2 . 1 0Thisresult s known y-tlelhain rulefor entopy.SinceF(BrA) = H(B),one immldiate consequencef Equation1.2.10s

    n _ IH(Ao; ' . ,An-r)=2u(e) ,, =0

    Srtemrteanr

    wherePu*-'-

    For anpossibSolutionmust ieeach teample 1withp^:

    Therefo

    Houwdeuotbe s6qneslr quncer

    urRigorSolur

    7.2 .1 \with equality f and only if all of the symbols n the sequence re statistically ndepen-ent' (A sourcewith this p.op"rtt;:;]i; a memoryressource;itdoesn,t,,remem-er" from onetime to ttt^" t"i*ti,;yd;i, it haspreviousryemitted.)whle we haveeveloped .2.r0 and1.2.11 ssumtd;;;".e source.4 emitreda, of the symbors,heseequationsarealso rue ir rrr" ,yii.i."are emittedby differentsources.when alrof the symbotrur"

    ",nitiJ;;n, thesame "r"r'nutio" source, nd f therobabilitiesof the sourcedo not_cha" " ou"r ai.e, then att oi trr" .Fl ermsunder theummationn I.2.1I areequal o theeritropyof sourceA, and.weaveH(Ao;", An-1)< n. H(A), L2.I2l|tff1Tt*:ffi1Jr:y.'jr'.T svmbolsre tatisticallvndependent.qualitynr.2.r2

    -r_ una

    Suppose memoryressourcewith,4 : {0,1}havingequalsymbolprobabilitiesemitsa sequence:fi:'Jfrfi }",i::llt"::1:11 +oi'1'.^'yr*;seventhvmlors ransmittedhichs he;t-dffi;" "".,l#jr',?:',:ilfi1flij|i:L'|:1..,",1eexcrusive-orriheymborsmitteoy

  • 8/12/2019 ch1 examples

    4/15

    5Solution: Letb: > es, wheret= 0entropyof the sequences H(Ao,Ar,lent to

    Section .2 Discretenformation ourcesndEntropy 9) O denotesa summationmodulo 2 ands,e,4. The'.. ,As, b). By the chain ule for entropieg his. s equiva_

    I:y::::yJ*l:)^-r, ,)if the svmbolsare statisticallvndependent. staristical ndepen-oencemeans hat Aitells.usnothing aboutA,it i + j). Since he first six symbolsaredrawn romtne samesource,l, the first six_terms n the right-handsideof the foregoingexpression re alle.qual, nd sincePo:Pr:0'5, these ermssumlo 6. n the ast erm onihe right-hand ide, hesix symbolsAo;",As completelydetermine b so knowledgeof thesesix symbols eavesno

    H(Ao, 1,..., s,b) H (Ai + H (ArtAi + H (A2t 0, 1) + ...H (btAo;.., s) .

    uncertaintyn b andH(blAo, .., Ar) 0.Therefor ,H(Ao,Al, ..., Ar, b1 6.

    H ( b t A o , . . . , A ) : ,e, log2(1'f P otoo,.,a,),

    H ( b l A o , " . , A ) : - 0 . l o g 2 ( 0 )* 1 . l o g 2 ( 1 ) : 0 * 0 : 0 .

    H(1.7' iri log2(l '41) log,(rAl).

    Rigorously how hatH(blAo, .. ,Ar): 0 in exampl 1..2.4.Solution: From the definitionof entropy,we canwrite

    L) Pu,o,,.b = 0

    wherep6140;..,dr.is.h: conditional robabilityof b givenAo;,.,As.Now, or anygivenAoi, ,,As,b is either0 or 1, depending n h,... 0,4), so we have either-p61a,;..,;::0;;Pbt.t;",Ar: L,dependingon the valueof D n the summation.Therefore,

    For an information sourcehaving alphabet 4 with l,4lsymbols,what is the rangeof entropiespossible?

    solatton: The definition of entropy is givenby Equation 1.2.5.Since he symbolprobabilitiesmust ie in the range0 = 4-= r. for eveiy symboriogr(1,1p^)> 0 for everysymuot.Therefore,each term in the summation n Equation i.Z.S i, non_negative. herefore,H(A) -_0. In Ex_ampleL.2.2,wesaw hat the entropy s maximizedwhen ev'erysymbol n a is equally.probablewithp^ : I I lAl.Consequently,

    Therefore.0- H(A)

  • 8/12/2019 ch1 examples

    5/15

    10 Chapter Discrete ources nd Entropy1.3 SOURCEODING'1.3.1 MappingFunctions nd Eff ic iency

    Example 1'2'6showed hat arbitrary informationsources an have a considerablerangeof possibleentropies.The entropyof a source s the averagenformation carriedper symbol'Sinceeachsymbolwill either be transmitt"J-1in rr" caseof a communica-tion system) r stored in thecase f a storage ystem), ndsinceeachuseof the chan-nel (or eachunit of storage) assomearsoCiat"dort, it is clearlydesirableo obtainthe most information possibleper symbol (on the uu"*g"). If we havean inefficientsource' 'e',H(A) < toglia)' our system anbe mademoie costefTectivehrough heuseof a sourceencoder.A sourceencoder anbe ookedat asa dataprocessing lementwhich akesaninput sequencef so, 1,'''symbols ,e ,4 from the nformationsource ndproduces noutputsequence 6,si,"' usingsymbols i drawn o^ i ria, afuhabetr.'rrr"s" ,y*_bolsare calledcodewordl ]\e objectiveof the encoder s to process he input in sucha way that the average nformation transmitted (or stored) per channelur" "1o."rypproaches (l) (If ,4 is.acompound ymbol, u"h u, in-"ru*pt" r.2.3,H(A)wouldactuallybe a oint entropy.)In its simplestorm, he encoder anbeviewedasa mappingof thesource lpha-betA to a codealphabetB. Mathematically,his srepres"ni"au,C:A--+B,since the encodedsequencemust eventuallybe decoded,he function c must benvertible'Thismeanslere mustexistanothei unctionc-1 such hat f C(a),->b thenc-'(b) e a. This.ispossibre "ry ii zloy ? b i, unique, .e., for everybe.B, there sexactly onea e A suchthat C(a) rt 6, uo for every'ae A,'thereis exactly oneb e B:::lrtlffr'(b) - l.(we areassuminghatevery e Ahasa on-zerorobabilityfThe sourceencoderactsasa data-process ingnit. n thissection,we will discusssomeof the detailshow dataprocessing ffectsnlormation. n particular,we mustbeconcernedwith the.relationshipsetween nformationcarriedby codewordsb andinformation carriedby sourcesymbolsa. Beforediving into itresedetails,however, t isJ"""fJlt" to illustrate the function of a source"rr.6d", by meansof the following

    [d0*h;n'f

    [fufrri- l--wd.fu ffitur

    sqp,n*Oddlpdcnmcrhe

    kt-{-4 6 hteriHc

    SHlAxber o

    The emitted

    LetA be a 4-ary ourcewithsymbolprobabilities sgiven n Example .2.l.Letc be anencoderhichmaps he symbolsn,4 into stringsof binarydigits,as ollows:po:0.5 C(a) ,> 0pr:0.3 C(ar) > ept :0.L5 C(ar) > l lgpr:0.05 C(ar)r-->l1

  • 8/12/2019 ch1 examples

    6/15

    FromExample1.2.r,H(A):1,'6477 bits.Theefficiency f rhisencode is H (A)lL : 0.96924.rtthe sourceencoderwere not used,we would need two binary digits to representeachsourcesymbol.The efficiencyof the uncodedsourcewould be n1A11z":0.g2385.put another way,almost 18% of the transmitted binary digits for the uncoded source would be redundant,

    Sect ion.3 Source oding 11Let L- be the number of binary digits n codeword b^. rf thecodewordsare transmittedonebinarydigit at a time, he averagenumberof transmifte; binarydigits p", loo" word is givenby

    L : 9^o*t*: .s(1) .3(2) .1s(3) .05(3) 1..70.

    whereasonly about3o/o f the ransmitiedb-inary igits io- a ur",Juiaurr,.

    \aaal Pr(a,,a) b. (oro) Pr(ai,a)

    ^ More sophisticateddata processingby the encoder can also be carried out.Suppose hat symbolsemitted by sourc A *"r" first groupeJ i.rto ordered pairs(a,,a) and he codewordsprodu."a uy the encoder*"r" bur"d on thesepairs.The setof all possiblepairs (aua) is called the Cartesian roduct of set A with itself and isdenotedby AX A. Theencoding rocesshenbecomes functionof two variables ndcanbe denotedby a mappin CA X A_+ B or by a functionC1a)a,5,, U.

    L:t A bea 4-atymemorylessourcg ith thesymbol robabilitiesivenn Example .3.1. ince1 is a memorylessource,heprobability r anygiven airof syibors sgiventy pr(o,,-if:i:{i , Pr(a,)'Letheencodermappairsbt symuotsntothecodewords hownn the ollovling

    As; Aga6 a1ao, a24g743ay ajay a1ay 42ay a3

    .25

    . r l.075.025.1 5.09.045.015

    0010011001110010 10100110111100

    az,ao421 41az,aza2;a3431 dg431 A1ay az431 43

    .075 1101.045 0111.022s 111110.0075 1111110.025 11101.015 tr1.101,.0075 11111110.0025 71.1111,1,r

    Sincehesymbolsro-T1 are ndependent,eknow romEquatio L.2.r2thatheentropy1(A\9:zH(A):3.2954.Sinc" 4ry Pr(b^1:pr(ai,aj) "r J;h;t;bol, theaverageum_berof bitsper ransmittedodeword; ' "-

    T: 2P't 'm: 0The efficiencyof the encoder s thereforeH(A Xmitted bitsareredundant.

    - 3.3275.A)/L : .99035.esshan1%of the rans-

  • 8/12/2019 ch1 examples

    7/15

    f Chapter Discrete ourcesnd EntropyIf this result seemsa bit unsettling,,itshould. t appearswe havegotten ,,some-thing for nothing" out of the encoder.However, t"t'r'irrirrt about what the resulteans'H(B)> 1{'4) says here s more.uncertaintyn B than there s in,4. where didhis uncertainty come irom? t d: discussion t;;; ie saia we courd alrow thencoder o pick from a selectionof b symbols"r*"i","4 withaat random. f thencoder s otossinga coin," the extrauncertainty n a comes rom this randomprocessndnot from A. There s indeedmore information in r u* ,t

    " "*cess,F(B) _ u67 i"selessn termsof gettingmore nformation out.i;. Th;;rcoder is ike the town gos-ip:Along with the facts ome other unretatedor specurativeidbits.

    1.3.3 A BriefDigressionn EncryptionAll of this is not to saythat suchan encoder s useless. he difficulty and expenseofuilding a decoder,isdirectryr"tut"o,.to th"^";;;;'ot'rt" signarbeing decoded.uppose ou wanted o makeit.noi" aini"ult for asriJopv rrriroparty to decodea pri-ate message etweenyourserfand a_trienJ.o;;;;;;Irdo so would be to consrructn encoder uchas hat n the example il;".;;;;e;;areof encatle en yp on encodes.Ther functi,o;,,.o*-"'":lt*"[t":T H] JtT#;aintyo the ransmittidm"rsage.e., aiserre^entrofy,itrroutaddingo the nowr-dgeconveyed y the tn"rrug"lfh"'"*"., information,,H(B)- H(A),is useressoour friend but courdbe terrib-ryrustrating," ,*rJr.pp"r. 1urra, o,of varue o you).

    Let the nformationource avealphabet :1g,1].with po=pr:0.5.ThenH(e1=1. y",ncoder have lphabet{:J0, , j' ,lj u"Jr"t irr" "t"-"nir'ot., haieuinaryepresentation: (brbrbir. Theencoderi shownri rr^g*" 1.3.1 erowwh;;;.,addirion,, brocks reffifllftiif:li,li;1Ji',iiX|u"-.'t"'j'nfJir'""n,,opvr'l."i"i ""tputndheutputa(r) {1010010100000100111011}

    and he nitial ontentsf the egistersrellt=0) : 5 ( = 101).fiT:'#rr*:Jl,.oo"t is a finite statemachine. v directexamination,heoutputsequence

    _ [o t ol l-olb(I1)Lr3 Jt.''Li-1""'

    drc dFutfirre

    "Ar y th &lllg$ldea$t

    srtfr:ctt

    In thestelemen

    The encoof8fromelemenspseudorasequence

    Thiis too simfollowed"crackin-reverse-enexampleble. We sh

    1.3.4 SummarL,et'ssumforming thto improvdancy in t

    boFigure 1.3.1 Encoderor Example bl1.3.3

  • 8/12/2019 ch1 examples

    8/15

    Section1.3 SourceCoding 15whereall addit ions re modulo2.Let ns: pr(D :0),nt: pr(b-: 1), etc., nd define herobabilitystaleas he vector

    pa : Lno r. . . nt l r .1' Ty timet, Pugives he probability of the encoderbeing n eachof the eight possiblestates.By drawing a state diagramfor the encoderof Figure 1311,we .un .hor" that the probabilitystateasa function of time is givenby

    Ps(t+ t1 :

    . 5 0 0 0 0 . 5 0 0. 5 0 0 0 0 . 5 0 00 . 5 0 0 . 5 0 0 00 . 5 0 0 . 5 0 0 00 0 . 5 0 0 0 0 . 50 0 . 5 0 0 0 0 . 50 0 0 . 5 0 0 . 5 00 0 0 . 5 0 0 . 5 0subject o the constraint

    7

    )o '= t '

    t - vIn thesteady_state,"(t * 1): puQ).Bydirectsubstitution, eelementsof B satisfieshis condition.Tirerefore,

    PoU),

    see hat rr,:'l.f8,for al l/H (B ) : - ) r 1 t og2@) : 3 > H (A) .j: o

    The encoderof Figure1''3.1s a pseudorandom umbergeneratorwhich generates ll elementsof I from 1 hrough7 f input a;.O,-with equiprobable 10 and : i,ih"'"n.oder generates llelementsof B with eqpal probability.The increase n entropy is due to thememory of. theseudorandom umbersequences ound"-,f;:*'#_:i;ff#,11ffiff:[t'J givenbove,he ncoder'sutputiD) : {0 ,0 , ,2 , ,2 , ,2 ,50L,2 ,,3 ,6 ,6 ,,7 ,5, ,1 ,3 ,6 ,6, 6 I .

    This examplewould not makea very good encryptioncodebecausehe encoderis too simple'Note,for instance,hat 0 is at-waysoiloieJ by either 0 or 1., is alwaysfollowedby either4 or 5,and so o".tl .*p"ri in "".tyption wouldhaveno difficulty"cracking" this codeby compiling a list of what coae woras follow eachother andreverse-engineeringhe encoder try it ). However, he basic hemeoutlined n thisexamplecanbe expandedon to createan encryptionsystemwhich is rather formida-ble'we shat saymore on the subjectof encryptio" i" cituft., s.'1.3.4 Summary f Section 1. 3Let's summarizewhat we've learned n this section.Sourcecodingconsistsof trans-forming the informationsource'salphabet o a codeword alphabet.one use or this isto improve the transmission fficiencyby data"ornpr"rriorr.In this application, edun-dancy n the sourcemessages removedso that the averagenumbei of bits actually

  • 8/12/2019 ch1 examples

    9/15

    Section .4 HuffmanCoding 17words n,a dictionary,your computerwould probably deliver the secondpunctuationabove'You would have to add iome rules governingsentencestructure to delineatebetween he two punctuationsgivenabove.Suppose iu did this.consioer the sentence

    TIME FLIES LIKE AN ARROW(noun)(verb) (predicate).

    If your program could parseTIMEFLIESLIKEANARROW using the examplesen_tencestructure ule givenabove,what would it do withFRUIT FLIES LIKE A BANANA (wouldn'r youif you werea fruit fly?)

    . ft9 Fnglish languages not generallyself-punctuatingbecauset containssuffi-cient ambiguities hat somemessagesouldbe correctlypu.i o irr-t*o differentwaysIF I WANTED To 'ICK oNE vs. IF I WANT ED To 'ICK oNEand' n other cases,he rulesconnectingpossiblewords to valid sentences re compli-cated. t is for this reason hat we ur ,fu r, commas, nd other punctuationmarks nwritten language.A prefix code is a -codehaving punctuation built in to the structure (ratherthanadded,n usingspecialpunctuatioi symbols)- his s accomflisheouy designinghecodesuch hatno codeword saprefixof someoiher (longer)coie woro.Suchcodesaresaid o satisfy heprefix conditioi.They arealsosaid tobe instantaneouslyecodable.

    s. lutilry Scan he sequencerom left to right,andplacea commaafter everystringof bitsthatis a valid codeword.observe hat neither noi ro is a valid ,oo *orJ, Lut 100 s a validcodeword.Parsing he sequencen thisway,we get

    For he code n Example1,.3.Z,decodetheequence100000111 11101101

    assuminghe codewordsare ransmittedbit_seriallyrom left to rieht.

    whichdecodes s100, 0, 111,11110,1101,

    AOA1 A0 AO A2 AI A2 A2 As Ao.Notice hat this parsings unique.For example,hereareno codewordssuchas 1000or 01111.Any codewhich satisfieshe piefix condition s structuredso that whenevera string of n bits isl:::::i:1tl::j .:.^d, T::d, jhgre is no possibitityof someotr, , 1r,ong .) ode word whichb gt*f the samewayas he first recognizable ode*;;J;;, t ;J;;;

    1.4.2 Construction f Huffman CodesHuffman codes are constructedusinedefined in a mannerwhich quarantees a tree-building algorithm. The algorithm isthe codewill satisfy he prefix coniition. The

  • 8/12/2019 ch1 examples

    10/15

    18 Chapter Discrete ourcesndEntropyfirst step n the procedure s to list the sourcesymbolsn a column n descending rderof probability.This is illustrated n Example r.4.2.Thesourcesymbols orm the leavesof thetree.InExample .4.2,symbol owith probabilitypo 0.j is at thetop of thecol_umn,andsymbolarwithpr: 0.05 s at thebottom.Step2 of the algorithm s to combinesymbols, tartingwith the two owest roba-bility syrnbols,o^forma new compoundsymbol.In Examlb r.4.2,symbols .r'and.,arecombinedo form compound ymbolararwithprobabilityp,r:l pr* pr:\lZO.ft "ombiningof two symbols orms a branchln the tiee with stemsgoingfrom the com-pound symbolto the two symbolswhich formed it. The top stem s labeledwith a ,,0,,and the bottom stem s labeledwith a "1-". We could equallywell label the stemsas"1"'and"0", respectively,nstead.)After thisstep, he tree n example1.4.2 onsists fthe threesymbols_ao,11a2,3with heir respective robabilities.Step wo is then repeated,using he two lowest probability symbols rom thenewsetof symbols'This process ontinuesuntil all of the original sym-bolsavebeencom-bined nto a single ompoundsymbol,asa1a2a3;havingirobability 1.The final tree isshown nExampleI.4.2.Code words.are-assigned y reading the labels of the tree stems rom right toleft back o the originalsymbor. or inshnJe, n Exampre .4.2,thesymbola, is asfilnedcodeword 111.Code-wordassignmentsor the other symbolsaredone n th same*way.Note that the final result s the sameas he codewordLssignmentsn Example1.3.1.

    CodeWord0

    fflm(mtlqetscsatit

    1.4.3 HHuaateueacinTRswYcdthbislonte

    Constructa Huffman code or the4-arysourcealphabetof Example1.3.2.Solution* See he tree constructionn the followingFigure:

    Probability Symbol

    0.50

    0.30 r1

    0.15 d2

    0.05 o3

    1 0

    1 1 0

    1 1 1

  • 8/12/2019 ch1 examples

    11/15

    24 Chapter DiscreteourcesndEntropyrf (n,a) s already n the dictionaryn step ;the en-coders processing stringof sym_bols that has occurr.ed. t.least once pieviously. setting the next value of rz to thisaddress onstructsa rinkedrist that alrows he stiing or ri-uor, to be traced.rf (n, a) s not already n the dictionary in stJp 2,ihe encoders encounteringanew string hat hasoo1.b"": previouslyprocessed.lthansmitscodesymbolzl,whichlets the receiverknow the diciionary uioi"r, of the lastsource ymbol n the previousstring'whenever the encoder ransmitsa codesymbol, t alsocreatesa new dictionaryentry(n'c)' n this entty,n s a pointer to the lastsourcesymbot n ttr" previousstringofsourcesymbolsanda is theroot symborwhichbegin, , ;; ,;;;;. code word n is thenresetto the address, f (0,a).Th9"0" pointer m tlms"rr,ry po-i-.rt,o the null characterwhich indicates he beginningof the new string. h" "n"oaer,sJictionary-buildingandcodesymbol ransmission ro."r, is illustrateJin the folrowin &ampte.

    A binary nformationsource mits he sequencef symbols 1000101100101110001111etc.onstruct the encodingdictionaryanddetermine hesequenceor t.anrmitt"o codesymbols.solution: Initialize the dictionaryasshownabove.Since he source s binary, he initial dictio-ary will containonly the null entry and the entries 0,0i;; (q1ilil;.nary addresses , 1,nd2, espectively.he nitial valuei forn andm are : 0 andm:' 3.T\eencoder,s perationshen describedn the followins table.

    sourcesymbol presentn ptesenlm transmlt next n dictionaryentry22I

    2 , L2 , 01 , 05,14 , 13 , 0

    6 , 01 ,

    4 ,0

    o , r

    61

    22115242J1561,232+1,5f)231 1

    JJ45o67788999101 111.7212

    1J1313I41,4

    0221122315612

    J247562J

    170001,011,001,0I1,1.000

    l-5-3 T

  • 8/12/2019 ch1 examples

    12/15

    Section l.5 DictionaryCodes nd Lempel_ZlV odinoTheencoder,s ictionary o thispoint is as ollows:

    dictionary address dictionaryentry

    25

    01,2345o789101 i12

    I JI4

    0,null0 ,00,1) 12 ,01 , 04 , 13 ,06 ,01 , 1

    J ' I4,06,1.noentryyet

    1.5.3 The DecodingprocessThedecoderat.the eceivermustalsoconstruct dictionary dentical o the oneat theransmitterand t must decode t" t"""iu.o-il"*r;ffi;t';.Notice from the previousxamplehow the encoderconstructs odesymborr"ro, i.rngs and that the encoderoesnot transmit T -?ny "oo. *rrar as ii trassourc""ryrrruorr.operation of theecoders governed y the'followinfobservations:

    t l:t"rifflo,ot unt codewordmeanshat a newdictionary ntrymusrbe2' Pointern for this newdictionaryentry is the sameas he receivedcode wordn;' Sourcesymbola for this entry is not yet known,since t is the rootsymbolof theextstring whichhasnot y"f b""n transmitted y theencoder).If the addrer.oi-tl1. nexJdictionaryentry s m,wesee that the decoder anonrycon_truct a partial entry(n, ?)since t -urt await the n"*t ."."iu"d codeword to rina tneoot symbola for this entry'It can,however, ill in the missingsymbola in its reviousictionaryentry at address -t.'i-",unalso decode rr"-rour"" symbolstringassoci_3ff#lfi:T:::|;:h5ift tos""i'o*tr'i'i"aon"';;;;i;,;;;;;;;;;;il",

    Decode he received ode-wordsransmittedn Example .5.r. (Note:As you fonow alongwithlxil;:,fi,',f.1i"13"',fri11.:T,Ha:Fi"'1i'J-oi.,i;;;.}';ffiij,,on.toccurhat

  • 8/12/2019 ch1 examples

    13/15

    30 Chapter Discrete ourcesndEntropysmall setof compoundsourcesymbols,within somesmall rangeabout nH(A),makesup the great majority of all blocks emitted by the source.Tlis subsetof tompoundsymbols s called the typicalsel.As a consequence, sourceencoder hat assigns odewordsto elementsof the typical setusingnH(A) bits per codeword will achievedatacompressionwith very high efficiency.We will make this idea more formal when wediscussShannon's ource-codingheorem n Chapter9.Now, generally,nH(A) is not an integer.Huffrnan codesmust ,oroundup,, thenumber of codebits used o representelemLntsof the typical set to obtain an integernumberof bits per codeword.Arithmetic codesdo not o-o t ir.

    1.6.2 TheArithmeticCodingMethodHow is it possible o have a codeword which doesnot contain an integer number ofbits?supposeeacha, e A is encodednto a real numberB,lyingin the iriterval

    0 = 8 i 1 7 .Clearly this ideamight require more bits to representB,,since alargernumber of bitscould be necessaryo representa real numbet.Now'suppose hit we encode hesequenceo r"' by adding ogetherscared ersionsofthe B,accordingob: Bo*A$r+ Lz l }z+ . . . ,

    where eachB,is the codenumber correspondingo s,andwhere the A, are monotoni-cally decreasing cale actorsalso ying in the interval bet*een 0 and i. If we pick theA, n sucha way that it is possible o later decompose back nto the original ,"qu"rr."of B,s, his codecan be decoded.Now, magin" we ,eprerent eachB, iri binary fixed-point format. For now,we'll alsoassumewe canuseextremely ong iixed-point num-bers o do this.As we add the successiveermsof b, the bits representing he differentscaledB, are added ogether.Thus, t is the sum of these epresentationwhich repre-sentshe entirecodedblock.In a veryrealway,we can egard he bit positions sed orepresentb asbeing. shared"amongseveraldifferent B,.Th" net effect of this sharingis that_e_achB,effectivelyusesa non-integernumberof'bits (on the average).We now turn to the detailsof how to carry this out.Aiithmetic coal" beginsbyassigningeachelement of.A a subintervalof the real numbersbetween0 and 1. Thelengthof the assignednterval is set equalto the symbolprobability.

    Let Abe the nformationsource iven n Example1.2.1. ssigneach ,e ,4,a fractionof the realnumber nterval0 = Bt

  • 8/12/2019 ch1 examples

    14/15

    Section .6 Arithmetic oding 31The arithmeticencodingprocessconsistsof constructinga code nterval (ratherhan a codenumberb).whiclliniquery aeGues a brock or ruZ. r*iu sourcesymbols.hen this interval,lo, is complet d, ;*iiih;ve upper and lower boundswithin whichnyb will satisfy he toding conditiondiscussedabove.i.e..

    4: fL ,A ,L=b

  • 8/12/2019 ch1 examples

    15/15

    Sectionl.6is assignedts own intervalL When the decoderdetects hatsymbol, t exits he decoding oop.

    Arithmetic-Coding 33it has decoded the STOP

    For thesource ndencoder f exampleJ..6.2,

    d.ecode :0.57470703t25.solution: The step-by-stepesultsof the algorithmare given n the fo'owing table:

    next Il next L nextA ai00.50. 50. 50.57725

    10. 8u.o)0.5750.575

    1,0.30.150.0750.00375

    0. 8U.OJ0.575u.) )0.57481.25

    0. 50.50.50.577250.57425

    0.30.150.0750.003750.0005625

    4Iot3r 2

    a1a0aoa3a2

    In tracing hrough he stepsof exampre .6.3:{9} mustpay attention o the precisionwith which you calculate he quantity (b- ry1t aiitiJ other variables.Roundoffrror in this calculationcan eadto an erroneousanswer.1.6.4 Other ssuesn Arithmetic Coding

    In the practicalapplicationof arithmeticcodes,here are severalother issueshat theode mplementatiol usuallyneeds o address. n. ,.t oi issuesarises rom the facthat computersandhardwarl encodersalwayshave inite frecision in their arithmetic.e must therefore pay attention to the irru"r-or;;.;#; overflow and underflow.et us suppos:-thatH .and I. are represented sn-bit binarynumbers. et us furtherupposewe represent he subintervalcdcurations s";;l A . ,s,usingf uit, "i p.".i_ion'Finally,letus supposehe machine's umericur tJ.iJon is imited op bits. t haseenshown hat avoidanceof overflowor underflo* "rro., requiresf = n - 2

    andf * n = p .

    This imits the sizeof the sourcealphabet,4toVl=zr,

    since Sr, S, re imited to /bits of precision.For example, uppose : 16bits and, : 9its.Then/< 7 andour aiphabetAis thereforeri-it"'Ji" iis symbors.Another issue hat usually s important n practice s the need or beingable oransmitand decode he information ion the fly.;'rn the Jg;rithms presentedearlier,


Recommended