IEICE TRANS. COMMUN., VOL.E99–B, NO.3 MARCH 2016723

PAPER

Pseudo Distance for Trellis Coded Modulationin Overloaded MIMO OFDM with Sphere Decoding∗

Ilmiawan SHUBHI†a), Student Member and Yukitoshi SANADA††b), Senior Member

SUMMARY Efficient detection schemes for an overloaded multiple-input multiple-output (MIMO) system have been investigated recently. Theliterature shows that trellis coded modulation (TCM) is able to enhance asystem’s capability to separate signal streams in the detection process ofMIMO systems. However, the computational complexity remains high asa maximum likelihood detection (MLD) algorithm is used in the scheme.Thus, a sphere decoding (SD) algorithm with a pseudo distance (PD) isproposed in this paper. The PD maintains the coding gain advantage of theTCM by keeping some potential paths connected unlike conventional SDwhich truncates them. It is shown that the proposed scheme can reduce thenumber of distance calculations by about 98% for the transmission of 3 sig-nal streams. In addition, the proposed scheme improves the performanceby about 2 dB at the bit error rate of 10−2.key words: overloaded MIMO, trellis-coded modulation, sphere decoding,pseudo distance

1. Introduction

Multiple-input multiple-output (MIMO) is one of the majorbreakthroughs in the wireless communication area. Utiliz-ing multiple transmit and receive antennas, MIMO systemscan yield significant improvement in capacity [1]. Due tothis ability, MIMO systems are being widely used as a stan-dard in the recent communication systems. Wireless LAN,a popular technology for the short range communications,also employs MIMO to achieve higher data rates. Typically,MIMO systems require the number of receive antennas tobe equal to or larger than the number of transmit anten-nas. However, some mobile entities might not be able tomeet this ideal condition due to their form factor limitations.Therefore, an efficient detection algorithm is always desir-able in an overloaded MIMO system, where the number oftransmit antennas is larger than that of receive antennas.

The joint detection scheme presented in [2] has shownthat it is possible to demodulate multiple co-channel sig-nals. Furthermore, a demodulation algorithm for a multiple-input single-output (MISO) system has been proposed in[3]. However, both [2] and [3] do not exploit potential

Manuscript received May 23, 2015.Manuscript revised October 26, 2015.†The author is with the Graduate School of Informatics, Kyoto

University, Kyoto-shi, 606-8501 Japan.††The author is with the Departement of Electronics and Electri-

cal Engineering, Keio University, Yokohama-shi, 223-8522 Japan.∗The part of this manuscript has been presented in the Interna-

tional Symposium on Intelligent Signal Processing and Communi-cation Systems 2013.

a) E-mail: ilmiawan@dco.cce.i.kyoto-u.ac.jp.b) E-mail: sanada@elec.keio.ac.jp.

DOI: 10.1587/transcom.2015EBP3218

coding gain in the schemes. Without channel coding, 2 dBperformance degradation occurs with each additional signalstream for the case of single receive antenna systems.

Trellis coded modulation (TCM) has been combined in[4], [5] to enhance the system capability to separate a de-sired signal and its co-channel interference. In order to re-duce the complexity addition, the scheme in [4], [5] havealso used state reduction algorithms called T-Algorithm [6],and M-Algorithm [7]. However, since multipath compo-nents are also counted as signal streams in these schemes,the computational complexities remain high.

Complexity reduction in MIMO systems has been in-vestigated by many researchers. One of the most promisingalgorithms is sphere decoding (SD), firstly introduced by U.Fincke and M. Pohst in [8]. SD offers equal performanceof that of maximum likelihood detection (MLD) with largereduction in computational complexity. In the overloadedMIMO case, the SD algorithm can also be applied with apseudo-antenna augmentation scheme as described in [9].It only requires simple modification in the receiver matrix.However, the research in [9] did not implement any codingfor performance enhancement.

Combining SD with TCM brings a new challenge. AsSD calculates distances only for several candidates, it islikely that the correct trellis path will be truncated if SDdoes not include the correct symbol as its candidate. Thetruncated path will force the erroneous decoding of severalconsecutive symbols. In this paper, a pseudo distance (PD)scheme is introduced in order to counter this problem. ThePD will keep some potential paths that are truncated in TCMwith conventional SD connected. By keeping the potentialpaths, the coding gain provided by TCM can be realized.

The rest of the paper is organized as follows. The over-loaded MIMO OFDM system and the SD algorithm with anaugmented pseudo-antenna are briefly introduced in Sect. 2.The proposed scheme is described in Sect. 3. Numericalresults obtained through computer simulation are shown inSect. 4, and finally, conclusions are presented in Sect. 5.

2. System Model

2.1 Overloaded MIMO OFDM

A MIMO OFDM system with NT transmit antennas and NR

receive antennas is shown in Fig. 1. Herein, as in the over-loaded case, NT > NR. The bit data are firstly demultiplexedinto NT branches. In every branch, TCM is employed to

Copyright c© 2016 The Institute of Electronics, Information and Communication Engineers

724IEICE TRANS. COMMUN., VOL.E99–B, NO.3 MARCH 2016

Fig. 1 Overloaded MIMO OFDM system with TCM.

convert L information bits to M coded bits. Afterward, theM coded bits are modulated to 2MQAM coded symbol. Inorder to maximize the coding gain of TCM, an interleaver isemployed to avoid weak channel responses appear consec-utively and provides frequency diversity. The interleavedcoded symbols are then assigned to subcarriers.

Suppose that Sp[k] represents the interleaved symbolon the kth subcarrier at the pth branch, the OFDM signal onthe pth branch is then given by

up[n] =N−1∑k=0

S p[k] exp

(j2πnk

N

)(1)

where n is the time index (n = 0, 1, . . . ,N − 1) and N isthe size of an inverse discrete Fourier transform (IDFT). Aguard interval (GI) is then added by replicating the last partof the OFDM symbol.

In the receiver side, the received signal at the qth re-ceive antenna, yq[t], is converted into digital samples at therate of Ts and can be written as

yq[n] = yq(nTs). (2)

After removing the GI and taking the DFT of N samples, thesignal on the kth subcarrier can be expressed as

Zq[k] =N−1∑n=0

yq[n] exp

(j−2πnk

N

)

=

NT∑n=0

Hqp[k]S p[k] +Wq[k] (3)

where Hqp[k] is the frequency response on the kth subcar-rier between the pth transmit antenna and the qth receiveantenna while Wq[k] is the noise on the kth subcarrier at theqth receive antenna.

2.2 SD Algorithm with Augmented Pseudo-Antenna

In general, Eq. (3) can be written in the matrix form as

Z[k] = H[k]S[k] +W[k] (4)

where

Z[k] =[Z1[k] Z2[k] ... ZNR [k]

]T , (5)

H[k] =

⎡⎢⎢⎢⎢⎢⎢⎢⎢⎢⎢⎣H11[k] · · · H1NT [k]...

. . ....

HNR1[k] · · · HNRNT [k]

⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦ , (6)

S[k] =[S 1[k] S 2[k] ... S NT [k]

]T , (7)

W[k] =[W1[k] W2[k] ...WNR [k]

]T . (8)

MLD, which is the best detection algorithm in terms of per-formance, solves Eq. (4) as

S[k]ML = argminS 1[k],S 2[k],...,S NT [k]∈{S }

‖Z[k] −H[k]S[k]‖2 (9)

with {S } implies possible symbols in the constellation.However, the MLD needs exhaustive search of S[k].

The computational complexity increases exponentially withthe number of the signal streams and the constellation size.In order to reduce the complexity, SD calculates the dis-tances only over those points which lie inside a sphere witha certain radius, r. The searching constraint is expressed as:

‖Z[k] −H[k]S[k]‖2 ≤ r2. (10)

Letting S[k] becomes the center of the sphere, the searchingprocess is then represented as

(S[k] − S[k])HH[k]HH[k](S[k] − S[k]) ≤ r2. (11)

As described in [10], Eq. (11) is possible to be solved byusing Cholesky factorization as

NT∑q=1

U2qq[k]

∣∣∣∣∣S q[k]−S q[k]+NT∑

p=q+1

Uqp[k]

Uqq[k](S p[k]−S p[k])

∣∣∣∣∣2

≤ r2

(12)

where U[k] is the upper triangular NT × NT matrix such thatU[k]HU[k] = H[k]HH[k] and Uqp[k] is the (q, p)th elementof U[k].

Factorizing matrix H[k]HH[k] requires the square ma-trix to be positive definite which cannot be attained in theoverloaded MIMO case as H[k] does not have full columnrank. In this circumstance, Cholesky factorization gives 0 in

SHUBHI and SANADA: PSEUDO DISTANCE FOR TRELLIS CODED MODULATION IN OVERLOADED MIMO OFDM WITH SPHERE DECODING725

Fig. 2 Overloaded 2×1 MIMO system with augmented pseudo-antenna.

the diagonal elements of the matrix U[k], and Eq. (12) can-not be solved. The concept of an augmented pseudo-antennascheme proposed in [9] is shown in Fig. 2. It is a 2×1 MISOwhich has been modified into a 2×2 MIMO with pseudo an-tenna addition.

In matrix representation, the modified channel matrix,H[k], can be written as

HNT×NT [k] =

[αINT−NR 0(NT−NR)×NR

H[k]

]. (13)

By this modification, the channel matrix, H[k], will havefull column rank so that H[k]HH[k] can be factorized andthe solution can be obtained. In addition, the value of thepseudo channel response, α, needs to be minimized as longas the numerical stability is maintained in the computingprocess.

3. Proposed Scheme

TCM is a channel coding technique well known for its band-width efficiency. TCM can be used as the inner code thatworks for outer coding techniques such as low density par-ity check (LDPC). In TCM, a joint decoding can be utilizedto separate a signal from its co-channel interference. It canbe realized using the super-trellis diagram [14]. To providedistance information for the joint decoding, SD can be em-ployed as the detection algorithm. However, problems oc-cur as the SD calculates distances only for several candi-dates. Therefore, it is likely that a correct trellis path will betruncated when SD does not include correct symbols as itscandidates.

Figure 3 shows the detection process on the receiverusing the super-trellis. It uses 42 states that represent all thepossible state combinations of 2 4-states TCM signals from2 transmit antennas. In the figure, SXY denotes the state Xof the transmitter Y. The SD symbol candidates of 3 con-secutive TCM symbols after deinterleaving is shown in thefigure. The channel response of Stream 1 is assumed to bestrong enough so that the SD chooses the correct candidatesfor all of 3 slots. On the other hand, Stream 2 has a weakchannel response in slot l. Therefore, the SD chooses in-correct candidates in this slot. As can be seen in the figure,once the correct path is truncated, it will affect not only thesymbol in that slot, but also its neighboring symbols will bemistakenly decoded.

In this paper, PDs are proposed in order to improvethe performance of the conventional SD. As the scheme is

Fig. 3 Example of truncated path in TCM using SD Algorithm (Trellisdiagram is as specified in [11]).

implemented in the receiver, it does not require any addi-tional process on the transmitter which is already describedin Sect. 2. The PDs could enhance the performance by keep-ing potential trellis paths which are truncated in the conven-tional SD. To keep these potential trellis paths, the PDs de-termine the distances for unselected candidates utilizing thedistance between the selected candidate and each constel-lation point. In addition, as errors in the candidate selec-tion will often occur for the transmitter with a weak channelresponse, the PDs will be given to the signal stream withthe worst channel response when the number of the signalstream is more than two. For the case of the worst channelresponse belongs to two or more signals, PD is applied toone of those signals randomly.

The example of the PD calculation in the 8PSK modu-lation on the subcarrier k is shown in Fig. 4. In the sequel,the subcarrier index, k, will be omitted for simplicity. Letd(CW/CS ) represents the vector between the combination ofthe coded symbol candidate of the stronger signal, CS , andthe coded symbol candidate of the weaker signal, CW . Thesquared-distance for the selected candidates is calculated by

d2(CW/CS ) =NR∑q=1

∣∣∣∣Zq −(HqS S S + HqWS W

)∣∣∣∣2 (14)

where S p denotes the signal representation of the coded

symbol Cp in the constellation point by S p = ej2πCp

2M whileHqS and HqW denote the channel responses of the strongersignal and the weaker signal in the qth receive antenna.

For this example, it can be seen from Fig. 4 that thecoded symbol “1” is chosen for the stronger signal andcoded symbol “0” is chosen for the weaker signal by the SDalgorithm. Afterward, SD calculates the squared-distance of

726IEICE TRANS. COMMUN., VOL.E99–B, NO.3 MARCH 2016

Fig. 4 PD calculation concept.

these selected candidates and denotes the result as d2(0/1).Using this information, the squared-distances for the othercandidates can be attained. Let d(0/1) = dr(0/1)+ jdi(0/1),d2(1/1) can be obtained by

d2(1/1) =

(|HqW |

(1− 1√

2

)+dr(0/1)

)2

+

(|HqW |√2−di(0/1)

)2

= |HqW |2(2 − √2)+d2r (0/1)+d2

i (0/1)

+|HqW |(dr(0/1)(2− √2)− √2di(0/1)

= |HqW |2γ21+d2(0/1)

+|HqW |(dr(0/1)(2− √2)− √2di(0/1))

(15)

where γC denotes the distance between the coded symbol,CW , and the selected coded symbol, CW , on the constella-tion diagram. If the SD selects more than 1 candidates ofthe weakest signal in the same coded symbol candidate’scombination of the stronger signals, the PD will select thecandidate which has the smallest distance. This coded sym-bol is then denoted as CW .

For XPSK modulation, the value of γC for each codedsymbol can be obtained by

γ2C = 2

(1 − cos

2π(CW−CW)2M

). (16)

On the other hand, if XQAM modulation is being used in-stead of the XPSK, the value of γC can be determined by

γ2C = 4

⎛⎜⎜⎜⎜⎜⎝∣∣∣∣∣∣⌊CW

M

⌋−⌊CW

M

⌋∣∣∣∣∣∣2

+(∣∣∣CW − CW

∣∣∣ (modM))2⎞⎟⎟⎟⎟⎟⎠ (17)

where �x denotes the largest integer that is not greater thanx, and a (modb) implies a modulo operation of a by b. Equa-tions (16) and (17) are assuming the normal binary orderingconstellation. If the other constellation such as Gray coding

ordering is being used, the values of CW and CW in the equa-tions need to be changed to the corresponding values in thenormal binary constellation with the same coordinates.

Let δC = |HqW |γC , Eq. (15) then can be written as

d2(1/1)=δ21+d2(0/1)+|HqW |(dr(0/1)(2− √2)− √2di(0/1)

=δ21+d2(0/1) +2|HqW |(dr(0/1)(1−cos

(π4

))−di(0/1) sin

(π4

)).

(18)

Using the same calculation as in Eq. (18), {d2(CW/1)}(CW =

2, 3, . . . , 7) are obtained by

d2(CW/1)=δ2C + d2(0/1) +2|HqW |(dr(0/1)(1−cos

(CWπ

4

))− di(0/1) sin

(CWπ

4

)).

(19)

The calculation of Eq. (19) for all {CW} is actuallyequivalent to choosing all of 8 symbols as candidates thatwill increase the computational complexity. In this equa-tion, the second part is obtained from SD. As can be seen inEq. (14), SD requires NR squaring processes for each candi-date. In the third part of Eq. (19), multiplication processesare required to both dr and di which has different value foreach receive antenna. Therefore, the third part of Eq. (19)requires 2NR times additional multiplication processes. Inaddition, if SD selects two or more candidates, the valuesof dr and di will vary for each candidate. Another addi-tional multiplication processes, therefore, are required fromthe third part of Eq. (19). Fortunately, when the value of sig-nal to noise ratio (SNR) is sufficiently large, the values of dr

and di are relatively small. Therefore, to reduce the com-putational complexity, the PD neglects the last part of theequation and thus yields

d2(CW/1) = δ2C + d2(0/1). (20)

The value of δC involves only one multiplication and de-pends only to the value of |HqW | and γC . The value of δC ,therefore, can be used for any candidates of SD. In addition,the value is also equal for all receive antenna. By keepingonly the first and second parts of (19), the PD could main-tain the additional complexity to be negligible compared toconventional SD. This simplification, however, brings thedifference between the actual and pseudo distances. FromEq. (19), we can observe that this difference is getting largerwhen the value of |HqW | is larger. However, as it is onlya small possibility for a transmitter to have two consecu-tive weak channel responses, it is likely that this gap will becompensated in the next slot.

In general, for NT transmit antennas, the PD for all thecoded symbols of the weakest signal, CW , can be written as

d2(CW/C1, . . . , CNT |p�W )=δ2C + d2(CW/C1, . . . , CNT |p�W )

(21)

SHUBHI and SANADA: PSEUDO DISTANCE FOR TRELLIS CODED MODULATION IN OVERLOADED MIMO OFDM WITH SPHERE DECODING727

Fig. 5 SD + PD example for 2 transmit antennas.

where Cp represents the coded symbol candidate of the pthtransmit antenna. In addition, the shortest distance which isused as the reference in the PD calculation in the Eq. (21) isobtained by

d2(CW/C1, . . . , CNT |p�W ) = minS W

NR∑q=1

∣∣∣∣∣Zq −NT∑p=1

HqpS p

∣∣∣∣∣2

.

(22)

An example of the SD with the PD for 2 transmit anten-nas and 1 receive antenna using 8PSK modulation is shownin Fig. 5. In this example, the SD selects the 4 pairs ofthe coded symbols as its candidates, which are (CW/CS ) =(0/0), (1/1), (2/2), and (3/3). The distances are then calcu-lated for d(0/0), d(1/1), d(2/2), and d(3/3). Meanwhile, thePD calculates the values of {δC} by multiplying the channelresponse of the weakest signal with the distance between thecoded symbol candidate and each point of the 8PSK con-stellation diagram. These values are then added to obtainthe pseudo distances over the coded symbols which are notselected as the SD candidates. For unselected symbols inthe stronger signals, pseudo distances are not calculated. Toavoid computational complexity increases, the stronger sig-nals are assumed to be strong enough so that the possibilityof the SD does not include the correct candidates is small.

4. Numerical Results

Simulation conditions are described in Table 1. Some of thespecifications such as the bandwidth, the number of subcar-riers, and the guard interval samples follow the IEEE 802.11standards. Information bits are coded and modulated withTCM. A convolutional code with the rate of 2/3 and the con-straint length of 2 is employed for the 8PSK TCM [11]. Forthe 16QAM TCM simulation, a convolutional code with therate of 2/4 and the constraint length of 3 is employed [12].Afterward, the TCM symbol is interleaved using a 6x8 blockinterleaver and demultiplexed with OFDM. In the simula-tion, 3 transmit antennas and 1 receive antenna are used. The

Table 1 Simulation conditions.

Modulation Scheme 2/3 4-states 8PSK TCM2/4 8-states 16QAM TCM

Multiplexing Scheme OFDMNumber of subcarriers 64Number of data subcarriers 48Guard interval length 16Transmit antennas 3Receive antennas 1Interleaving 6 × 8 block interleavingChannel model Indoor Residential-A

Indoor Office-BChannel estimation Idealα 0.1 + 0.1 jNumber of trials ≥ 1 × 106 bits

Fig. 6 Average candidates of the SD algorithm for 3 stream signals 8PSKsystem.

channel models in the simulation are Indoor Residential-Aand Indoor Office-B, with perfect channel estimation.

In the simulation, the value of α is set to 0.1 + 0.1 j, asthis value is small enough to maintain the accuracy of the SDalgorithm calculation without affecting the numerical com-putation stability. Unless it is specified, the sphere radiusis set to

√3|α| for the 8PSK simulation and

√5|α| for the

16QAM simulation. These values are applied to all Eb/N0

values to keep the number of candidates to be the same. Itwill give a clear performance comparison between the con-ventional SD algorithm and the SD algorithm with the PD.

4.1 Computational Complexity

Figure 6 shows the average number of the candidates cho-sen by the SD for 3 signal streams with 8PSK modulation.It can be seen from the figure that with the chosen radius,(√

3|α|), the SD algorithm selects 8.5 candidates in average.Therefore, it needs only for about 2% distance calculationscompared to the MLD that needs 512 distance calculations.Moreover, it can be seen from Fig. 7 that using the radius of(√

5|α|) for 3 signal streams with 16QAM modulation, theSD selects 85 candidates in average. Therefore, it also re-quires only for about 2% distance calculations compared tothe MLD which needs 4096 distance calculations.

The number of candidates in the sphere decoding is a

728IEICE TRANS. COMMUN., VOL.E99–B, NO.3 MARCH 2016

Fig. 7 Average candidates of the SD algorithm for 3 stream signals16QAM system.

Fig. 8 Probability of the number of total candidates for 3 stream signals8PSK system.

Fig. 9 Probability of the number of total candidates for 3 stream signals16QAM system.

random variable depending on the radius, the channel con-dition, and the noise level. The distribution of the numberof candidates for 3 signal streams 8PSK and 16QAM mod-ulation can be seen in Figs. 8 and 9. From the figures, we

Fig. 10 BER vs Eb/N0 (4-States 8PSK, 3 signal streams, IndoorResidential-A).

can observe that the most of the number of candidates arevary up to 32 for 8PSK modulation and 400 for 16QAMmodulation. Even though the sudden increase of the num-ber of candidates exceeds the maximum complexity to bedealt with in the decoder, the PD has been proposed in orderto preserve the trellis paths so that the decoding process canbe continued properly.

Applying the PD to the SD algorithm basically doesnot increase the computational complexity. As described inthe previous section, the PD only needs simple multiplica-tion between the weakest channel response and the distancebetween the referred symbol and the selected symbol can-didates of the weakest signal on the constellation diagram.The other process is only simple additions which are negligi-ble compared to the distance calculation that involves com-plex multiplications. Therefore, it can be concluded that thecomputational complexity of the SD with the PD will be thesame as the conventional SD algorithm.

4.2 BER Performance

Figures 10–13 show the BER versus Eb/N0 of the 3 sig-nal streams on the Indoor Residential-A and Indoor Office-Bchannels both for 4-States 8PSK TCM and 8-States 16QAMTCM. It is clear from the figures that applying the PD to theSD algorithm could reduce the performance degradation ofthe conventional SD, especially in the low SNR. This perfor-mance improvement can be achieved as the PD can maintainmore potential paths at the TCM decoding process.

Numerical results obtained through computer simula-tion for 4-States 8PSK TCM systems are shown in Figs. 10and 11. From the figures, it can be seen that the proposedscheme improves the BER performance of the conventionalSD by about 1.2 dB on the Indoor Residential-A channel andby about 1.9 dB on the Indoor Office-B channel. At Eb/N0

of more than 25 dB, the performance of the MLD, the con-ventional SD, and the SD with the PD is similar, as the prob-ability that the SD does not include the correct candidate is

SHUBHI and SANADA: PSEUDO DISTANCE FOR TRELLIS CODED MODULATION IN OVERLOADED MIMO OFDM WITH SPHERE DECODING729

Fig. 11 BER vs Eb/N0 (4-States 8PSK, 3 signal streams, Indoor Office-B).

Fig. 12 BER vs Eb/N0 (8-States 16QAM, 3 signal streams, IndoorResidential-A).

Fig. 13 BER vs Eb/N0 (8-States 16QAM, 3 signal streams, IndoorOffice-B).

Fig. 14 BER vs Eb/N0 (4-States 8PSK TCM + LDPC, 3 signal streams,Indoor Office-B).

Fig. 15 BER vs Eb/N0 (8-States 16QAM TCM + LDPC, 3 signalstreams, Indoor Office-B).

very small. For the case of 8-States 16QAM TCM systems,it can be seen from Figs. 12 and 13 that applying the PDto the conventional SD reduces the performance degrada-tion by 0.4 dB on the Indoor Residential-A channel and by0.9 dB on the Indoor Office-B channel at BER=10−2 com-pared to the MLD.

To verify the effectiveness of the proposed scheme,computer simulations are also conducted employing theTCM as the inner code for the LDPC. A (1944, 972) LDPCcode is used in the simulation on the Indoor Office-B bothfor the 4-States 8PSK TCM system and the 8-States 16QAMTCM system. To obtain soft input required by the LDPC, asoft output Viterbi algorithm (SOVA) is employed in the de-coding of the TCM code. The performance of the MLD, theSD, and the SD with PD as the SOVA input is compared inthis simulation. From Figs. 14 and 15, it can be observedthat the proposed scheme surpasses the performance of con-ventional SD by about 1.9 dB for 4-States 8PSK TCM andby about 1 dB for 8-States 16QAM TCM. The value of this

730IEICE TRANS. COMMUN., VOL.E99–B, NO.3 MARCH 2016

Fig. 16 BER vs Average candidates at Eb/N0 of 15 dB (4-States 8PSK,3 signal streams, Indoor Office-B).

Fig. 17 BER vs Average candidates at Eb/N0 of 15 dB (8-States16QAM, 3 signal streams, Indoor Office-B).

performance enhancement is actually the representation ofthe value of performance enhancement for the case of nochannel coding at the BER around 10−2.

4.3 Reduction of the Computational Complexity

Applying the PD to the conventional SD can also be seen asan algorithm to reduce the calculation complexity from theconventional SD. In Figs. 16 and 17, the simulation is con-ducted at Eb/N0 of 15 dB using several values of the sphereradius.

The sphere radius is gradually increased until the con-ventional sphere decoding attains the equal performancewith the MLD for the BER of 10−2. The number of averagecandidates from the selected sphere radius are then plottedagainst the BER. As the additional complexity in applyingthe PD to conventional SD is negligible, the number of aver-age candidates in the graph can represent the computationalcomplexity for both conventional SD and SD with the PD.In this figure, LDPC is not implemented.

Figure 16 shows the numerical results for the 4-States8PSK TCM system. It is shown that the conventional SDrequires for about 27 candidates to have the equal perfor-mance with the MLD while the SD with the PD only re-quires 21 candidates. Therefore, the number of distance cal-culations will be reduced by about 22%. For the case of the8-States 16QAM TCM system, it can be seen from Fig. 17that the conventional SD requires 210 candidates to achieve

the equal performance with the MLD. On the other hand, theSD with the PD only requires 150 candidates that is about29% lower than that of the conventional SD. As mentionedbefore, a BER of 10−2 of a system without channel codescould represent the BER of 10−5 of a system with channelcodes. Therefore, a similar percentage of reduction on thecomputational complexity can be expected for the case ofLDPC is implemented with the BER of 10−5.

5. Conclusions

This paper proposed an SD scheme with the PD for over-loaded MIMO-OFDM systems that use TCM. The PD main-tains the coding gain advantage of the TCM by keepingsome potential paths connected unlike conventional SDwhich truncates them. It has been shown that for a 4-States 8PSK TCM system, the proposed scheme improvesthe BER performance by 1.2 dB on the Indoor Residential-A channel and by 1.9 dB on the Indoor Residential-B chan-nel at BER=10−5. Moreover, the proposed scheme can re-duce the performance degradation by 0.4 dB on the IndoorResidential-A channel and by 0.9 dB on the Indoor Office-B channel for the 8-States 16QAM TCM system comparedto the MLD. The improvement is achieved with negligiblecomputational complexity addition relative to the conven-tional SD. For a case of three signal streams, the proposedscheme requires less than 2% distance calculations com-pared to the MLD. It has also been shown that to match theperformance of MLD, our SD with PD proposal needs up to29% fewer average candidates than the conventional SD.

Acknowledgment

This work is supported in part by a Grant-in-Aid for Scien-tific Research (C) under Grant No.25426382 from the Min-istry of Education, Culture, Sports, Science, and Technologyof Japan.

References

[1] G.J. Foschini and M.J. Gans, “On limits of wireless communicationsin a fading environment when using multiple antennas,” Wireless.Pers. Commun., vol.6, no.3, pp.311–335, March 1998.

[2] S.J. Grant and J.K. Cavers, “Performance enhancement through jointdetection of cochannel signals using diversity arrays,” IEEE Trans.Commun., vol.46, no.8, pp.1038–1049, Aug. 1998.

[3] K.-K. Wong and A. Paulraj, “Near maximum-likelihood detectionwith reduced-complexity for multiple-input single-output antennasystems,” Conference Record of the Thirty-Eighth Asilomar Con-ference on Signals, Systems and Computers, 2004., pp.1158–1162,Nov. 2004.

[4] H. Murata and S. Yoshida, “Maximum-likelihood sequence estima-tion for coded modulation in the presence of co-channel interferenceand intersymbol interference,” Proc. Vehicular Technology Confer-ence, VTC, pp.701–705, May 1996.

[5] H. Murata and S. Yoshida, “Trellis-coded cochannel interferencecanceller for microcellular radio,” IEEE Trans. Commun., vol.45,no.9, pp.1088–1094, Sept. 1997.

[6] S.J. Simmons, “Breadth-first trellis decoding with adaptive effort,”IEEE Trans. Commun., vol.38, no.1, pp.3–12, Jan. 1990.

SHUBHI and SANADA: PSEUDO DISTANCE FOR TRELLIS CODED MODULATION IN OVERLOADED MIMO OFDM WITH SPHERE DECODING731

[7] J.B. Anderson and S. Mohan, “Sequential coding algorithms: Asurvey and cost analysis,” IEEE Trans. Commun., vol.32, no.2,pp.169–176, Feb. 1984.

[8] U. Fincke and M. Pohst, “Improved methods for calculating vectorsof short length in a lattice, including a complexity analysis,” Math.Comput., vol.44, no.170, pp.463–463, April 1985.

[9] C.-Y. Hung and T.-H. Sang, “A sphere decoding algorithm forMIMO channels,” Proc. 2006 IEEE International Symposium onSignal Processing and Information Technology, pp.502–506, Aug.2006.

[10] B.M. Hochwald and S. Ten Brink, “Achieving near-capacity on amultiple-antenna channel,” IEEE Trans. Commun., vol.51, no.3,pp.389–399, March 2003.

[11] R.G. McKay, P.J. McLane, and E. Biglieri, “Error bounds for trel-lis-coded MPSK on a fading mobile satellite channel,” IEEE Trans.Commun., vol.39, no.12, pp.1750–1761, Dec. 1991.

[12] F. Yang and G. Bi, “Performance of TCM DS/SSMA in urban mul-tipath fading channel,” Proc. ISSSTA’95 International Symposiumon Spread Spectrum Techniques and Applications, pp.1295–1299,Sept. 1996.

[13] Y. Li and S. Sun, “Comparison of TCM and BICM in wireless LANsystem over indoor wireless channel,” Proc. 8th International Con-ference on Communication Systems, ICCS 2002, pp.299–303, Nov.2002.

[14] P.R. Chevillat and E. Eleftheriou, “Decoding of trellis-encoded sig-nals in the presence of intersymbol interference and noise,” IEEETrans. Commun., vol.37, no.7, pp.669–676, July 1989.

Ilmiawan Shubhi received his B.E. de-gree in electrical engineering from Gadjah MadaUniversity, Yogyakarta, Indonesia and his M.Eng. degree in electronics and electrical en-gineering from Keio University, Yokohama,Japan, in 2007 and 2014, respectively. Since Oc-tober 2014, he has been a Ph.D. student in theDepartment of Communications and ComputerEngineering, Kyoto University, Kyoto, Japan.His research interest are mainly concentrated onOFDM, MIMO systems, interference cancella-

tion, and channel coding.

Yukitoshi Sanada was born in Tokyo in1969. He received his B.E. degree in elec-trical engineering from Keio University, Yoko-hama, Japan, his M.A.Sc. degree in electri-cal engineering from the University of Victoria,B.C., Canada, and his Ph.D. degree in electricalengineering from Keio University, Yokohama,Japan, in 1992, 1995, and 1997, respectively. In1997 he joined the Faculty of Engineering, To-kyo Institute of Technology as a Research As-sociate. In 2000, he joined Advanced Telecom-

munication Laboratory, Sony Computer Science Laboratories, Inc, as anassociate researcher. In 2001, he joined the Faculty of Science and En-gineering, Keio University, where he is now a professor. He received theYoung Engineer Award from IEICE Japan in 1997. He also serves as aneditor of IEEE Trans. on Wireless Communications from 2009. His currentresearch interests are in software defined radio, cognitive radio, OFDM andMIMO systems.