This chapter treats a simple class of parallel FFTs, namely those algorithms whichinvolve no inter-processor data permutations. That is, no part of a processor’s initialcomplement of data migrates to another processor.
For purposes of this chapter, it is assumed that the multiprocessor available hasP = 2d processors, and each processor has its own local memory. That is, the machinein question is a distributed-memory multiprocessor, where each processor is connectedto the others via a communication network with a prescribed topology. A commontopology is a hypercube, but others such as a regular grid or a ring are also commonlyused.
19.1 A Useful Equivalent Notation: | PID |Local M
As discussed in Chapter 17, a key step in parallelizing the FFT on such multiprocessorcomputers is the mapping of the elements of x to the processors. Assuming thatthe x elements are stored in the global array a in natural order, i.e., a[m] = xm,m = in−1in−2 · · · i0, the array address based notation
in−1 · · · ik+1|ik · · · ik−d+1|ik−d · · · i0
has been used to denote that bits ik · · · ik−d+1 are chosen to specify the data-to-processor allocation.
In general, since any d bits can be used to form the processor ID number, it iseasier to recognize the generic communication pattern if one concatenates the bitsrepresenting the ID into one group denoted by “PID,” and refers to the remainingn− d bits, which are concatenated to form the local array address, as “Local M .”
For the class of cyclic block mappings (CBMs) introduced in Chapter 17, one canuse the following equivalent notation, where the leading d bits are always used toidentify the processor ID number.
In either notation, the d consecutive bits are marked by the symbol “ | ” at both ends.The two notations are equivalent and both are used in the text.
To fully demonstrate the usage of the “ |PID|Local M” notation, Tables 19.1, 19.2,19.3, and 19.4 below show the local data of each processor after a naturally orderedinput series of N = 32 elements is divided among P = 4 processors using all possiblecyclic block mappings.
For each mapping, the content of each local array element A[M ] is identified bya[m] in the adjacent column. Observe that m = i4i3i2i1i0 can be easily reconstructedfor each A[M ] in each processor, because the given |PID|Local M specifies exactlywhich bits can be recovered from the processor ID and which bits can be recoveredfrom the local M .
Table 1 9 . 1 Identifying Po,~‘s different local data sets for all cyclic block mappings.
Local data of processor E’,s,~ expressed in terms of
19.1.1 Representing data mappings for different orderings
When the input x elements are stored in a in bit-reversed order, i.e., a[r] = xm, wherem = in−1in−2 · · · i0, and r = i0 · · · in−2in−1, a cyclic block mapping should be denotedby
instead. For example, suppose N = 32 and the mapping is denoted by |i0i1 |i2i3i4.To locate xm = x26, one writes down m = 2610 = 110102 = i4i3i2i1i0, from which oneknows that x26 is stored in a[r], r = i0i1i2i3i4 = 010112 = 1110, and that a[11] = x26
is located in processor Pi0i1 = P01.
It is useful to keep in mind that the bit sequence in−1in−2 · · · i0 is always thebinary representation of the subscript m of data element xm or its derivativex
(k)m , and the order in which these bits appear in the array address r, when
a[r] = xm or a[r] = x(k)m , refers to permutations that xm or its derivatives
undergo in a initially or during the computation.
This convention is strictly adhered to throughout this text.To make this absolutely explicit, the mappings demonstrated in Tables 19.1, 19.2,
19.3, 19.4 are repeated for bit-reversed input in Tables 19.5, 19.6, 19.7, and 19.8.Note that the actual distribution of array elements appears unaltered from that in
Table 10.4 Identifying PII’s different local data sets for all cyclic block mappings.
Local data of processor Z’I~ expressed in terms of
global array element a[m], m = id iziz il io ,
for all possible cyclic block mappings (CBMs)
Block&e=8
PIDiLocal M
[iii3ii+* i0
iid is[OOO
iidi3[001
li*&lOlO
[iai31011
lidi3llOO
li4i3llOl
[i4i3lllO
li,i&ll
Blocksize=
?IDlLocal M
Iis i2 Iid il i0
l&i2 1000
li3i2lOOl
li3i*lOlO
li~i2~011
Ii&l100
[i3i2llOl
li3i2lllO
li3i2llll
Blohize L
PIDlLocal M
li2il]i4i3i0
ji2il/OOO
li2illOOl
li2illOlO
li2iljOll
li2il[lOO
li2il[lOl
li2il[llO
Ii&[111
Blocksize=1
|PID| Local M
NAMACR
the corresponding table for naturally ordered input, because each cyclic block mapdetermines how to distribute a[r] based on the value of the address r independent ofthe content of a[r].
The reason one must have some way to indicate that a[r] = xm for bit-reversedinput is that the FFT arithmetic operations are performed on the “content” of a[r],and its content matters in specifying how an array element is used and updated in bothsequential and parallel FFT algorithms. Recall that a different sequential algorithmmust be used for differently ordered input.
Observe that both r = i0i1i2i3i4 and m = i4i3i2i1i0 can be easily reconstructedfor each A[M ] in each processor, because, as noted before, the given |PID|Local Mspecifies exactly which bits can be recovered from the processor ID and which bits canbe recovered from the local M .
Table lg.5 Identifying Pm’s (bit-reversed) local data sets for all cyclic block mappings.
Blo&size=S
PIDiLocal M
~i&~i&i~
ii& 1000
l&i1 1001
li0il[OlO
Ii&l011
Ii&l100
]i&llOl
~i&~llO
l&&j111
Local data of processor POO expressed in terms of
global array element a[r] = z~, I = i&i&id, and m = i~i~i~i~io
Computing the butterflies involving the address bits used to define the processor IDnumber will involve exchange of data between processors whose ID numbers differ inexactly one bit. Of course these processors may or may not be physically adjacent,depending on the network topology.
For example, if the p processors form a hypercube network, data communicationbetween such a pair is always between neighboring processors. If the p processors forma linear array or a two-dimensional grid, such a pair of processors can be physicallymany hops apart, and the simultaneous data exchange between all pairs can causetraffic congestion in the network.
In general, in the parallel context, if permutations are allowed, it may turn outthat part of a processor’s complement of data may migrate to another processor. Thischapter deals with the case where no permutations are performed.
19.2.1 Parallel DIFNR and DITNR algorithms
Consider first the sequential FFTs for naturally ordered input data. To parallelize theDIFNR algorithm from Chapter 4, one may use any one of the cyclic block mappingsfrom Chapter 17. For N = 32, the computation using the block (or consecutive data)mapping is depicted below.
Table 19.8 Identifying PII’s (bit-reversed) local data sets for all cyclic block mappings.
Local data of processor PII expressed in terms of
global anay element o[r] = z,,,, r = i&iziaid, and m = idisi&io
for all possible cyclic block mappings (CBMz)
P i o i l Blocksize=
= P l l ?IDlLocaI M
f i li~i+&i,
oL24l Ii1 is I000
01251 [iI is 1001
01261 ii1 ia 1010
01271 Ii~i~lOll
a[281 IiAllOO
oP91 li1izllOl
01301 li1izlllO
d311 liIi$ll
P. . ~1~2
= P l l
e1
OD21
G31
Q41
a51
@81
mI
~~3~1
01311
B l o c k s i z e ~ 2 P iBlocksize=l Pi,i,
PID[LocaI M = P IPIDlLocaI M = PII
I i 2 i s i i 0i * i , + I I i 3 i , l & l i I i 2 +I Ii&l000 +I Iis i, poo d31 lizi~lool 0171 Ii&[001 d71 li+~lolo a41 [i~i~jOlO owl ji+Sloll a51 Ii&l011 a51 ligi&OO a21 ~i&llOO aI li+$ol @31 Ii&l101 01231
[izi$lO ~~3~1 Ii&l110 a[271
lisi&ll 01311 Ii&l111 01311
Blocksize=8
|PID|Local M
|i4i3|i2i1i0 |�i4�i3|i2i1i0 |τ4
�i3�|i2i1i0 |τ4τ3|
�i2i1i0 |τ4τ3|τ2
�i1i0 |τ4τ3|τ2τ1
�i0
(Initial Map) ⇐==⇒ ⇐==⇒
The shorthand notation previously used for sequential FFT is augmented above by twoadditional symbols. The double-headed arrow ⇐==⇒ indicates that NP data elementsmust be exchanged between processors in advance of the butterfly computation. Thesymbol ik
�identifies two things:
• First, it indicates that the incoming data from another processor are the elementswhose addresses differ from a processor’s own data in bit ik.
• Second, it indicates that all pairs of processors whose binary ID number differ inbit ik send each other a copy of their own data.
The required data communications before the first stage of butterfly computation areexplicitly depicted in Figures 19.1 and 19.2; the required data communications beforethe second stage of butterfly computation are depicted in Figures 19.3 and 19.4.
Of course, the other three possible cyclic block mappings may be used, and thecorresponding parallel algorithms can be similarly expressed in the shorthand notationsbelow.
i4|i3i2|i1i0�i4|i3i2|i1i0 τ4|
�i3�i2|i1i0 τ4|τ3
�i2�|i1i0 τ4|τ3τ2|
�i1i0 τ4|τ3τ2|τ1
�i0
(Initial Map) ⇐==⇒ ⇐==⇒
i4i3|i2i1|i0�i4i3|i2i1|i0 τ4
�i3|i2i1|i0 τ4τ3|
�i2�i1|i0 τ4τ3|τ2
�i1�|i0 τ4τ3|τ2τ1|
�i0
(Initial Map) ⇐==⇒ ⇐==⇒
i4i3i2|i1i0|�i4i3i2|i1i0| τ4
�i3i2|i1i0| τ4τ3
�i2|i1i0| τ4τ3τ2|
�i1�i0| τ4τ3τ2|τ1
�i0�|
(Initial Map) ⇐==⇒ ⇐==⇒
Since the input sequence is in natural order, after in-place butterfly computation,the output is known to be in bit-reversed order. Therefore, the processor initiallyallocated xi4i3i2i1i0 will finally have Xi0i1i2i3i4 if inter-processor permutation is notallowed. In fact, Xi0i1i2i3i4 is the last derivative which overwrites xi4i3i2i1i0 in the samelocation.
Since the DITNR algorithm differs from the DIFNR only in the application of twiddlefactors, the shorthand notations given above also represent the DITNR algorithm.
19.2.2 Interpreting the data mapping for bit-reversed output
As a concrete example, suppose the initial mapping is “|i4i3|i2i1i0.” Then processorP0 initially contains a[0] = x0, a[1] = x1, a[2] = x2, · · · , a[7] = x7 in their naturalorder as depicted in Figure 19.1. When the parallel FFT ends, processor P0 containsa[0] = X0, a[1] = X16, a[2] = X8, a[3] = X24, a[4] = X4, a[5] = X20, a[6] = X12, a[7] =X28, which are the first eight elements in the output array in Figure 4.4. In this case,x and X are said to be comparably mapped to the processors [64, page 160].
Note that |i4i3| is a subsequence from the subscript of xm when m = i4i3i2i1i0, but|i4i3| is obviously not a subsequence of the subscript of Xr when r = i0i1i2i3i4. Thuswhen the data mapping is a CBM of naturally ordered input x, it is not a CBM withrespect to the subscript of the bit-reversed output data element Xr.
19.2.3 Parallel DIFRN and DITRN algorithms
The remaining two in-place sequential FFT variants deal with bit-reversed input data,and they are the DIFRN algorithm from Chapter 5 and the DITRN algorithm fromChapter 8. For the same example of length N = 32, the parallel algorithms corre-sponding to the four possible cyclic block mappings are represented below.
|i0i1|i2i3i4 |i0i1|i2i3�i4 |i0i1|i2
�i3τ4 |i0i1|
�i2τ3τ4 |i0
�i1�|τ2τ3τ4 |
�i0�τ1|τ2τ3τ4
(Initial Map) ⇐==⇒ ⇐==⇒
i0|i1i2|i3i4 i0|i1i2|i3�i4 i0|i1i2|
�i3τ4 i0|i1
�i2�|τ3τ4 i0|
�i1�τ2|τ3τ4
�i0|τ1τ2|τ3τ4
(Initial Map) ⇐==⇒ ⇐==⇒
i0i1|i2i3|i4 i0i1|i2i3|�i4 i0i1|i2
�i3�|τ4 i0i1|
�i2�τ3|τ4 i0
�i1|τ2τ3|τ4
�i0τ1|τ2τ3|τ4
(Initial Map) ⇐==⇒ ⇐==⇒
i0i1i2|i3i4| i0i1i2|i3�i4�| i0i1i2|
�i3�τ4| i0i1
�i2|τ3τ4| i0
�i1τ2|τ3τ4|
�i0τ1τ2|τ3τ4|
(Initial Map) ⇐==⇒ ⇐==⇒
19.2.4 Interpreting the data mapping for bit-reversed input
Observe that the mapping | i0i1 | i2i3i4 maps xi4i3i2i1i0 to processor Pi0i1 . That is,processor P0 will be allocated a[0] = x0, a[1] = x16, a[2] = x8, a[3] = x24, a[4] = x4,a[5] = x20, a[6] = x12, a[7] = x28.
Thus, while the initial mapping is a cyclic block mapping with respect to the arrayaddress r = i0i1i2i3i4, it is obviously not a CBM mapping with respect to the subscriptof the bit-reversed input data element xm.
However, since a[r] = Xr on output and r = i0i1i2i3i4, the mapping will be a CBMmapping with respect to the subscript of naturally ordered output data element Xr.
As noted above, butterfly computations will cause communication between processorsif the two input elements are stored in different processors. Since both input elementsare needed to update each of them, the two processors involved must exchange the N
P
data elements for each other to update their local data.The butterflies in any one of the parallel FFTs introduced in the previous section
require data to be exchanged in exactly d = log2 P = 2 stages, regardless of theblocksize used in the mapping. Algorithms of this type are described in [23, 46, 59].This is also version 1 of the distributed-memory FFTs in [64, pages 156–162].
19.4 Uneven Distribution of Arithmetic Workload
A possible consequence of this class of schemes is that one half of the processors updatetheir local data according to a formula not involving the twiddle factor. For example,in the parallel DIFNR algorithm, one half of the processors each update N
P elementsaccording to
y� = (x� + x�+N/2) ,
while the other half of the processors update their local data according to a formulainvolving the multiplication of a pre-computed twiddle factor, i.e., they each update NPelements according to
z� = (x� − x�+N/2)ω�N .
Thus, the arithmetic workload is not evenly divided among all processors unless eachprocessor computes both y� and z�. This problem is addressed in the next chapter.
