Abstract—Contemporary field programmable gate arrays(FPGAs) combine the fine-grained design capability of thetraditional lookup table with the speed of medium-scale andlarge-scale logic components such as RAM blocks or DSPblocks to provide for significant computational capabilityfrom a single FPGA. High performance reconfigurablecomputers, which typically use FPGAs as computationalelements, have been commercially used to accelerate compu-tational kernels. However, the deep pipelines and extensiveparallelism needed for FPGAs to compete with GHz-scalegeneral purpose processors make mapping of floating-pointkernels a challenging research area. In this paper, wedescribe some of the progress that has been made towardssolving some of these mapping challenges.
I. INTRODUCTION
Modern field programmable gate arrays (FPGAs) havegone beyond the traditional lookup table (LUT) plusrouting model that characterized their early counterparts.FPGAs now incorporate medium-scale and large-scalelogic components such as RAM blocks, DSP blocks, shiftregisters, on-chip clock controllers, and high-speed I/Oblocks within the programmable FPGA interconnectionfabric. This combination provides for the fine-grained,gate-level (albeit slow) design capability associated withLUTs, yet allows for the use of the much faster fixedcomponents within a design. One of the results of thismodern architecture is the ability to provide for signifi-cant computational capability from a single FPGA. Highperformance reconfigurable computers (HPRCs), whichtypically use FPGAs as computational elements, havebeen commercially used to accelerate kernels for bothembedded and traditional applications [1], [2]. However,the deep pipelines and extensive parallelism needed forFPGAs (which run at a few hundred MHz) to competewith general purpose processors (which run at a few GHz)make mapping of floating-point kernels a challengingresearch area. In this article, we describe some of theprogress that has been made towards solving some ofthese mapping challenges. The article is organized asfollows. Section II provides background information onHPRCs including a discussion of the high-level language(HLL)-based development flow and the specific platformused in this research. Section III addresses some of
the heuristics that allow developers to determine whichkernels are good candidates for mapping onto an FPGA.Section IV gives a specific example of how to map hard-ware description language (HDL)-based components intoan HLL-based development flow. Section V illustrates theuse of these ideas to map a simple floating-point Jacobiiterative solver onto an HPRC, and Section VI presentsthe conclusions.
II. BACKGROUND
A. High performance reconfigurable computers
The reconfigurable computer (RC) was introduced inthe 1960s by Estrin [3] as a “fixed plus variable structurecomputer.” However, technological limitations such ashand placement and wiring of components hamperedresearch progress. Freeman’s invention of the FPGA [4]in the 1980s generated renewed interest in RC-basedresearch and development. Perhaps the earliest example ofa commercially available RC was the Algotronix CHS2x4,which was featured in the international version of BYTEmagazine [5]. While not a commercial success, it didset the stage for future efforts. A number of modernHPRCs that combine general purpose processors (GPPs)and FPGAs as the “fixed plus variable structure” are nowavailable. Maxeler Technologies, for example, offers theMAX3 dataflow compute card [6], which was used by JPMorgan to reduce its risk analysis run time from 8 hoursto about 4 minutes [2]. SRC Computers [7] offers theMAP processor, which was used by Lockheed Martin inthe Synthetic-Aperture Radar (SAR) unit that flies aboardthe U.S. Army’s MQ-9 Unmanned Air Vehicle (UAV) [8].Mercury Computer Systems offers FPGA-based computeboards such as the Ensemble MXI-205 [9], which is usedin several of their application ready subsystems [10].
B. HLL development flow
As noted above, RCs have been successfully used tospeed up applications. However, acceleration of floating-point applications is still challenging. There are a hostof contributing issues; the loop-carried dependence asso-ciated with pipelined floating-point functional units, forexample, makes it difficult to fully pipeline floating-point
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 859
kernels such as sparse matrix vector multiply. There havebeen some successes in these application areas [11], [12].However, mapping kernels onto RCs is still primarilyan art form, which relies upon the skill and experienceof the developer to craft a customized solution on acase-by-case basis. By way of example, the JP Morganeffort motioned earlier involved the mapping of only twokernels, yet took about 3 years. Mainstream computerusers simply can not tolerate such a lengthy developmentcycle. If HPRCs are to be a part of mainstream computing,development environments must move away from HDL-based hardware design toward HLL-based programming.
Several companies including Mentor Graphics andSRC Computers [13], [14] have introduced HLL-to-HDL compilers, which allow scientists and engineersto program HPRCs using HLLs (as with traditionalcomputers) rather than employing HDL-based hardwaredesigns (as with traditional FPGA-based circuits). Itshould be noted that these HLL-to-HDL compilers are nota panacea, especially for floating-point applications. Oneof the authors’ attempts to speed up a simple sparse matrixkernel using an earlier generation HLL-to-HDL compilerresulted in a 10-fold slowdown. Even integer kernelscan be problematic; Park’s attempt to accelerate theBlowfish kernel showed over a 40-fold slowdown whenimplemented using the DIME-C HLL-to-HDL compiler[15]. To make MHz-scale FPGAs competitive with GHz-scale GPPs, we must have deeply pipelined, highly par-allelized FPGA-based designs. Therefore, modern HLL-to-HDL compilers include enhanced HLL features suchas pipelined loops, parallel code blocks, communica-tion channels, synchronization primitives, and intellec-tual property interfaces (IPI) to access vendor-suppliedor (when necessary) user-supplied intellectual property(IP) cores. Note that the IPI allow the IP cores to be“called” as though they were standard parameterized HLLsubprograms. Section IV gives a specific example on howto map HDL-based user IP components into an HLL-based development flow.
Fig. 1 illustrates the HLL development flow. Thedesign is partitioned into software modules, which aretargeted for execution on the GPP, and hardware modules,which are targeted for execution on the FPGA. Duringdevelopment, the HLL software modules are compiledwith a standard HLL compiler to produce object files.The linker uses the object files, library files, hardwaremodule call specification, and the FPGA configurationbitstream (treated as data by the linker) to producea binary executable. The HLL hardware modules areingested by the HLL-to-HDL compiler, which emits HDL.This HDL output is used by the standard FPGA toolchain to produce an FPGA configuration bitstream, whichwhen loaded onto the FPGA programs the FPGA with ahardware implementation of the design specified in theHLL code. From the viewpoint of the software moduleHLL code, the hardware module looks like a simpleparameterized subroutine call. Note that the hardwaremodule developer provides an API, e.g., header file, that
HLL-to-HDL
binary
standardFPGA
tool chainlinker
compiler
lib
API
obj
GPP
HDL
FPGA
software modules hardware modules
enhancedHLL
standardHLL
IP
IPI
loader
runtime
bitstream
“data”“code”
bitstream
liblib
APIAPI
APIAPIAPI
objobj
Fig. 1. HLL development flow
describes the call specification. From the viewpoint ofthe hardware module HLL code, the IP cores also looklike subroutine calls. In this case, the vendor-suppliedor user-supplied IPI describes the call specification. Atthe beginning of execution, the configuration bitstreamis extracted from the binary executable data segmentand loaded onto the FPGA. The GPP then executes themachine code instructions and invokes the FPGA-basedkernel as needed via a run time support capability.
C. Description of target HPRC
There are a number of different RC architectures. Oneof the more popular architectures, as suggested by Fig. 2and alluded to in Section II-B, is when the FPGA-based processing element (PE) essentially functions as areconfigurable coprocessor that is invoked by a traditionalGPP-based PE. In this model, there is some form ofhigh-speed interconnect between the GPP-based fixed PEand the FPGA-based variable structure PE. It should benoted that other RC node architectures are extant, forexample, FPGA as peer PE, FPGA as as sole PE, etc.In a typical application based on the architectural modelshown in Fig. 2, the GPP would marshal a data set intothe global common memory of the variable structure PE,and then call the FPGA-based kernel. The FPGA kernelwould copy the data into the multibank local memory toallow for multiple simultaneous accesses. In the SRC-7MAPstation-based [7] system used in our research, forexample, there are sixteen local memory banks, whichallow the FPGA to simultaneously fetch up to sixteen64-bit words per FPGA clock cycle. The simultaneousmemory access is required to support the parallelismneeded to achieve a speedup on an FPGA.
860 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
The HPRC used in this research is a cluster of SRC-7RC compute nodes [16] with a traditional login/controlnode, as idealized in Fig. 3. According to McGrath [17],the Jackson State University SRC-7 cluster is describedas “the world’s first InfiniBand-based SRC-7 MAPstationcluster.” In Fig. 3, each RC compute node is as idealizedby Fig. 2. Note that all InfiniBand MPI communicationis handled by the Xeons; the variable structure MAPprocessors (which contain the FPGAs) serve strictly asreconfigurable coprocessors to accelerate selected kernelswithin the portion of the application that is running onthe given compute node. The first Ethernet port on thelogin node is used to connect to the outside world overssh. The second Ethernet port is used as a local (private)interconnect between the login node and all the computenodes. This allows our researchers to ssh into a computenode if necessary. In addition, we use an NFS-mountedfile system (over Ethernet) to export the user’s RAID-protected home directory on the login node to all thecompute nodes. In a production system, one would mostlikely use a parallel file system such as pNFS [18].
Each SRC-7 RC node has two 3.0GHz Intel Xeonprocessors with a 16K L1 cache, 2MB L2 cache, and6GB RAM. The MAP Series H reconfigurable processorcontains two Altera EP2S180F1508C3 FPGAs running at150MHz. There are sixteen 64-bit-wide banks of localon-board memory (OBM) associated with the FPGAs,which provide up to 64MB of local memory; and thereare two 64-bit banks of global common memory (GCM),which provide 2GB of memory that can be read to andwritten from by both the Xeons and the FPGAs. As notedearlier, the GCM is used primarily as a store whereinthe Xeons and FPGAs marshal and retrieve large datasets. The MAP H processor is connected to the Xeonmotherboard through a SNAP D interface, which has an8GB/s bandwidth. To maximize performance, the SRC-7 uses a memory-based mapping rather than an I/O-based mapping. Direct memory access (DMA) is used tomove data between the Xeon memory and MAP memory.In addition, the MAP processor has a streaming DMAcapability, and an inter/intra-FPGA streaming capabilitythat allows overlapping communication and computation
localmemory banks
variable structureprocessing element
FPGA
RAM
fixedprocessing element
GPPglobal
commonmemory
NIC ...
KVM
Fig. 2. RC node architecture
• • •
ssh, NFS
MPI
ssh
login
PCI-X
dual quad-core Xeon
8GB RAM
eth IBeth
compute
MAP H
PCI-Xdual Xeon6GB RAMSNAP D
eth IBeth
compute
MAP H
PCI-Xdual Xeon6GB RAMSNAP D
eth IBeth
10GigE
SDR-IB
LAN, WAN
RA
ID-1
KVM
Fig. 3. HPRC cluster
to facilitate parallelism within the MAP processor.SRC’s Carte v4.0 programming environment is tightly
integrated with their HPRC hardware. Carte automaticallyhandles interchip communication, I/O pin mapping, andother necessary (but uninteresting) hardware details. Thisfrees the developer to concentrate on mapping the al-gorithm onto hardware. Carte also directly supports theHLL development flow using either C or FORTRAN. Inour research, we have elected to use Carte C. For thesoftware-only components, the Intel C compiler v11.1was used. Mapping of algorithms onto this novel SRC-7 HPRC system was successfully demonstrated in [19],[20], [21], [22], [23], [24], [25].
III. DESIGN CONSIDERATIONS
This section takes a detailed look at the three p’s, whichhighlights the critical relationship among performance,pipelining, and parallelism. It then examines the FPGAdesign boundary, which addresses some of the heuristicsallowing developers to select candidate kernels that canbe mapped onto the FPGAs.
A. The three p’s
FPGA clock rates are in the 100s of MHz range,whereas GPP clock rates are on a GHz scale. Given thisorder-of-magnitude advantage, something must be done atthe design level for the FPGA to compete with the GPP.As suggested by Fig. 4, the performance of an algorithmon an FPGA is proportional to the extent to which it ispipelined and parallelized.
There are five simultaneous datapaths producing sixoutputs. Each of the datapaths is also pipelined to allowmultiple operations within a data path to overlap. Thismultiplicative effect, which is known as the three p’s,expresses the important relationship among performance,pipelining, and parallelism. Failure to either pipeline orparallelize a kernel generally results in poor performance
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 861
3: a2 ← 0.5/a // break into atomic ops4: bb ← b · b // to be parallelized5: ac ← a · c // and pipelined6: mb ← −b // by HLL-HDL compiler7: ac4 ← 4 · ac8: D ← bb− ac49: sqr ← √
on an RC. We can pipeline and parallelize the quadraticformula [25] as depicted in Fig. 6, where αm, αa, αs, andαd represent the latencies of the floating-point multiplier,adder, square rooter, and divider IP, respectively. A hugeadvantage of a modern HLL-to-HDL compiler is thatit builds a data-flow graph, determines the true datadependences, and automatically calculates delays such
that all the parallel paths are of equal length. Thedeveloper simply has to “give the compiler a chance” tofind the parallelism, e.g., as depicted by the algorithmsnippet in Fig. 5. Ideally, several of these fully pipelineddatapaths should be used in parallel to implement theFPGA module.
Other techniques associated with parallelizing andpipelining a kernel include performing parallel I/Oand overlapping communication with computation.Obviously, the best scenario is when we can crunchthe data as they flow by, i.e., when we can create asystolic array. At the other end of the spectrum are thosecases, such as an iterative solver, when we must firstbring in the data set, then do the computation, and thendo the output. The in-between cases include partiallyor completely overlapping computation with input oroutput.
B. FPGA design boundary
Determining the FPGA design boundary, i.e., determin-ing which application modules should be mapped ontoFPGAs, is not straightforward. As with many engineer-ing disciplines, we must also rely on heuristics derivedfrom empirical observation. Some areas to be consideredwhen determining the FPGA design boundary include,1) the three p’s, 2) expected overall speedup, 3) expectedresource utilization, 4) control/memory intensive vs. com-pute intensive, 5) monolithicity of modules, 6) availablebandwidth, 7) opportunities for data reuse, 8) algorithmdesign stability, 9) algorithm efficiency, and 10) memoryaccess patterns, as follows:
The three p’s – Perhaps the most important heuristic isthe three p’s previously described. If a module cannot bepipelined and parallelized, then it is unlikely to achievehigh performance when mapped onto an FPGA. Even ifa module is three p’s compliant, it still needs to haveenough data to keep the pipelines filled, i.e., to amortizepipeline latency across multiple problems.
Expected overall speedup – If the objective of mappinga kernel onto an RC is to obtain a speedup relativeto the performance of a GPP, then we need to look atthe expected overall speedup. Overall speedup can bequantified via Amdahl’s Law [26]
so =1
1− fe + fe/se
where so is the overall speedup, fe is the fraction ofthe system to be enhanced, and se is the speedup of theportion to be enhanced. Amdahl’s Law is often the basisfor design decisions and can help us avoid costly designmistakes based on deceptive intuition. For example, sup-pose the estimated speedup for the postprocessing systemwere a thousandfold, i.e., an FPGA implementation ofthe postprocessing kernel were estimated to run an in-credible 1000 times faster than an equivalent softwaremodule. Intuition says the FPGA-based postprocessingkernel would yield a significant overall speedup. Suppose
862 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
postprocessing only constitutes 10 percent of the runtime.After applying Amdahl’s Law,
so =1
1− 0.10 + 0.10/1000= 1.11,
we see that the overall speedup associated with the FPGA-based kernel is only about 10 percent. This relationshipbetween overall speedup and the fraction of a systemthat can take advantage of the speedup is depicted inFig. 7. The abscissa represents the fraction of the systemto be enhanced (fe), and the ordinate represents overallspeedup (normalized to the speedup of the portion tobe enhanced, i.e., so/se). The plotted lines representthe normalized speedup values for five scenarios, se ∈(2, 5, 10, 100, 1000). The vertical lines are the actualoverall speedup for fe ∈ (0.5, 0.75, 0.9, 0.95). Notice thatfor fe = 50 percent, the overall speedup values are only2.0 for se = 100 and se = 1000. If 75 percent of thesystem could take advantage of a thousandfold speedup,the overall speedup value will only be 4.0. Clearly, theuse of Amdahl’s Law in determining the FPGA designboundary is important.
Expected resource utilization – Another importantFPGA design boundary consideration is the expectedresource utilization of the candidate module. Sincefloating-point IP cores can be quite large, the developerneeds to determine if the candidate will even fit on theFPGA. The developer also needs to consider the neededlocal memory capacity, number of simultaneous memoryaccesses, anticipated clock rate in the light of complexrouting, etc.
Control/memory vs. compute intensive – It isalso important to consider whether an algorithm iscontrol/memory intensive or compute intensive. Inhardware, control flow is implemented as multiplehardware paths with a mux at the end. Therefore, akernel with many control clauses is likely to resultin large (and slow) hardware. Harkins et al. illustratethe importance of this concept when they show thatcomparison-sorting algorithms do not perform well onan HPRC [27].
Monolithicity of modules – If the candidate FPGAmodule contains procedure calls, they have to be inlined
or the module cannot be considered as a viable candidate.Obviously, this will be impacted by the available FPGAresources.
Available bandwidth – The GPP to FPGA bandwidthalso deserves attention. Obviously, the FPGA memoryaccess and processing time should be less than theGPP memory access and processing time. Accordingto Herbordt et al., when they discuss latency hiding, adesign should try to overlap computation with communi-cation [28]. This might minimize the effects of bandwidthlimitations. A closely related issue is data reuse, to bediscussed next.
Opportunities for data reuse – Algorithms that have asignificant potential for data reuse may be suitable FPGAmodule candidates. We used this principle to speed uptwo well-known iterative solvers [29]. This is similar tomethods used by the GPP where frequently used dataare stored in nearby memory such as general-purposeregisters or cache.
Algorithm design stability – Since mapping an al-gorithm to an FPGA is not the easiest of tasks, it isimperative to make sure that the algorithm is as stableas possible. If the algorithm is altered while in the midstof a hardware implementation process, we might discoverthat the new algorithm no longer fits onto the FPGA, orthat it can no longer deliver on the promised speedup.
Algorithm efficiency – Another application design con-sideration is to make sure an efficient algorithm isemployed. For example, Cramer’s rule, which has expo-nential complexity, O(n!) (Habgood and Arel notwith-standing [30]), might run faster if implemented on anFPGA. However, Gaussian elimination, with complexity,O(n3), is a much more efficient algorithm. We shouldfirst try a more efficient software solution rather than mapinefficient algorithms onto an FPGA.
Memory access patterns – If a candidate kernel haslarge or irregular stride memory access patterns, then itis more likely to do well on an FPGA, which does notdepend upon the memory access patterns like a cache-based GPP. See [24] for a more detailed discussion ofthis issue.
IV. INTEGRATING HDL-BASED COMPONENTS
A. Standard integration approach
In some cases, it is desirable to use HDL-based IP coresin an RC design, e.g., when the enhanced HLL does notprovide the needed capability. The standard way in whichCarte integrates IP (Carte refers to these IP cores as usermacros) is by 1) specifying a header file to be included bythe hardware module code, and 2) specifying a blackboxfile and an info file via Carte environment variables. Wehave coined the phrase intellectual property interface (IPI)to describe the integration mechanism. As suggested byFig. 8, the blackbox is the HDL interface to the IP; theinfo file contains several properties of the IP including aconnection between the IP core name and the HLL name;and the header file is the HLL interface. Note that a user
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 863
macro can also be a synthesized IP, i.e., netlist only (noHDL).
The default Carte mechanism for incorporating usermacros does not directly support a multiple file, multipledirectory VHDL hierarchy. The following sections givean example of integrating such hierarchical componentsinto the Carte environment. We will use an HDL-baseddot product user macro. Note that Carte can deal with thegeneration of a dot product unit at the HLL level; we useit as example because it is an easy kernel to understand,yet can be built in the hierarchical fashion typical of anHDL-based design.
B. Primitive floating-point components
The Quartus [31] Mega-wizard tool allowed us to createRTL VHDL codes using a simple graphical user interface(GUI). We used the tool to generate pipelined, IEEE 32-bit floating-point adder and multiplier components enti-tled fpadd32.vhd and fpmul32.vhd, respectively.After testing the designs using ModelSim [32], theseprimitive components were used to build a pipelined, 32-bit floating-point, 8-input structural VHDL dot productcomponent entitled fpdot32x8.vhd.
C. Dot product component
Fig.9(a) is an elided representation of the directoryhierarchy for the VHDL-based floating-point dot product.Associated with the adder (for example) are the Model-Sim VHDL compiler library directory work/, Quartusproject file fpadd32.qpf, design file fpadd32.vhd,test bench file fpadd32_tbv.vhd, ModelSim simu-lation scripts sim.do and wave.do, support package../packs/sizePack.vhd, and several other files.An analogous substructure exists for the multiplier and dotproduct unit. Fig. 9(b) illustrates the 51-stage pipelineddot product binary tree. Since fpdot32x8.vhd waswritten using a structural VHDL approach, it incorpo-rated the fpadd32 and fpmul32 components. The6-cycle first stage used the multiplier. The 15-cyclesecond, third, and output stage used the adder. A keymechanism in the creation of fpdot32x8.vhd wasthe use of generate loops and the associated arrays
- fpdot/
- fpadd32/
+ work/
fpadd32.qpf
fpadd32.vhd
fpadd32_tbv.vhd
sim.do
wave.do
...
+ fpdot32x8/
+ fpmul32/
- packs/
sizepack.vhd
...
(a) directory hierarchyx0
x1
x2
x3
y0
y1
y2
y3
×+
×
×
×
+
+
x4
x5
x6
x7
y4
y5
y6
y7
×+
×
×
×
+
+
+ ∑=
8
1iii yx
6 cycles
15 cycles
(b) binary tree pipeline
Fig. 9. Dot product
of standard logic vectors. We only had to write 164lines of VHDL code, since the Quartus Mega-wizardplug-in produced 6K or more lines of highly tunedRTL VHDL code. The architectural hierarchy of thisdesign requires sizePack.vhd to be compiled beforefpmul32.vhd and fpadd32.vhd. These latter twohave to be compiled before fpdot32x8.vhd.
D. Dot product intellectual property interface
The directory structure of the entire RC-based dotproduct is shown in Fig. 10. We will only mention someof the files in passing, since the purpose here is to describethe IPI. The software module that is executed on the
864 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
general purpose processor is in main.c. It generatessome vectors and calls the hardware module code thatis represented in the dot.mc file. The dot.h file isthe user-generated API that describes the user interfacefor the hardware module. The Makefile, which wewill discuss later, orchestrates the system build. Thevhdl.libs will also be described later. The fpdot/directory is as discussed earlier with two importantadditions, fpdotbb.v and fpdot.info. These twofiles and fpdot.h constitute the IPI described in thenext section.
1) Header, blackbox and info files: The header file,fpdot.h, does not require much elaboration; it simplycontains the prototype of the C interface used by thedot.mc hardware module code when it calls the dotproduct IP. Of more interest are the blackbox and infofile used to tell the Carte compiler about the VHDL-basedIP. The first file, fpdotbb.v, is a Verilog blackboxfile that describes the interface for the dot product andassociated IP. In essence, the blackbox file tells Cartehow to deal with the user macros when it begins to syn-thesize the HDL emitted by the HLL-to-HDL compiler.The blackbox entry for the fpadd32.vhd file wouldbe as shown in Fig. 11. Analogous entries would becreated for fpmul32 and fpdot32x8. The second file,fpdot.info, is the info file that contains informationabout the user macro that is needed by the Carte HLL-
void myfpAdd__dbg(float x,float y,float *z) {// functional equivalent// of VHDL code behavior*z = x + y;
}#;
END_DEF
Fig. 12. Info entry for fpadd32
to-HDL compiler. This includes such things as latency,name of netlist file, name to be used in the Carte C code,etc. The info file also contains debug code that provides asoftware-only functional equivalent of the IP that is usedby Carte when creating a debug build. The debug buildis a software-only equivalent of the hardware moduledesigned to test the functionality without requiring alengthy synthesis, place and route, and bit generationcycle. The info file entry for the fpadd32.vhd file,for example, would be as shown in Fig. 12. Analogousentries would be created for fpmul32 and fpdot32x8.
2) Makefile modifications: The Carte development en-vironment includes a Makefile that is tailored for eachnew development. Without going into too much detail,the Makefile is different than what we normally seewith a software build. Essentially, a Carte Makefiledefines a set of environment variables and then firesoff the Carte compiler to do the work. A default CarteMakefile assumes the user macro is a single VHDLfile contained in a single directory. As noted above, thisis usually not the case. Therefore, we had to do someresearch to figure out how to coax Carte into dealing witha multiple-file, multiple-directory scenario. Our modifiedMakefile is as shown in Fig. 13. We have includedline numbers to facilitate the discussion that follows.The first part of the puzzle is to tell Carte the nameof the VHDL library that will contain the compiled
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 865
01: FILES = main.c02: MAPFILES = dot.mc03: BIN = dot04:05: # location of blackbox and info files06: MY_BLKBOX = fpdot/fpdotbb.v07: MY_INFO = fpdot/fpdot.info08:09: # tell Carte about the default VHDL library11: # and directories containing the netlists12: MCCFLAGS += -param_file $(PWD)/vhdl.libs \13: -ngo_dir ../fpdot/packs \14: -ngo_dir ../fpdot/fpadd32 \15: -ngo_dir ../fpdot/fpmul32 \16: -ngo_dir ../fpdot/fpdot32x817:18: # from here on it is just like normal19: CC = icc20: LD = icc21: CFLAGS = -O3 # optimize the C code22:23: # No modifications are required below24: MAKIN ?= $(MC_ROOT)/opt/srcci/comp/lib/AppRules.make25: include $(MAKIN)
netlists. To accomplish this task, we create an auxiliaryfile called (in this case) vhdl.libs and send it to theCarte compiler via the -param_file flag, as shownon line 12 of the Makefile. As shown in Fig. 14,vhdl.libs simply lists each VHDL file and associatesit with a VHDL library. During the early stages of asystem build, the Carte compiler invokes the Quartussynthesizer to compile the VHDL code in the fpdot/directory. At that time, the Carte compiler’s defaultdirectory is the same as the directory from which theMakefile was invoked. Therefore, the full path nameof the vhdl.libs file is just $(PWD)/vhdl.libs,where PWD is the environment variable containing thepresent working directory. The second part of the puzzleis to tell Carte where the compiled netlists are located (theUnix directory, not the VHDL library). For a single netlist,this is usually accomplished via the predefined Cartevariable called MY_NGO_DIR. Since Carte interprets thisenvironment variable as a single directory name, wecannot use it to pass along multiple directories. Therefore,we use multiple instances of the Carte -ngo_dir flag,one for each netlist directory. The complication is that bythe time Carte references the MCCFLAGS environmentvariable to extract the netlist directories, Carte is operatingin a different (new) subdirectory it has created underdot/. Therefore, we prepend a relative path referenceto obtain the full path name of each netlist directory, asshown on lines 13 – 16 of the Makefile.
After creating the IPI infrastructure, we simply have todo a make, and the Carte compiler produces the desiredexecutable. See [22] and [21] for additional details on thisprocess.
V. MAPPING AN ITERATIVE SOLVER ONTO AN HPRC
To illustrate the use of some of our ideas, we willshow the mapping of a simple floating-point sparse matrixJacobi iterative solver onto an HPRC.
A. Derivation of the Jacobi method
Discussions of the Jacobi iterative method can be foundin introductory numerical analysis textbooks such as [33]or [34]. For our simple solver, we assume real-valuedmatrices and vectors. Let A be an n×n strictly diagonallydominant matrix; let x be the unknown n-vector; and letb be the constant n-vector. To solve Ax = b iteratively,the next approximate solution, x(δ+1), is computed as afunction of the current approximation, x(δ), where δ isthe iteration index. Substituting A = L + U + D intoAx = b, where L is the strictly lower triangular partof A, U is the strictly upper triangular part of A, andD is the diagonal, and manipulating it into the standarditerative form x(δ+1) = f(x(δ)) results in
x(δ+1) = D−1(b− (L + U)x(δ)
), (1)
866 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
Combining residual vector, r(δ) = A(x−x(δ)) with Ax =b produces
r(δ) = b−Ax(δ). (2)
Substitute A = L + U + D into Equation 2 to obtain
r(δ) + Dx(δ) = b− (L + U)x(δ). (3)
Combining Equation 1 and Equation 3 yields
x(δ+1) = D−1r(δ) + x(δ). (4)
The Jacobi iterative solver first calculates Equation 2 andthen uses the result to calculate Equation 4. This approachyields the residual vector needed for the termination test.A common termination test, often referred to as residualnorm, is shown in Equation 5
‖r(δ)‖‖b‖ < ε, (5)
where ‖·‖ ≡ ‖·‖2 is the 2-norm, e.g., ‖x‖ =√∑
x2i ,
and ε is some suitably small value. This is the terminationcriteria used by our Jacobi iterative solver.
B. Compressed sparse row format
If a matrix of order n has a small number of nonzerovalues, nz , then compressed sparse row (CSR) format [35]can reduce both storage and computational requirements.CSR uses vectors val, col, and ptr to store only thenonzero values and identify the row and column indices.Vector val is a real vector of length nz containing thenonzero values obtained via a row-wise matrix traversal.The col integer vector is also of length nz and containsthe column index of each nonzero value. The ptr integervector is of length n + 1 and contains the index inval where each matrix row starts, i.e., the first nonzeroelement of matrix row i is found at index ptri of val.To facilitate consistent usage, we let ptrn+1 = nz + 1.For example, the order n = 4 sparse matrix with nz = 8nonzero values shown in dense format in Fig. 15(a) isshown in CSR format in Fig. 15(b). Consider ptr3 = 5;this indicates that row 3 of the matrix begins at index 5of val and col. Notice that val5 = a33 and that col5 = 3as required.
To parallelize our Jacobi processor, k = 8 valuesare read from val and col during each clock cycle.This is accomplished by striping these vectors acrossmultiple memory banks as alluded to in Section II-C. Toavoid large multiplexers and the associated degradationin performance discussed in Section III-B, matrix rowsare padded with zeros (as needed) to ensure they have anexact multiple of k values. The ptr vector is modified toexpress indices in terms of k-sized groups of data, and theterm knz (analogous to nz) is the number of k-groups.This entire process, which is carried out by the softwaremodule when it marshals the data for the FPGA-basedprocessor, is known as “k-alignment.” This k-alignmentprocess produces scalar knz and vectors kval, kcol, andkptr. As before, kptrn+1 ≡ knz + 1. If we assume thatk = 2, then the sparse matrix represented in Fig. 15(b)could be represented in k-aligned CSR format as shownin Fig. 16. Each bracketed k-group of data representsthe contents at a given index across a striped data set.Consider kptr4 = 5; this indicates that matrix row 4begins at index 5 of striped memory banks kval andkcol. Notice kval5 = [a42, a44] (2 banks, 2 values) andthat kcol5 = [2, 4] (2 banks, 2 indices). Also notice thatwe simply set the column index of the padded 0 valuesto 1 since it does not really matter anyway
D. Sparse matrix Jacobi algorithm
An algorithm for the Jacobi iterative method is shownin Fig. 17; it deals with CSR format matrices.
1: algorithm SWJACOBI( val, col,ptr,b,x(δ), δ)2: δ ← 03: repeat4: for i in [1, n] do5: r
(δ)i ← bi
6: for j in [ptri, ptri+1) do7: r
(δ)i ← r
(δ)i − valj · x(δ)
colj8: if colj .EQUATION i then aii ← valj9: end for
10: x(δ+1)i ← r
(δ)i /aii + x
(δ)i
11: end for12: x(δ) ← x(δ+1)
13: δ ← δ + 114: until ‖r(δ)‖/‖b‖ < ε .OR. δ > δmax15: end algorithm
Fig. 17. Sparse matrix Jacobi algorithm
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 867
One of the features of the latest Carte compiler is astreaming accumulator, which can be used to overcomethe loop-carried dependence associated with variable-length inner loops like that of the sparse Jacobi methodshown in Fig. 17. In previous research efforts, we wereforced to build our own HDL-based streaming accumula-tor [36], so we were quite pleased when the new compilerprovided this capability. In some sense, this justifies oneof the goals of our research, i.e., to get good ideas to beadopted by hardware and compiler vendors. At any rate,the accumulator accepts two input FIFO streams, valuesand counts, and emits an output stream, sums. Therelationship between these three streams is perhaps bestunderstood by way of a simple example. In Fig. 18, thevalues stream represents the three vectors, v1 = [6, 9, 4],v2 = [3, 5, 1, 7], and v3 = [8, 2]. The three valuesin the counts stream indicate the number of values ineach vector, i.e., |v1| = 3, |v2| = 4, and |v3| = 2.The streaming accumulator uses values and counts tocompute the three vector sums, s1 =
∑v1i = 19,
s2 =∑
v2i = 16, and s3 =∑
v3i = 10 and send themto the sums FIFO stream. The streaming accumulatorgenerally requires four parallel sections as suggested bythe pseudo-code snippet shown in Fig. 19. In the Jacobiprocessor, which we will see momentarily, the valuesstream corresponds to the series of partial dot productsemitted, one per clock cycle, by a fully pipelined 8 · 8dot product unit. The counts stream corresponds to thenumber of partial dot products per matrix row. The sumsstream contains the full dot products,
∑aij ·x(δ)
j , neededto compute the residual vector shown in Equation 2.
algorithm WHATEVER(· · · )· · ·parBegin
p1: // feed values stream· · ·VFIFO ← · · ·
p2: // feed counts stream· · ·CFIFO ← · · ·
p3: // streaming accumulatorSFIFO ←
∑STREAM(VFIFO, CFIFO)
p4: // consume sums stream· · · ← SFIFO· · ·
parEnd· · ·
end algorithmFig. 19. Streaming accumulator usage
�1 ... �
mA1 ... Am
b1 ... bm x1 ... xm
main
coocsrmmio
Jacobi Jacobi?software FPGA
compile-time decision
Matrix Market SPARSKIT
Fig. 20. High-level Jacobi design
F. High-level Jacobi design
The high-level design for the Jacobi iterative solver isshown in Fig. 20. It consists of four major components:a main routine and matrix support libraries; severalstrictly diagonally dominant sparse matrices, A1 . . . Am;the software or hardware (FPGA-based) Jacobi iterativesolver; and the output result and statistics files, x1 . . .xm
and Θ1 . . . Θm. The bi vectors are shown as inputs, butfor the experiments in this research they are generatedfrom a known x vector at run time. The main routineis a driver program, which essentially measures howlong it takes for Jacobi to solve each set of equations.The coordinate-format matrices are read in using theMatrix Market I/O library [37] and converted to CSRformat using Saad’s SPARSKIT library [35]. The softwareJacobi kernel implementation is based on the algorithmshown in Fig. 17, and the FPGA-based Jacobi kernelwill be described later. A compile-time decision selectseither the software or FPGA-based version of Jacobi. Atrun time, main reads in each coordinate-format matrix,converts it to CSR format, and uses a known x vectorto generate b. It then invokes the selected Jacobi kernelsending matrix, A (val, col, and ptr), starting pointx(0), and constant vector b. After convergence, Jacobistores the result and returns. The main routine writes thesolution to the results file; it also writes the input matrixname, number of iterations, and wall clock execution timeto the statistics file and then terminates.
G. Jacobi design considerations
This section parallels Section III and shows how thespecific criteria were applied to the Jacobi design.
The three p’s – The Jacobi iterative method is inher-ently parallelizable. Given sufficient hardware, one coulddo all the dot products in parallel. In the actual design,we were able to fully pipeline all inner loops and toparallelize the dot product unit, i.e., to adhere to the threep’s.
Expected overall speedup – To determine what fractionof the run time was consumed by the Jacobi kernel,gprof was used to profile the software version of thecode. Not too surprisingly, given that the main routine isonly a driver, the Jacobi kernel consumed over 96 percent
868 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
of the run time for all data sets, i.e., fe = 0.96. Thus,assuming even a modest value se = 2, an overall speedupso = 1/(0.04 + 0.48) = 1.9 is anticipated.
Expected resource utilization – In the case of the Jacobisolver, the limiting factor was the amount of block RAM(BRAM) in the FPGA fabric, with the number of OBMbanks running a close second. Despite these limitation,we were able to build an 8-wide parallel Jacobi datapathcapable of handling sparse matrices up to order, n = 8K,with up to approximately 4M nonzero entries.
Control/memory vs. compute intensive – Our Jacobiprocessor employs design features that minimize thenumber of control clauses, e.g., the use of k-aligned CSRformat precludes the need for muxes at the dot producttree input. Furthermore, unlike a GPP memory hierarchy,the HPRC memory organization does not penalize irregu-lar memory accesses [24]. Therefore, on an FPGA, Jacobiappears to be primarily compute intensive.
Monolithicity of modules – In the case of the Jacobiprocessor, the Carte compiler provided IPI for some ofthe lower level routines such as square root. Therefore,the entire Jacobi kernel was effectively monolithic andsuitable for mapping onto an FPGA.
Available bandwidth – In the Jacobi processor, we wereable to reduce the bandwidth requirement by marshalingthe data into GCM banks and then using DMA to load theOBM and BRAM memory used during the FPGA-basedcomputation.
Opportunities for data reuse – In the case of the Jacobiiterative solver, the matrix A and vector b are reusedduring every iteration. This had the effect of amortizingthe transfer costs across all iterations.
Algorithm design stability – The Jacobi iterative solverhas been around for a long time; obviously we did notanticipate any algorithm design modifications.
Algorithm efficiency – In the case of the Jacobi method,we acknowledge that the efficiency rule is being violated.Jacobi is significantly slower than other solvers. However,our paper is not suggesting that Jacobi is the best way tosolve a set of equations nor is it trying to demonstrate thespeedup of a particular application. Our primary purposeis to illustrate the mapping process.
Memory access patterns – Since the Jacobi processoruses a sparse matrix, it clearly demonstrates an irregularmemory access pattern and is a good mapping candidate.
H. Parallelized Jacobi algorithmThe parallelized, pipelined algorithm for performing
the Jacobi iteration on HPRC hardware is shown inFig. 21. The algorithm operates in three phases: input,compute, and output.
1) Input phase: During input, three parallel blocksuse direct memory access (DMA) to input the problemdata from GCM. Lines 2–5 bring in the k-aligned CSR-format kval and kcol and store them, stripe-8, in theOBM banks. The striping of matrix values across multiplememory banks allows compute to read eight values perclock. Lines 6–9 and 10–13 bring in kptr, b, d (1/aii
19: for i in [1, knz] do20: a ← (a1 · · · a8) stripe-8 from kvali21: j ← (j1 · · · j8) stripe-8 from kcoli22: x ← (x1j1 · · ·x8j8)23: VFIFO ← dot8Tree (a,x)24: end for
p2: // feed counts stream25: for i in [1, n] do26: CFIFO ← kptri+1 − kptri
27: end forp3: // streaming accumulator
28: SFIFO ←∑
STREAM(VFIFO, CFIFO)p4: // residual & next approximation
29: for i in [1, n] do30: dotN ← SFIFO31: r ← bi − dotN32: x
2) Compute phase: Lines 16–39 constitute the com-pute phase. App. V-E provides details on Carte’s newstreaming accumulator. As mentioned, the need for anHLL-based streaming accumulator was identified by ear-lier research, e.g., [38], [36], [39], and that the compilervendor subsequently provided that capability. Since theFPGAs do not have multiport memory to support nineaddress and data buses on a single memory bank, andsince parallel sections p1 . . .p4 operate simultaneously,independent banks are needed to avoid a multicyclepipeline. Therefore, line 17 creates nine copies of x(δ);eight for the dot product tree, and one to calculate x
(δ+1)i .
Parallel section p1 (lines 19–24) is a fully pipelined8 · 8 dot product unit. Each clock cycle it consumes thenext eight aij values from kval and the matching eightvalues from x(δ), and outputs the resulting partial dotproducts (dot8s) to the VFIFO stream. Parallel sectionp2 (lines 25–27) calculates the number of dot8s for
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 869
each row and sends them to the CFIFO stream. Parallelsection p3 (line 29) is the streaming accumulator thatconsumes the VFIFO and CFIFO streams, computesthe n dot products, dotNi =
∑j aijx
(δ)j for all i,
and feeds the results into the SFIFO stream. Parallelsection p4 (lines 29–34) consumes the dotNs from SFIFO
and uses them to calculate the residual vector and nextapproximate solution. Via a multiply-accumulate (MAC),p4 also calculates
∑r2i needed for the convergence
test. Lines 36–38 complete the iteration; line 39 is thetermination test.
3) Output phase: During output, the converged x(δ)
value is DMAed to GCM as shown on line 40.
I. Detailed description of Jacobi processor
If the algorithm in Fig. 21 is implemented and compiledwith the Carte compiler, it produces the FPGA-basedJacobi processor idealized in Fig. 22. The main routineis instrumented with a microsecond-resolution timer. Thetimer is started as the first possible executable statementand ended as the last possible executable statement inorder to capture wall clock execution time. The mainroutine k-aligns the input data and marshals it into GCMwhere it is subsequently DMAed into either OBM orBRAM by the FPGA. OBM banks are used to storekcol and kval, while BRAM is used to store kptr, d,b, the multiple copies of x(δ), and x(δ+1). Recall, theprocessor has three phases: input, compute, and output.As suggested by the circular arrow in Fig. 22, computeconsists of three subphases: update, which creates the ninecopies of x(δ); iterate, which computes the residual vectorand next approximate solution; and test, which determines
FPGA
x(�+1)
x(�)
p1 p3 p4p2
r2b2� no
stream out x(�) and return
9 copies
input
update
iterate
test
GCM bank 1 GCM bank 2
kval kptr d kcol b x(�)
OBM
kval
kcol
output
kptr d b
r2b2 > ��<
�max
no++
compute
Fig. 22. Jacobi processor block diagram
FPGA
GCM bank 1 GCM bank 2
OBMBanks
kval kptr d kcol b x(�)
kval
kcol
kval
kcol
kval
kcol
kval
kcol
bufferedDMAstripe-4
bufferedDMA
streamDMA
d
streamDMA
x(�+1)
256b
64b
kptr b
256:32 256:32
32b 32b
streamDMA
streamDMA
256:32 256:32
32b 32b
BRAM
iaii
allfor 1 BRAM
A
E
B
F
C
G
D
H
FIFO
FIFO
Fig. 23. Jacobi input phase
if the Jacobi iteration should terminate. As previouslydescribed, the processor has three phases: input, compute,and output. The following sections describe each of theJacobi processor phases.
1) Input phase: As shown in Fig. 23, the input phaseconsists of three parallel sections. In the first parallelsection, the Jacobi processor uses buffered DMA to read256-bit words representing the kval and kcol vectorsfrom the GCM and store them, stripe-4, in OBM banks.The result is four 64-bit packed representations of kvaland four 64-bit packed representations of kcol. Each 64-bit value is later unpacked to produce two 32-bit values.In effect, kval and kcol are stored stripe-8, as indicatedin lines 3–4 of Fig. 21. In the second parallel section,the Jacobi processor uses streaming DMA and a 256-bitto 32-bit stream width converter to store the kptr andb vectors in BRAM arrays on the FPGA. In the thirdparallel section, the processor again uses streaming DMAand a stream width converter to store the d (1/aii for alli), and x(0) vectors in BRAM. Note that the latter vectoris stored in the x(δ+1) array to simplify loop entry.
2) Compute phase: The principal features of the com-pute phase are shown in Fig. 24. This block diagram doesnot show the trivial update phase (see Fig. 22). It doesshow the four parallel sections that constitute the iteratesubphase, and the relatively simple calculations associatedwith convergence in the test subphase.
Values stream: Parallel section p1 is an 8 · 8 dotproduct unit. Notice that it reads in the next four 64-bit packed values from kval and kcol and splits theminto the corresponding eight 32-bit values and eight 32-
870 JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013
bit indices. The split kval values correspond to the Amatrix values, and the split kcol indices are used to findthe matching values from x(δ). These pairs are appliedto the eight leaf-node multipliers of the fully pipelinedbinary tree dot product unit as detailed in Fig. 25. Thislatter diagram illustrates most clearly why multiple copiesof x(δ) are needed. The dot product unit must read eightpairs of values on every clock cycle. Striping allowsparallel access to the A values, but each x(δ) resides ata different memory location. Since the FPGA does notsupport multiport memory, the choices are 1) multiplecopies of the vector, or 2) multiple clock cycles for eachmemory access. For performance purposes, option 1 waschosen. The output of the dot product unit is inserted intothe values FIFO stream that will eventually be consumedby the streaming accumulator. It is important to notethat the dot product unit does not need to deal withrow boundaries; it simply emits a stream of partial dotproducts. As alluded to in App. V-B, the k-alignmentassociated with the marshaling process on the softwareside ensures each matrix row has an exact multiple ofeight values (zero padding being used as necessary).
Counts stream: Parallel section p2 simply computesthe number of dot8s per matrix row. This calculation isinserted into the counts FIFO stream that will eventuallybe consumed by the streaming accumulator.
Streaming accumulator: Parallel section p3 computesthe full dot products, dotNi = ai · x(δ) associated witheach matrix row and inserts them into the sums FIFOstream that will be consumed by parallel section p4.
p1
OBM
kval
kcol
kval
kcol
kval
kcol
kval
kcol
a1
a2
a3
a4
a5
a6
a7
a8
x1
x2
x3
x4
x5
x6
x7
x8
×
×
×
×
×
×
×
×
j1j2
j3
j4
j5
j6
j7
j8
+
+
+
+
+
+
+
split
1 .. knz
)(1δx
)(2δx
)(3δx
)(4δx
)(5δx
)(6δx
)(7δx
)(8δx
32b
64b
Fig. 25. Dot product unit
Residual and next approximation: Parallel section p4
consumes the sums FIFO stream and uses it, in conjunc-tion with the other inputs, to calculate the residual andnext approximate solution vectors. Notice that it also usesa multiply-accumulate (MAC) unit to calculate
∑r2 for
the residual norm calculation.
Convergence test criteria: This section of hardwaredoes not produce a meaningful value until the fourparallel sections mentioned above have completed. It thencomputes the residual norm ratio r2b2 = ‖r(δ)‖/‖b‖ andupdates the number of iterations. These are needed toperform the termination test shown on line 39 of Fig. 21.
J. Results
1) Description of test problems: To create test matri-ces, Matgen [40] was used. This tool accepts an input filedescribing the features of the matrix (including diagonaldominance) and writes the output in Matrix Marketcoordinate format. Three sets of matrices were used forthe tests. Each set consists of 8 · 3 = 24 matrices. Thereare eight matrix orders, 1, 000, 2, 000, . . . , 8, 000. Foreach order, three matrices having sparsity percentages ofapproximately two, four, and six percent were generated,that is, nz ≈ 0.02n2, nz ≈ 0.04n2, and nz ≈ 0.06n2,respectively. These are referred to as trial (1), trial (2),and trial (3), respectively. The b vector was generatedas b = Axs, where the solution vector, xs, consistsof all 1,000s. The initial value of the solution vector,x(0), consisted of all zeros. Since 32-bit floating-pointdata were used, the termination threshold was set atε = 5 · 10−6. The maximum iteration count was set atδmax = 20, 000.
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 871
2) Test results: For all 72 software and 72 hardwaretest cases, Jacobi was run on an unloaded system andterminated with the expected solution, x(δ) ≈ xs. Theaverage wall clock times for both Jacobi versions areshown in Fig. 26. The wall clock values are the averagetimes for the three sets of 24 matrices. As expected,by virtue of the data set sizes, trial (1) takes less timethan trial (2), which takes less time than trial (3). Inaddition, the hardware execution times for larger data setsare significantly lower than software execution times. Theexplanation for this phenomenon is simple. When the datasize exceeds the cache limits of the GPP then the GPPperformance suffers. See [24] for the supporting research.Notice that several of the hardware cases show a nearlythreefold speedup over software.
VI. CONCLUSION
Mapping floating-point kernels on FPGA-based HPHCsvia HLL-to-HDL compiler technology can still be aformidable task that relies mostly on the skill and intuitionof individual developers. In this article we looked atsome HPRC application design considerations includingthe three p’s, expected overall speedup, expected re-source utilization, control/memory intensive vs. computeintensive, monolithicity of modules, available bandwidth,opportunities for data reuse, algorithm design stability,algorithm efficiency, and memory access patterns. We alsoshowed a simple example of how to interface HDL-basedIP cores into an HLL-based design flow. By way of amore complete sparse matrix Jacobi iterative solver, weillustrated the challenging floating-point mapping processwhile simultaneously showing that such a mapping can re-sult in a significant speedup compared with an equivalentsoftware implementation. Our HPRC-based sparse matrixJacobi iterative solver demonstrates a nearly threefold
wall clock run time speedup when compared with asoftware implementation. If FPGA-based computationalunits are to be part of mainstream scientific computing,more research is necessary to simplify the mappingprocess. Ideas from research such as this should continueto be incorporated into tools to facilitate a more automatedmapping process.
ACKNOWLEDGMENTS
This work was supported in part by the DoD HighPerformance Computing Modernization Program undercontract numbers W912HZ-(08-C-0073, 09-C-0108, and10-C-0107), “High Performance Computational Designof Novel Materials,” in part by Army Research OfficeHBCU/MSI grant number W911NF-07-1-0527, and inpart by the U.S. Army Engineer Research and Develop-ment Center.
REFERENCES
[1] Jane’s Information Group, “Tactical reconnaissance and counter-concealment enabled radar (TRACER)(United States),” Jane’sElectronic Mission Aircraft, 2011.
[2] M. Feldman, “JP Morgan buys into FPGA supercomputing,”HPCwire (www.hpcwire.com/hpcwire/2011-07-13), July 2011.
[3] G. Estrin, “Organization of computer systems – the fixed plusvariable structure computer,” in Proceedings of the Western JointComputer Conference, San Francisco, CA, USA, May 1960, pp.33 – 40.
[11] L. Zhuo and V. K. Prasanna, “High performance linear algebraoperations on reconfigurable systems,” in Proceedings of theACM/IEEE SuperComputing 2005 Conference, Seattle, WA, USA,November 2005, pp. 2 – 13.
[12] ——, “Design tradeoffs for BLAS operations on reconfigurablehardware,” in Proceedings of the 34th International Conferenceon Parallel Processing, Oslo, Norway, June 2005, pp. 78 – 86.
[15] S. J. Park, “Reconfigurable computing for HPC computationalscience,” in Proceedings of the 2007 HPCMP User GroupConference, Pittsburgh, PA, USA, June 2007, p. www.hpcmo.hpc.mil/UGC2007/UGC 2007 Agenda.pdf.
[17] D. McGrath, “Reconfigurable cluster computing installation couldbe a first,” COTS Journal, April 2009.
[18] Panasas, “Parallel file system for HPC storage,” www.panasas.com,2011.
[19] A. R. Anderson, G. R. Morris, and K. H. Abed, “Achieving trueparallelism on a high performance heterogeneous computer viaa threaded programming model,” in Proceedings of the IEEESoutheastCon 2011, Nashville, TN, USA, March 2011, pp. 283– 286.
[20] A. N. Malone, G. R. Morris, and K. H. Abed, “FPGA-basedimplementation of Horner’s rule on a high performance heteroge-neous computer,” in Proceedings of the IEEE SoutheastCon 2011,Nashville, TN, USA, March 2011, pp. 277 – 282.
[21] N. S. Peay, G. R. Morris, and K. H. Abed, “IntegratingQuartus Wizard-based VHDL floating-point components into ahigh performance heterogeneous computing environment,” inProceedings of the IEEE SoutheastCon 2011, Nashville, TN, USA,March 2011, pp. 413 – 417.
[22] G. R. Morris and K. H. Abed, “Mapping hierarchical multiple fileVHDL kernels onto an SRC-7 high performance reconfigurablecomputer,” in Proceedings of the High Performance ComputingModernization Program Users Group Conference 2010, Schaum-burg, IL, USA, June 2010, pp. 524 – 533.
[23] G. R. Morris, R. Y. McGruder, and K. H. Abed, “Acceleratinga sparse matrix iterative solver using a high performancereconfigurable computer,” in Proceedings of the High PerformanceComputing Modernization Program Users Group Conference2010, Schaumburg, IL, USA, June 2010, pp. 517 – 523.
[24] K. H. Abed and G. R. Morris, “Improving performance ofcodes with large/irregular stride memory access patterns viahigh performance reconfigurable computers,” in Proceedings ofthe High Performance Computing Modernization Program UsersGroup Conference 2009, San Diego, CA, USA, June 2009, pp.422 – 429.
[25] J. L. Rice, K. H. Abed, and G. R. Morris, “Design heuristics formapping floating-point scientific computational kernels onto highperformance reconfigurable computers,” Journal of Computers,vol. 4, no. 6, pp. 542 – 553, June 2009.
[26] J. L. Hennessy and D. A. Patterson, Computer Architecture: AQuantitative Approach, 3rd ed. Morgan Kaufman, 2003.
[27] J. Harkins, T. El-Ghazawi, E. El-Araby, and M. Huang,“Performance of sorting algorithms on the SRC 6 reconfigurablecomputer,” in Proceedings of the 2005 IEEE International Con-ference on Field-Programmable Technology, Singapore, December2005, pp. 295 – 296.
[28] M. C. Herbordt, T. V. Court, Y. Gu, B. Sukhwani, A. Conti,J. Model, and D. DiSabello, “Achieving high performance withFPGA-based computing,” Computer, vol. 40, no. 3, pp. 50 – 57,March 2007.
[29] G. R. Morris and V. K. Prasanna, “Sparse matrix computations onreconfigurable hardware,” Computer, vol. 40, no. 3, pp. 58 – 64,March 2007.
[30] K. Habgood and I. Arel, “A condensation-based application ofcramer’s rule for solving large-scale linear systems,” Journal ofDiscrete Algorithms, vol. 10, pp. 98 – 109, January 2012.
[31] Altera Corporation, “Quartus II v9.1,” www.altera.com.[32] Mentor Graphics, “ModelSim-Altera Edition,” www.altera.com.[33] R. S. Varga, Matrix Iterative Analysis, Second Edition. Springer,
2009.[34] E. Isaacson and H. B. Keller, Analysis of Numerical Methods.
John Wiley & Sons, 1966.[35] Y. Saad, “SPARSKIT: A basic tool-kit for sparse matrix com-
[36] G. R. Morris, R. D. Anderson, and V. K. Prasanna, “An FPGA-based application-specific processor for efficient reduction ofmultiple variable-length floating-point data sets,” in Proceedingsof the 17th IEEE International Conference on Application-SpecificSystems, Architectures and Processors, Steamboat Springs, CO,USA, September 2006, pp. 323 – 330.
[37] NIST, “Matrix Market,” math.nist.gov/MatrixMarket, June 2004.[38] L. Zhuo, G. R. Morris, and V. K. Prasanna, “High-performance
reduction circuits using deeply pipelined operators on FPGAs,”IEEE Transactions On Parallel and Distributed Systems, vol. 18,no. 10, pp. 1377 – 1392, October 2007.
[39] G. R. Morris, L. Zhuo, and V. K. Prasanna, “High-performanceFPGA-based general reduction methods,” in Proceedings of the13th IEEE Symposium on Field-Programmable Custom ComputingMachines, Napa, CA, USA, April 2005, pp. 323 – 324.
Gerald R. Morris is a researcherat the U.S. Army Engineer Researchand Development Center, ScientificComputing Research Center, Vicks-burg, MS. He is also Adjunct Profes-sor of Computer Engineering at Jack-son State University, Jackson, MS,and Adjunct Professor of Computer
Science at Mississippi State University, Mississippi State,MS. His research interests include high performancecomputing and mapping of algorithms onto alternativecomputational technologies. Morris received the B.S. inelectrical engineering from the Ohio State University,M.S. in computer engineering from the Air Force Instituteof Technology, and Ph.D. in electrical engineering fromthe University of Southern California. He has publishedextensively and is a Senior Member of the IEEE.
Khalid H. Abed is Professor ofComputer Engineering at JacksonState University, Jackson, MS. Hisresearch interests include high per-formance heterogeneous computing,field programmable gate arrays, verylarge scale integrated circuit design,and digital signal processing. He
received the B.S., M.S., and Ph.D. in electrical engineer-ing from Wright State University. Abed has numerouspublications in IEEE journals and conferences and is atechnical reviewer for several IEEE journals and confer-ences. He has received funding from sources such as theNational Science Foundation, the Department of Defense,and the Army Research Office. Abed is a Senior Memberof the IEEE.
JOURNAL OF COMPUTERS, VOL. 8, NO. 4, APRIL 2013 873
