Autotuning in High- Performance Computing Applications

This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

Autotuning in High-Performance ComputingApplicationsBy PRASANNA BALAPRAKASH, JACK DONGARRA , Fellow IEEE, TODD GAMBLIN, Member IEEE,

MARY HALL , Senior Member IEEE, JEFFREY K. HOLLINGSWORTH, Senior Member IEEE,BOYANA NORRIS, AND RICHARD VUDUC, Member IEEE

ABSTRACT | Autotuning refers to the automatic generation of

a search space of possible implementations of a computation

that are evaluated through models and/or empirical measure-

ment to identify the most desirable implementation. Autotun-

ing has the potential to dramatically improve the performance

portability of petascale and exascale applications. To date,

autotuning has been used primarily in high-performance appli-

cations through tunable libraries or previously tuned applica-

tion code that is integrated directly into the application. This

paper draws on the authors’ extensive experience applying

autotuning to high-performance applications, describing both

successes and future challenges. If autotuning is to be widely

used in the HPC community, researchers must address the

Manuscript received July 7, 2017; revised December 7, 2017; acceptedJanuary 15, 2018. The work of P. Balaprakash and M. Hall was supported in partby the Exascale Computing Project (17-SC-20-SC), a collaborative effort of theU.S. Department of Energy Office of Science and the National Nuclear SecurityAdministration. The work of J. Dongarra was supported by the National ScienceFoundation under Award ACI-1642441. A portion of the work of T. Gamblin wasperformed under the auspices of the U.S. Department of Energy by LawrenceLivermore National Laboratory under Contract DE-AC52-07NA27344. The workof M. Hall, J. K. Hollingsworth, and B. Norris was supported by the U.S.Department of Energy, Office of Advanced Scientific Computing Research(ASCR), Scientific Discovery through Advanced Computing (SciDAC) programunder Award ER26054. The work of M. Hall was additionally supported byNational Science Award SHF-1564074. The work of J. K. Hollingsworth wasadditionally supported by the ASCR X-Stack Project under Award ER26143 andthe Department of Defense through a contract with the University of Maryland.(Corresponding author: Mary Hall.)

P. Balaprakash is with the Argonne National Laboratory, Argonne,IL 60439 USA.

J. Dongarra is with the University of Tennessee, Knoxville, TN 37996 USA, withthe Oak Ridge National Laboratory, Oak Ridge, TN 37831 USA, and also with theUniversity of Manchester, Manchester M13 9PL, U.K.

T. Gamblin is with the Lawrence Livermore National Laboratory, Livermore,CA 94550 USA.

M. Hall is with the University of Utah, Salt Lake City, UT 84112 USA (e-mail:[email protected]).

J. K. Hollingsworth is with the University of Maryland, College Park,MD 20742 USA.

B. Norris is with the University of Oregon, Eugene, OR 97403 USA.

R. Vuduc is with the Georgia Institute of Technology, Atlanta, GA 30332 USA.

Digital Object Identifier 10.1109/JPROC.2018.2841200

software engineering challenges, manage configuration over-

heads, and continue to demonstrate significant performance

gains and portability across architectures. In particular, tools

that configure the application must be integrated into the

application build process so that tuning can be reapplied as

the application and target architectures evolve.

KEYWORDS | High-performance computing; performance

tuning programming systems.

I. I N T R O D U C T I O N

Since the first petascale supercomputer nearly a decadeago—the RoadRunner comprised of standard AMD64 mul-ticores and custom IBM Cell processors—we have wit-nessed a diversity of supercomputing architectures thatpose significant challenges for scientific application devel-opers. Indeed, the four most powerful supercomputersin the world at the end of 2017—TaihuLight, Tianhe-2,PizDaint, and Gyoukou—rely on fundamentally differentprocessor architectures from distinct hardware vendors.Because of profound differences in architecture and pro-gramming models, high-performance applications must beoptimized and frequently rewritten in an architecture-specific way to attain acceptable performance. A manualcode rewrite necessitated each time a new supercomputerarchitecture or architecture generation enters the scene isprohibitively expensive and limits the porting of applica-tions to new platforms.

Clearly, a desirable feature of high-performance appli-cations is performance portability, whereby the sameapplication code can achieve high performance across adiversity of architectures. Performance portability is cur-rently difficult to achieve. Even with programming modelchanges that enable this diversity of target architecturesto be expressed, such as support for CPUs and GPUs inOpenMP 4, a code that targets one type of platform stillmay not perform well on another.

0018-9219 © 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

PROCEEDINGS OF THE IEEE 1

https://orcid.org/0000-0002-3058-7573

https://orcid.org/0000-0003-3247-1782


Balaprakash et al.: Autotuning in High-Performance Computing Applications

Suppose that a single implementation is written byapplication developers and that porting of that applicationto different architectures is somehow automated. If suc-cessful, this strategy eliminates the high-performancecomputing (HPC) programmers’ burden in managingarchitectural diversity. This paper describes such anapproach, called autotuning, which involves automaticgeneration of a search space of possible implementa-tions of a computation that are evaluated through mod-els and/or empirical measurement to identify the mostdesirable implementation. Although autotuning is usuallyemployed to reduce execution time, multiobjective tun-ing may optimize across a variety of criteria includingperformance, energy efficiency, peak power, or reliability.The impact of using autotuning as opposed to manualtuning includes increased programmer productivity, easeof porting to new platforms and, in some cases, better-performing applications.

This paper describes our experiences working with HPCapplication developers to develop autotuning technologythat meets these goals and is compatible with the develop-ment of HPC production codes. In this paper we examinethe state of the practice in incorporating autotuned codeinto HPC applications, insights from prior work, and thechallenges in advancing this technology into wider andlong-term use.

To examine the various ways in which autotuning canbe employed, we first categorize a number of aspectsof autotuning. The subsequent sections describe existingautotuning tools, followed by case studies from prior workthat illustrate the strengths and weaknesses of approacheswith respect to these aspects. The last two sections discussthe software engineering challenges and future directionsthat will increase the benefits of this valuable technology.

II. O V E RV I E W

Fig. 1 presents an overview of the components of auto-tuning systems. Autotuning starts with an application or

kernel of interest, along with a set of known tuningparameters. Tuning parameters are used in conjunctionwith the kernel to generate a set of versions or variantsof the code. The goal of the autotuner is to select thebest-performing variant from the search space describedby the tuning parameters. Rectangles capture functionalitythat might be separate tools or tuning data. The figurealso shows the differences in autotuning frameworks onthe continuum from systems that perform autotuning atcompile time (or tuning time) to runtime. The verticalbars delineate this range. For example, the first or leftmostvertical bar indicates that all analysis, modeling, empiricaltests, and so on are deferred to actual program executiontime, whereas the rightmost vertical bar indicates that allthe autotuning is completed in an offline tuning phase.In the middle are online or incremental tuning, whichoccurs over one or multiple runs of an application, andruntime variant selection, where the selection of code to beexecuted is deferred to runtime based on a model derivedfrom offline training.

Table 1 characterizes several of these aspects of autotun-ing in current use. The aspects considered cover the spaceof the autotuning literature and will be described in moredetail, including tradeoffs in approaches, in the remainderof this paper.

A. How Packaged

At the heart of autotuning is a search space of codevariants that are functionally equivalent to an originalimplementation. These are often packaged as libraries ofcommonly used numerical functions such as linear algebraand fast Fourier transform (FFT). However, one also canemploy compiler and code generation tools that composea collection of code transformations to generate opti-mized code or express the code variants at the applicationlevel. Recently, autotuning has been built into embeddeddomain-specific tools and execution frameworks, wherethe search space arises from the common framework.

Fig. 1. Components of autotuners on a continuum from compile time to runtime support, with examples.

2 PROCEEDINGS OF THE IEEE



Table 1 Overview of Issues in Autotuning for High-Performance Computing

B. Approach to Selection

Autotuners can select among code variants in many dif-ferent ways. The simplest approach is to execute each codevariant, measure its runtime (or other objective function),evaluate the performance of all variants, select the bestone, and include that variant in the final code to be run.These are called empirical autotuners. Because the searchspace may be large, intelligent search methods and modelsmay be used to iteratively prune the variant space asevaluation takes place. Instead of executing trials directly,some autotuners may train models from trial executions orfrom historical data. A runtime prediction model can beused as a proxy for real kernel executions, which allows atool to more rapidly search the tuning parameter space,especially for long-running kernels. Models arising fromtraining are particularly useful when selection dependson input data or other aspects of execution context; suchdecision models are consulted at runtime to select variantsbased on contextual features. Application developers mayalso embed hints in their code to influence the choice ofvariant at runtime.

C. Types of Decisions

Code variants may affect code organization, data struc-tures, high-level algorithms, and low-level implementationdetails. In order to achieve performance portability, deci-sions on parallelization (how much and how many levelsof parallelism) and memory hierarchy optimizations (e.g.,data placement, blocking/tiling and tile size) will necessar-ily depend on the architecture. If the transformations used

in code generation can alter the behavior or accuracy ofthe kernel, a functional equivalence verification step maybe added to ensure that each variant yields correct results.Incorrect variants may be regenerated or excluded fromconsideration.

D. When to Apply

Depending on the architecture, the application domain,and the type of code being tuned, the components canbe implemented in many ways as shown in Fig. 1 bythe dashed lines. An extensive autotuning search can beexpensive. Consequently, autotuning is typically performedoffline, prior to application execution (rightmost dashedline). The end user can be completely uninvolved in auto-tuning if it is applied only once each time the softwareis ported to a new architecture by the library or appli-cation developer (not shown). If the application is beingmodified, then offline autotuning is needed each time thetuned computation or its context changes (e.g., a newdata layout changes the performance). At the other endof the spectrum, the entire search is executed at runtime,with help from dynamic compilation servers to generatedifferent code variants (leftmost). Alternatively, when anend user is unable or unwilling to participate in an offlineor dynamic tuning phase, a changing application must beincrementally tuned. This requires significant additionalinfrastructure that includes a performance database thatrecords the history of prior measurements as well asdynamic compilation and linking (second from left). Othertools take a hybrid approach, deferring variant selection




decisions until runtime, and leverage training data fromprior or training runs (third from left).

E. Integration Into Applications

The most common use of autotuning in production HPCcodes is to directly incorporate the tuned code, particularlylibraries, into the application through either compilationor linking. In this way, the code does not change; once thecode is verified, the application developer can be confidentthat the code is correct. This approach has two disadvan-tages, however. First, the code resulting from autotuningwill be architecture specific (and possibly unreadable ifautomatically generated), and so the goal of performanceportability is lost. Second, because the tuned code wasgenerated for a specific execution context, over time it willbecome less optimized or unsuitable for new architecturesor in conjunction with application changes. Where multiplecode variants may be appropriate depending on executioncontext, these may all be compiled into the application,along with a selection function that decides which codevariant to execute. When the desired code variants becometoo large to compile into the application, then online codegeneration, compilation, and linking are also required fordynamic tuning.

F. Runtime Measurement

For tuning tools that rely on dynamic feedback at run-time, measurement tools need to be integrated with theapplication as well as autotuned code. This integrationcan be accomplished with binary instrumentation/profilingtools such as Dyninst [1], or with hints from the applica-tion using semantic annotation tools such as Caliper [2].In the former case, measurement is transparent, whereastools like Caliper are designed to collect useful tuninghints from the developer. For example, the Apollo systemdescribed in the next section can build tuning models thatmake different decisions based on the particular physicsphase or solver iteration of a simulation. Caliper simplifiesthe correlation of annotations at multiple levels of the soft-ware stack. Once collected, performance data can be storedin a database for retrieval in future experiments, with anaming scheme that clarifies the point in the search spacefor each measurement and adequate provenance informa-tion to understand its relevance, for example, TAUdb [3].

III. E X I S T I N G A U T O T U N I N G T O O L S

We describe in more depth a collection of autotuning toolsthat have been used for autotuning HPC applications.

A. Libraries

The first autotuning systems were packaged as libraries.This choice was enabled in part by layered library designsin which a relatively small number of performance-criticalsubroutines could be isolated behind a standard appli-cation program interface (API), which hardware vendors

could then tune for their platforms. However, whereversuch platform-specific implementations were unavailable,too costly, or not fast enough, researchers naturally studiedtechniques to generate and tune a library implementationautomatically.

One of the earliest motivating examples of such anAPI was the Basic Linear Algebra Subprograms (BLAS)standard, which defines a core set of primitives to supportdense linear algebra [4]. On top of the BLAS, it thenbecomes possible to build fully featured linear algebra,such as the widely used defacto standard, LAPACK and itsassociated libraries ScaLAPACK, CLAPACK, and LAPACK95.Within the BLAS, one of the most important primitivesis the general matrix multiply or GEMM operation [5].The GEMM subroutine became an immediate target forautotuning in the early 1990s, including the PHiPAC andthe Automatically Tuned Linear Algebra Software (ATLAS)systems [6], [7].

To see how a typical library autotuner system mightwork, consider ATLAS as an example. It generates efficientcode by running a series of timing experiments usingstandard performance engineering techniques (e.g., loopunrolling and blocking) to determine optimal parametersand code structures. The process begins with detectingspecific hardware properties, such as cache sizes and thefloating-point pipeline length. Then, ATLAS systematicallyexplores the different possible implementations (of, say,GEMM) of which there can be hundreds of thousands ofvariations. After eliminating unlikely candidates by usingheuristics based on gathered information, ATLAS gener-ates code to implement the remaining choices and thencompiles, executes, and measures their execution time tochoose the fastest. One also can extend the methodologyto work on modern platforms, such as GPUs [8], whichprovide end-user developers a measure of performanceportability. The result is that the best implementationsoften achieve large fractions of the highest possible per-formance for a given platform.

Library autotuners are, by design, domain specific. Theyexploit domain knowledge aggressively to improve per-formance. As the ATLAS GEMM example demonstrates,constraining the tuning problem to a specific kernel canmake it tractable to enumerate, prune, and then explorea space of candidates. Beyond code, domain knowledgecan also enable the exploration of variants at a higherlevel. For instance, the SPIRAL system encodes linearsignal transforms symbolically and then uses a symbolicalgebra engine to derive candidate algorithms [9]. Sym-bolic algebraic techniques have also been applied to auto-matic derivation of linear algebra methods [10], [11].These methods use structural information that is usuallyunavailable to more general-purpose code transformationsystems.

Additionally, domain-specific knowledge can extend toinputs. Examples include the Sparsity and OSKI systemsfor sparse matrix computations [12], [13], as well asmore recent extensions for sparse direct solvers and sparse




tensor computations [14], [15]. These systems use com-bined knowledge about the computational kernels, thelikely runtime use-cases and costs, and the kinds of inputpatterns likely to occur. This information is used to designspecific methods for analyzing the input and selecting adata structure and tuned implementation at runtime. Beingspecific to the library, these methods can be both effectivebut also hard to generalize.

A library is also a natural setting in which to try toexploit execution history with little or no user intervention.For instance, consider the FFTW library system for auto-tuning fast Fourier transforms [16], [17], developed inde-pendently at around the same time as PHiPAC and ATLAS.It had a “wisdom” feature that tracked the performance ofexecution candidates and used that information to improvetiming over subsequent executions. However, this processhad the downside of introducing variable or unpredictableexecution time cost.

Many ways exist for generalizing the techniquesdescribed above outside of a domain-specific library,including within the compiler, runtime system, or appli-cation. Over time, the simple code generators of earlysystems have adapted ideas from compilers, producingdomain-specific compilers [18]–[20]. However, librarymethods may still be preferable, at least initially, in certaincontexts, such as when selecting among candidate sparseiterative solvers or settings when explicit reasoning aboutnumerical accuracy tradeoffs are necessary.

More generally, packaging autotuning within a libraryhas several practical benefits related to use, distribution,and maintenance. First, applications naturally rely onlibraries, and a library API forms a natural abstraction forisolating parts of the program that might need platform-specific tuning. Second, if that API is already widely used,as is the case with the BLAS, using an autotuned librarycan be as simple as linking to a different implementation.Third, an end-user developer can also rely on a librarywhen it is backed by a community standard, even if theautotuned version becomes outdated or otherwise unavail-able, reference implementations or vendor-provided alter-natives exist.

However, packaging as a library can play out in differentways. The widespread adoption of ATLAS was encour-aged, at least in part, by the fact of and interest inhardware-specific implementations of the BLAS. In thecase of FFTW, the first library-based autotuner for (FFT)computations, it actually became a standard way to call thefast Fourier transform (FFT), in part because there was nostandard.1

By contrast, in the domain of sparse linear algebra, thecommunity has arguably not converged on an API for itscore computational kernels, despite community efforts to

1For instance, Intel’s Math Kernel Library (Intel MKL) andNVIDIA’s cuFFT adopted FFTW’s interface. Regarding an officialstandard, DARPA supported an effort to develop the Vector Signal andImage Processing Library (VSIPL) API, but that was still being activelydiscussed at the time of FFTW’s release.

develop one.2 One explanation is that the basic data struc-tures and calling sequences of sparse matrix computationshave much more variety, which has led to many candidatelibraries and APIs, but no convergence on a single one.Furthermore, proper tuning may involve reconsidering thechoice of data structure in an input-dependent way, whichmay involve a potential runtime cost to deploy a sparsematrix autotuner that an application developer must nowconsider. Consequently, despite the development of auto-tuners for certain sparse matrix primitives [12], [13], theytypically have not been integrated into applications orwidely used sparse solver libraries.

Notably, the history of library-based autotuning is longenough that we can look back on some of the earlysystems and assess their impact. The PHiPAC, ATLAS, andFFTW systems have received impact awards for the originalpapers in their respective publication venues.3 And beyondthe ideas, the resulting software has been just as important.ATLAS enjoys wide use and has been included as a part ofseveral Linux distributions. Before ATLAS, vendors chargedsignificant prices for their tuned libraries, which discour-aged some independent software vendors from using thework in their products. ATLAS removed this obstacle, amove that had significant implications for commercial soft-ware. Similarly, FFTW also has many users and receiveda major prize for numerical software.4 Indeed, vendorshave even adopted autotuning methodologies as part oftheir library-building processes. These include the CrayScientific Library (LibSci), Intel MKL and its more recentlibrary tuned for small matrices [25], and NVIDIA’s cuFFT,which like Intel MKL adopts an FFTW-like interface.

B. Compilers and Code Generators

While libraries can encapsulate common computationsand eliminate the need for programmer involvement inautotuning, libraries are limited in the scope of theirapplicability and the contextual information that allowscomposition of optimizations beyond individual librarycalls. For computations for which a library is unavailableor too limited, autotuning compilers and code generatorssuch as CHiLL [26], [27], Orio [28] and POET [29] canpotentially generate a collection of architecture-specificcodes from the same high-level input. Parallel code gen-eration can include parallelization (via SIMD pragmas,OpenMP, CUDA, etc.). Other transformations for HPCcodes, available in compilers but commonly applied duringmanual tuning, include loop tiling (often called blocking byapplication developers), loop unrolling, loop permutation,

2For example, the BLAS standards committee has defined an inter-face for sparse computational kernels, but it has not been widely adopted.

3FFTW was recognized as a “Most influential PLDI paper” in2009 [21]. PHiPAC received the “most influential paper in 25 years”award in 2014 [22]; ATLAS received a Best Paper award fromACM/IEEE Supercomputing (SC) in its publication year, as well as aTest-of-Time Award in 2016 [23].

4In particular, FFTW received the Wilkinson Prize for numericalsoftware in 1999 [24].




fusion, distribution, prefetching, and software pipelining.Data transformations may also be applied to reorganizethe data layout or copy it to other memory structures. Thedecisions that must be resolved during autotuning includewhich transformations to apply and in which order, as wellas adjusting values for parameters of the optimizations,such as number of parallel threads/ranks, tile size, unrollfactor, or prefetch distance.

Historically, compiler optimization decisions have beenbased on analytical models and heuristics. These decisionsare governed by a do-no-harm philosophy such that theyare not performed in cases where they may slow downcommon workloads. Using autotuning, a compiler may befar more aggressive and tailor optimization to the needs ofa specific application running on a specific target architec-ture, and thus it is more likely to achieve the performanceof manual tuning.

To integrate autotuning into a compiler frameworkrequires a search space of possible implementations ofeach computation. Such a search space potentially can begenerated automatically by a compiler decision algorithm,but this is a difficult challenge for general applicationsand architectures. Some success has been achieved whenspecialized for a specific application or application domainand a specific architecture or class of architectures, suchas the Nek5000 example in Section V. As another exam-ple, Orio has been used to optimize data layouts andgenerate optimized sparse linear algebra computations onGPUs for finite-difference stencil-based solution of partialdifferential equations [30], [31]. From this knowledge,the search space of desirable optimizations can lead to afixed decision algorithm that fully automates the tuningprocess. For other applications, however, the search spaceis unknown, or automating the decision algorithm is pre-mature until an expert programmer, compiler developeror machine learning algorithm has figured out what thesearch space should be. Therefore, it is desirable to designautotuning compiler and code generation frameworks thatare configurable and permit description of the search spaceby expert users.

The current state of the art in expressing a search spaceof transformations encodes these in scripts or transfor-mation recipes [28], [32], [33]. Tools such as POET andXevolver even support programmers’ expression of thetransformations to be applied [29], [34]. Users of suchsystems annotate loop nest computations with possibletransformations and the set of associated parameters. Froma set of such recipes or an encoding of multiple recipes,a large collection of code variants can be described andsearched by using the techniques described below withregard to selection approach.

As shown in the case studies and other work, compiler-directed autotuning can produce code that achieves per-formance comparable to and sometimes exceeding thatof manual tuning. The strength of such an approach liesin the ease of exploring completely different implemen-tations that would be time-consuming for a programmer

to produce. In particular, it can optimize more of anapplication than the time-consuming portion that is aprogrammer’s focus, and can try more combinations ofoptimizations. The future of compiler-directed autotuningrequires automating or encapsulating derivation of thesearch space so that nonexperts can benefit from thetechnology without having to interact with it directly.

C. Application-Level Autotuning

Autotuning may also be specified at the applicationlevel, and many programming systems have been devel-oped that permit expression of tunable parameters andcode variants representing alternative implementations[27], [35]–[37]. The advantage of specifying what totune at this highest level of semantics is that significantalgorithmic changes can be expressed. For example, fun-damentally different approaches to solving the problemcan be encoded for the autotuner. A solver with betterperformance or convergence properties may be selected,or a sort algorithm can be tailored to its input data set (seeSection III-D). Libraries and compilers or code generatorsare unable to provide such a dramatic change to theprogram.

A distinguishing characteristic of application-level auto-tuning systems is the criteria for selecting the appropriatecode variant. Much of the prior work selects among differ-ent implementations based on problem size. For example,PetaBricks [35] and Sequoia [36] are designed to recur-sively decompose algorithms to target different levels ofthe memory hierarchy or parallelism, with autotuning usedto find the inflection points based on problem size forselecting among implementations. For this purpose, offlineautotuning can be used to build a table of implementa-tions, and runtime code variant selection then involvesa simple table lookup based on problem size. However,suppose code variant selection is dependent on the inputdata set, known only at runtime. Recent systems havedeveloped selection criteria for input-dependent code vari-ant selection, where programmers express code variantsalong with metainformation that aids the system in variantsection at runtime; a training phase constructs a selectionmodel using machine learning, and this model is consultedat runtime when a new input is presented to make theselection [37]–[39]. Alternatively, runtime selection can beachieved through dynamic tuning; Active Harmony [40]and ADAPT [41] are capable of creating, compiling, link-ing, and testing new code variants in parallel with execu-tion during iterative computations and replacing defaultimplementations when better variants are found.

Another advantage of application-level autotuning isthat it allows the use of autotuning options that canchange the output of the program. In many cases, theaccuracy of the computer simulation is limited based onuncertainties in the underlying physical system. A domainexpert is aware of these limits and can specify tunableparameters that can change the answer but ensure that




any such changes are within the limitations of the inherentuncertainty of the simulation. For example, autotuning theGS2 plasma physics code [42] involved tuning the numberof grid points and the energy grid. Both parameters canchange the answer. However, domain experts assisted inthis process to constrain the permissible range of values forthese parameters so that any changes in the answer wereacceptable. This additional freedom resulted in an extra30% reduction in the program’s runtime compared to thebest autotuned version that only considered parametersthat left the answer the same.

However, a disadvantage of making changes at the appli-cation level is that each application developer must specifythe autotuning. When autotuning is done within commonlibraries or by the compiler, autotunable transformationscan be specified once and shared by multiple applications.

D. Frameworks and Domain-Specific Systems

An increasingly common strategy for achieving highperformance and performance portability on HPC app-plications is using specialization of high-level code forparticular architectures and application domains. For per-formance portability, application developers need tightintegration and the ability to adapt over time. Performanceportability frameworks such as RAJA [43] and Kokkos [44]are becoming more popular in HPC, and they provide C++

template abstractions around application loops and datastructures. These frameworks typically abstract a loop as atemplate function taking a lambda function as an argumentalong with several policy template parameters to controlhow the lambda is to be executed. This approach clearlyseparates tuning concerns from application semantics:application developers can write loop bodies in the codecontext where they are relevant, and performance expertscan write hardware-specific code in policy implementa-tions. Without a tool like Apollo [45], frameworks likethese are limited to static tuning decisions. Apollo allowstemplate instantiations to be treated as code variants andto be compiled with the application code. In addition, itallows code variant selection to be implemented as anexternal library, so that decision models can be updatedover time. This is a useful compromise between librariesand direct compilation, and it can be combined with onlinecode generation if the number of variants grows too large.

Another emerging approach that capitalizes on special-ization performs optimizations for particular applicationdomains, where the optimizations that are effective areknown and autotuning is used to fine-tune optimiza-tion decisions or retarget to different architectures. Forexample, Halide, an embedded domain-specific language(DSL) for image-processing workflows, emerged from theresearch community [46] and is now in production useby Adobe and Google. Many DSLs have been developed,particularly for stencil computations [19], [20]. Tools forbuilding DSLs have also emerged [47]. In spite of thissignificant progress, however, DSLs are not widely used

in HPC, for many of the adoption reasons that will bediscussed in Sections VI and VII.

IV. S E A R C H : M O D E L-F R E E , M O D E L-B A S E D , A N D H Y B R I D S E L E C T I O N

For all the different ways that autotuning search spacesarise, as described in the preceding section, a mechanismis needed to evaluate some points within that search spacein order to arrive at an optimized solution. In this section,we describe various approaches to explore the autotuningsearch space.

To find good parameter configurations, some autotunersperform complete enumeration either of all possible para-meter configurations or of a pruned set of parameter con-figurations obtained by exploiting expert knowledge andarchitecture-specific and/or application-specific informa-tion. Examples include application-specific autotuners suchas lattice Boltzmann computations [48], stencil computa-tions [49], and matrix multiplication kernels [50], [51].

The main drawback of these autotuners is scalability;as codes and architectures become more complex, thenumber of tunable parameters and parameter configura-tions grows rapidly. Consequently, the computation timeneeded to enumerate all parameter configurations in alarge decision space is prohibitively expensive. Hence,effective autotuners that examine a small subset of possibleconfigurations are required. Two classes of algorithm exist:model-free and model-based algorithms.

Model-free algorithms do not use models to navigatethe search space to find high-performing configurations.These algorithms can be grouped into global and localsearch algorithms. Global algorithms are characterized bytheir dynamic balance between exploration of the searchspace and exploitation of the accumulated search history.Examples include simulated annealing, genetic algorithms,and particle swarm optimization. They are theoreticallyguaranteed to find the globally best configuration at theexpense of a long search time. In practice, however, theyare run until a user-defined stopping criterion is met.In contrast, local search algorithms do not emphasizeexploration and instead repeatedly try to move from acurrent configuration to a nearby improving configuration.Typically, the neighborhood of a given configuration isproblem-specific and defined by the user or algorithm.These algorithms terminate when a current configurationdoes not have any improving neighbor and hence is locallyoptimal. The disadvantage of local search algorithms isthat, depending on the search space and initial configu-ration, they can terminate with a locally optimal configu-ration that performs much worse than the globally optimalconfiguration. Examples include the Nelder–Mead simplex,orthogonal search, and variable neighborhood search.

Several global and local search algorithms have beendeployed for autotuning. Seymour et al. [52] performedan experimental comparison of several global (randomsearch, a genetic algorithm, simulated annealing, particleswarm) and local (Nelder–Mead and orthogonal search)




Fig. 2. Illustration of model-based search. (left) At each iteration, the algorithm considers a set of evaluated configurations, (middle) fits a

surrogate model, and (right) evaluates configurations that are predicted to be high-performing by the model.

optimization algorithms. Similarly, Kisuki et al. [53] com-pared random search, a genetic algorithm, and simu-lated annealing with pyramid search and window search.In both these studies, the experimental results showedthat the random search was more effective than theother algorithms tested. The reason is that in the tuningtasks considered, the number of high-performing para-meter configurations is large and hence finding one ofthem is easy. Moreover, in all these works the localsearch algorithms are less effective since they were notcustomized. Norris et al. [54] implemented the Nelder–Mead simplex method, simulated annealing, and a geneticalgorithm in the empirical performance-tuning frame-work Orio. A number of previous works deploy localsearch algorithms for empirical performance tuning. Exam-ples include orthogonal search in ATLAS [55], patternsearch in loop optimization [56], and a modified Nelder–Mead simplex algorithm in Active Harmony [27], [57].Balaprakash et al. [58] investigated the issue of globalversus local search in autotuning using illustrative globaland local algorithms under short computation times. Theresults showed that the exploration capabilities of globalalgorithms are less useful; given good initial configura-tions, local search algorithms can find high-performingcode variants in short computation time. Moreover, poorinitial configurations can significantly reduce the effec-tiveness of both global and local search algorithms thatare sensitive to the starting point. When the availabletuning time is severely limited, carefully customized localsearch algorithms are promising candidates for empiricalperformance tuning problems that have integer parametersand bound constraints.

The primary goal of model-based selection algorithms inautotuning is to avoid the cost of running code on the tar-get machine by predicting performance metrics of a givenparameter configuration. Analytical performance models,which use closed-form expressions for predicting perfor-mance metrics, have enjoyed significant success in thecompiler optimization community for accelerating serialcodes. However, this approach is limited by the quality andextrapolatory power of the analytical model, which oftenfails to capture complex interactions between the code,

runtime systems, and architecture. Moreover, developinga complex mathematical model requires a wide range ofexpertise in the target system architecture, programmingmodels, and scientific applications. Consequently, analyt-ical models are less well suited for highly specializedkernels and libraries for scientific applications that requireportability, scalability, and performance. Another analyticalmodel-based autotuning approach involves analysis of thesource or binary code of the implementation to estimateanalytical model parameters. For example, a static analysistool can extract control flow information and instructioncounts from the compiled PTX code of a CUDA GPU imple-mentation, from which one can estimate (analytically)metrics such as occupancy, which can be used to determineparameters such as thread counts and block sizes. Whilethe autotuner still must generate different code versionsthat are compiled before the static analysis can be applied,this approach greatly reduces or in many cases completelyeliminates the need for executing and timing code variants.

When analytical performance models become toorestrictive for a given scientific workload and HPC archi-tecture, empirical performance modeling is an effectivealternative. In this approach, a small subset of parameterconfigurations (code variants) is evaluated on the tar-get machine to measure the required performance met-rics, and a predictive model is built by using machinelearning approaches. Here, the choice of the supervisedmachine learning algorithm for building the surrogate per-formance model is crucial. Often this choice is driven by anexploratory analysis of the relationship between the para-meter configurations and their corresponding runtimes.A typical model-based approach is a two-step process inwhich an analytical or empirical model is built first and asearch algorithm is used to find high-performing configu-rations using the model.

In recent years, a new class of empirical model-basedsearch has received considerable attention and has beenshown to be effective for autotuning. This approach con-sists of sampling a small number of input parameterconfigurations and progressively fitting a surrogate modelover the input–output space until exhausting the user-defined maximum number of evaluations. The surrogate




model is iteratively refined in the promising input para-meter region by obtaining new output metrics at inputconfigurations that are predicted to be high performing bythe model [59]–[61].

V. C A S E S T U D I E S

We describe a number of case studies from our priorwork that illustrate the current role of autotuning in HPCapplications.

A. Library Autotuning

Some libraries, such as PETSc [62]–[64], provide high-level data structure-independent interfaces that presentan opportunity for seamless integration of new, optimizeddata structures and low-level operations without havingto modify the application source code. Taking advantageof this design, researchers developed new matrix andvector data structures for PETSc stencil-based computa-tions specifically targeting GPUs. Orio [30] was used totune all matrix-vector operations involving the new datastructure [31]. Because the matrix structure is knowna priori and does not change, it can be represented byusing a packed dense format (instead of the typical com-pressed sparse row format), which is more storage-efficientand eliminates the indirect memory accesses that makesparse matrix algebra difficult to optimize. Any applica-tion involving a finite-difference, discretization-based PDEsolution on a regular grid can take advantage of the tunedimplementations without any code modification. More-over, because Orio generates size-specific optimizations,the resulting library generally performs better than manu-ally optimized libraries, which typically do not provide sizespecialization. In an evaluation of a PDE application dis-cretized by using a 3-D seven-point stencil, this approachachieved speedups of the matrix-vector computations rang-ing between 1.8 and 4.8 over vendor-optimized libraries(NVIDIA Cusp).

B. Solver Selection

Numerical toolkits such as PETSc and Trilinos [65] pro-vide a large number of parallel solution methods for largesparse linear systems. In the Lighthouse project [66]–[68],machine learning was used to classify solution methods(solver-preconditioner pairs) based on a small number(fewer than ten) of easy-to-compute linear system features.The classifiers are built through sampling the solver spaceon a large and varied training set of linear systems. Atruntime, the linear system features are used as input to themodel (classifier) to obtain a list of solver configurationsthat are likely to perform well. This algorithmic autotuningdoes not require application code change and is integratedinto the libraries’ existing solver interfaces.

C. End-to-End Autotuning

Another exercise demonstrated an automated, end-to-end optimization of the SMG2000 benchmark, a semi-coarsening multigrid on structured grids [69]. This

demonstration combined outlining using the ROSE com-piler, transformation and code generation using CHiLL,and search space navigation with Active Harmony. Withoutlining, ROSE extracts computationally intensive loopnests into separate executable functions with representa-tive input data that are to be the focus of autotuning. Theoutlined loop nests are then tuned by the framework andsubsequently integrated back into the application. Eachloop nest is optimized through a fixed series of composablecode transformations (permute, tile and unroll), with thetransformations parameterized by unbound optimizationparameters that are bound by Active Harmony during thetuning process. When the full application is run using thecode variant found by the system, overall performanceimproves by 27%.

D. Tuned Code Integrated Into the Application

Compiler-directed autotuning was used to optimizeNek5000, and the compiler-generated code was integratedinto the production application [70]. Nek5000, a spec-tral element code, spends the bulk of its computation inmatrix–matrix multiplication of small, rectangular matri-ces. Because BLAS libraries are typically optimized forlarge square matrices that exceed memory hierarchy capac-ity, there was significant opportunity to improve perfor-mance by specializing the generated code to the specificmatrix sizes arising in the application. The optimizationsapplied focus on SIMD code generation (for Intel SSE),register reuse, and instruction-level parallelism. Therefore,CHiLL was used to apply loop permutation and loopunrolling, to achieve a loop order that was best suitedfor SSE, and expose register reuse and instruction-levelparallelism to the Intel ICC native compiler. The smallsearch space that arose was explored in a brute-force man-ner. As presented in [70], we observed overall speedupsof up to 1.26X on the entire application running on 256nodes of Jaguar at Oak Ridge, and the optimized code wasintegrated into the production application.

Recent work expanded the scope of this optimizationof Nek5000 to the entire local_grad3 calculation thatsubsumes the matrix-matrix multiplication, and targetingan NVIDIA GPU [71]. Here, CUDA-CHiLL (a thin CUDAlayer added to CHiLL) and the SURF search algorithmin Orio were combined, along with a tensor contractionfrontend DSL called Octopi, to fully automate the GPUcode generation. This GPU code has not been adopted bythe application developers at the time of this writing.

E. Performance and Programmer Productivity

Compiler-directed autotuning in CHiLL was used tooptimize the Locally Optimal Block Preconditioned Conju-gate Gradient (LOBPCG) solver [72], and was shown tooutperform by 3% a manually tuned code for the samealgorithm [73]. Specifically, an important kernel withinLOBPCG is the sparse matrix multivector multiplication(SpMM), which is a generalization of the SpMV kernel in




which a sparse m-by-n matrix A is multiplied by a tall andnarrow dense n-by-k matrix B (k � n). While the manuallyoptimized SpMM implementation was over 2000 lines ofcode, the input to CHiLL was only 7 lines of code, a 300×difference, thus improving programmer productivity. Theoptimizations required in CHiLL to replicate the manuallytuned code were extensive. Using an inspector-executorapproach, a data transformation converted the large, sym-metric sparse matrix from a compressed sparse row formatto a compressed sparse block (CSB) format. This repre-sentation was well suited for parallelization of the matrixand its transpose, since it permitted storing only the uppertriangular portion. A number of additional transformationswere added to CHiLL, inspired by the manually tuned code.In order to reduce the data movement associated withindices of the matrices, a short integer was used as the typefor the matrices that pointed to the beginning of each CSBblock. Targets for AVX SIMD code generation were markedwith pragmas for the native Intel ICC compiler. To sum-marize, this experiment demonstrated that an autotuningcompiler could generate high-performance sparse matrixcode, but also that integration of transformations used inmanual tuning could greatly enhance the capability of suchcompilers.

F. Vertically Integrated Autotuning

Fast Fourier transforms are a critical part of many par-allel programs. FFTs require extensive communication andhave floating point requirements that necessitate carefulinstruction sequences to achieve good performance. Inaddition, an effective implementation requires overlappingcomputation and communication. Because of the need tooptimize these parameters for different processors and thetedious nature of properly tuning them, FFT has been apopular library for autotuning. Performance of FFT at scaledepends on a combination of the node-level computation,communication requirements, and the underlying commu-nication layer implementation, typically MPI. We refer tothe use of autotuning at multiple software layers as verticalintegration.

The Active Harmony system has been used to obtaina highly optimized FFT implementation using verticallyintegrated autotuning [74]. This approach includes acommunication optimization technique called dynamicpolling intervals. In many MPI implementations, to actu-ally overlap computation and communication, programsmust periodically call a polling routine to query whethera nonblocking communication operation has completed.This polling routine also must be called periodically toensure that the MPI implementation actually transfersdata. The frequency of polling can have a significantimpact on performance. If the polling is done too often,it results in wasted effort. However, if it is not done fre-quently enough, communication can stall. The optimizedFFT library allows this polling frequency to be tuned.

Overall, the autotuning approach encompasses 24 tun-able parameters: two communication tile sizes, two

Fig. 3. Strong scaling results for OFFT and other 3-D FFT libraries.

communication window sizes, eight places where pollingintervals are specified, and eight subtile sizes. Despite thelarge number of parameters (each of which have dozensof possible values), the Active Harmony system is able toconverge to an optimal configuration after trying about35 parameter combinations.

Fig. 3 shows the results of using the optimized libraryOFFT compared with other 3-D FFT algorithms. The resultsshown are for strong scaling from 128 to 32K cores onNERSC’s Edison system. The light gray line shows the idealspeedup. The solid black line shows the results for OFFT.The blue line shows the time for FFTW. Since the originalFFTW supports only a 1-D data decomposition, the resultsfor FFTW stop at 1024 cores. The red and green lines showthe results for running using DCMP and UPCF, respectively.The OFFT results are always faster than those of the otherimplementations, and the performance advantage growsas the number of cores is increased.

G. Fast, Data-Dependent Autotuning

In multiphysics, multimaterial models and in adaptivemesh refinement (AMR) codes, the same kernel may beexecuted on very different data structures. A time steploop in an AMR code may iterate over thousands ofpatches, each of different sizes and aspect ratios; andmultimaterial kernels may require more computation oncertain parts of a mesh and not on others. Fig. 4(a) showsthe range of runtimes for the top eight kernels in twohydrocodes: CleverLeaf, an Eulerian AMR hydrodynam-ics proxy application, as well as ARES, a multiphysicsArbitrary Lagrangian-Eulerian hydrodynamics code. Theruntimes can vary by orders of magnitude depending onthe input array sizes and aspect ratios and the particularexecution model (OpenMP, sequential, GPU) used for eachkernel.

These codes use RAJA [43], a performance portabilityframework developed at LLNL, which was extended withthe Apollo [45] autotuning framework. Apollo allows auser to train a lightweight decision model using machinelearning and to use it to select an execution model based




Fig. 4. Runtime of kernels in hydrocodes can vary by orders of magnitude depending on the execution policy (CPU, GPU, OpenMP, etc.) and

input arrays. We are able to achieve considerable speedup by dynamically tuning the execution policy based on input data. (a) Runtimes of

top kernels for different execution models. (b) CleverLeaf AMR, Sedov problem. (c) ARES, hotspot problem.

on the input arrays each time control passes over a RAJAkernel. These experiments chose between running a ker-nel sequentially or with OpenMP. CleverLeaf running theSedov problem [Fig. 4(b)] achieved up to 2.5x speedupfor a run on 256 cores, with the entire code using Apollo.ARES 4c achieved up to 15% speedup on 256 cores, withonly the Lagrange hydro phase using the autotuner. Inthese examples, speedup increases for larger-scale runs;the reason is that the domain becomes increasingly finelydecomposed with strong scaling, and with small patchesit is not worthwhile to pay the overhead of launching on-node OpenMP parallel regions. For these highly unstruc-tured domains, the code may iterate over thousands ofirregularly sized mesh patches in a single timestep, andone cannot possibly know the size and correct code variantwithout dynamic information.

VI. S O F T WA R E E N G I N E E R I N GC H A L L E N G E S F O R A U T O T U N I N G

For autotuning to succeed in production, it must integrateseamlessly into the application development process, andone must be able to maintain autotuned parts of largeapplications as the codes evolve. Indeed, the changesdemanded of the software development process are thebiggest obstacle to mainstream use of autotuning.

Because offline autotuning can involve many empiricalcompilations, it typically requires a significantly longercompilation process, and it may require execution of codevariants on the target platform. Thus it can severely impactthe speed with which a developer can iterate on the code.Having an option to disable lengthy compiles is essential.Perhaps more importantly, empirical autotuning burdensdevelopers with the task of managing the tuning process.This can be tedious for developers, as they must nowdecide how often to retune and how to manage profiledata.

Autotuners have historically complicated the buildprocess, even for simple codes. Designing autotuners to

leverage existing build infrastructure is ideal. For example,Orio can be used as a compiler wrapper (together withother wrappers such as those provided by MPI implemen-tations) to enable a single build configuration to be usedfor regular development and for autotuning. For largecodes, injecting compiler wrappers into all parts of thebuild may not be straightforward but we may be ableto leverage the work already done in build-from-sourcepackage management systems such as Spack [75]. Spackcan build over a thousand packages, and it provides aharness around each package’s build system that injectscompiler wrappers into the build. Depending on the hostbuild system, Spack may set the CC variable, patch thebuild, or explicitly set the compilers in the build. This couldbe a useful integration point for autotuners and wouldenable codes with potentially hundreds of dependencylibraries to be tuned easily.

Because the final autotuned binaries contain automati-cally generated code, debugging can be an issue, althoughthe debugging of any application that uses libraries devel-oped elsewhere is complicated by the presence of codethat may be less familiar or for which source code isnot available at all. The code generated by autotuners,on the other hand, is likely to have fewer bugs than dohuman-developed portions, and the availability of multipleversions also enables functional equivalence verificationduring autotuning.

VII. C H A L L E N G E S A N DF U T U R E D I R E C T I O N S

To summarize the challenges for widespread adoption,autotuning must become a standard or at least a commonpart of the build process for HPC programmers. Making ittransparent to end users seems to be the most desirableway to do this, but on the other hand autotuning is morelikely to achieve high performance with some support fromapplication or domain experts. Therefore, we envision aspectrum of possible interactions with autotuning by HPC




Table 2 Challenges to Autotuning Adoption

programmers, from complete transparency to completecontrol. Looking across this spectrum of options, Table 2summarizes the challenges to adoption we foresee andsome possible solutions, organized into three main cat-egories: 1) overhead concerns, which refers to compiletime, tuning time, and runtime overheads as well asincreased programming complexity; 2) scope of applicabil-ity, which acknowledges that widespread use of autotuningrequires that its scope of applicability must be expanded tonew and dynamically changing execution contexts, learn-ing from prior application runs; and 3) other program-mer concerns, which we have collected from applicationscommunities.

Perhaps the first concern that is often raised aboutautotuning is the inherent cost of searching across differentimplementations and its scalability for large applicationcodes (line 1). Section IV addresses how improvements insearch algorithms lead to better solutions and less searchtime, but the tolerance for search time may vary by userand tuning scenario. While we improve search algorithmsand incorporate incremental tuning support as describedin the next paragraph, it is also important to offer a hardcutoff, in terms of number of points to search or time tospend tuning, to limit the cost of autotuning.

For users who want to maintain some control of theautotuning process, ease of use will be advanced byimproving the mechanisms by which autotuning variantsand parameters are expressed (lines 2 and 3). Autotuningsystems must first and foremost allow programmers tocompactly describe a search space or derive it automat-ically. When selecting among code variants using a classi-fier, manually specifying features may also be too much of aprogramming burden, potentially costly and suboptimal. Itis desirable to automate the feature collection, for examplewithin a common framework like Apollo’s collection ofRAJA features from Section V-G or via a domain-specificframework. If the programmer is providing any of thisinformation, then tools must support the programmer inlogically mapping the vast performance or other dataarising from autotuning back to programmer abstractionsof the computation, so that the relationship between areasof the search space and optimization is understood. This

latter capability can be used to train programmers tounderstand how to establish effective autotuning searchspaces and participate in optimization.

Consider the requirements of users who may not want tobe involved in the autotuning process at all. Such users willbenefit from all the previous solutions whenever they arefully automated. Incremental or dynamic tuning (line 4)can hide the time spent in performing autotuning, as it isamortized over prior runs or within a single applicationrun. Dynamic tuning must be sparing in how much workis done in a run; therefore, it could greatly benefit fromprerun learning or integration with incremental tuningthat accesses the measurements from prior runs. Anotherstrategy for hiding the programmer burden of autotuningis to rely on domain-specific frameworks (e.g., Halide)that can be tuned by expert users. The effect of tuningcan benefit other users, but they can use the harnesses orresults of autotuning without directing it.

The next two challenges in the table (lines 5 and 6)involve changes in applications, their input data andresources, which we refer to as the context or executioncontext of an application. As discussed in the precedingsection, autotuning must adapt to these changes and musttherefore be integrated into an application’s build process.Further, an exascale platform, because of energy manage-ment and component failure, may have varying resourcesavailable to an application for execution. In addition,offline tuning or training may need to be performed ona proxy system rather than the target architecture. Bothchallenges imply that autotuning decisions may need topredict expected resources at runtime and tune accord-ingly. Early work in predicting performance for unseenexecution contexts has relied on information from priortuning on source architectures that are different from thefinal target architecture [76], [77]. Such models work bestwhen performance differences can be captured with prox-ies; for example, if synchronization costs are the dominantpredictor for autotuning, then proxies for synchronizationcan be used to predict execution-time behavior. Findings todate, however, show that prediction is most effective whentraining or offline tuning occurs on a similar architectureor with related resources.




Building good tuning models requires exploring alarge performance search space, and a single HPC useris unlikely to be able to train high-quality, general tuningmodels with only a limited set of applications and inputs.Users will be able to train robust performance modelsonly if autotuning achieves broad exposure and if theperformance of many different algorithms, inputs, andarchitectures are included in the training data (line 7).To gather such a corpus of data, we must make it easy tomonitor code performance in a wide variety of contextsand to record this data in a way that can be sharedamong HPC users to train versatile models. However,typical HPC users do not want to be in the business ofmanaging voluminous performance data or of controllingwhich runs are monitored and measured and which arenot. The only way we can achieve this kind of scale iswith help from the HPC facilities–who have visibilityacross the entire workload of their HPC centers and candeploy tools to measure a wide range of applications.This is a difficult task that requires not only technicalwork but also work to secure the performance data frompotentially sensitive codes. Ideally, users could use simple,transparent interfaces to measure their production codesand to provide performance data securely to HPC facilitiesat runtime. Performance data management for autotunersshould not require developers to manually manage anyhistorical tuning data or model outputs.

Autotuning can alleviate the burden of finding a high-performance code variant, but application developers musttrust that the optimized version of the code producesequivalent results, particularly if tuning is performeddynamically (line 8). The outputs need not match iden-tically in order to be functionally equivalent, particularlyin the presence of reordering operations like reductions.A number of ways exist for verifying functional equivalenceof the tuned code. The most common simple approach isto compare output from some trusted version, with somespecified tolerance for differences in results. A preferredsolution may be to exploit domain knowledge of the algo-rithms to verify output. For example, LAPACK providescomprehensive error bounds for most computed quantities.It includes a table for various routines that describes thebound so a user can determine how accurate the computedsolution is for a given problem. If autotuning was involved,the resulting software could be checked according to thesame criteria. Other features of the output or executioncould be used as a proxy for verification, such as numberof iterations to convergence, or sensitivity to input pertur-bations. A more comprehensive approach could use codesynthesis and formal correctness proofs to automaticallygenerate complicated members of the design space, butthis may be substantially more costly.

The final line reflects a common concern among HPCdevelopers that impedes adoption of new technology.HPC applications have long lifetimes, sometimes span-ning decades. In contrast, new technology goes througha lengthy process of exploring appropriate approaches

before it gains traction. Many tools simply do not reach alevel of maturity to support production applications. Othertools may no longer be supported once a funded projectends. Therefore, to adopt new technology, HPC developersmust be convinced that technology will be available forthe lifetime of the application. The only obvious solutionto this problem is to integrate the technology into trustedand widely used open source software.

VIII. S U M M A R Y

Autotuning is a proven technology to achieve high per-formance and performance portability. In this paper, wehave presented examples where autotuning tools havefacilitated high-performance implementations. Program-mer productivity is also enhanced when tools can simplifythe code the programmer writes and eliminate the need formanually writing low-level architecture-specific implemen-tations. As new and diverse architecture features continueto appear in exascale architectures and beyond, the needfor approaches that improve performance portability andprogrammer productivity will grow stronger. We believethat autotuning is a powerful and appropriate technologyfor addressing this need.

The goal of this paper was twofold: to describe the stateof the art in autotuning and to present future requirementsto move autotuning toward mainstream use in HPC. Webelieve that many of the challenges in gaining widespreaddeployment of autotuning relate to ease of use (includingoverhead), predictability, and a reimagining of its integra-tion into the application build process. We have seen otherfundamental changes to application development in HPCsucceed or fail based on these issues.

Looking to future architectures and applications, webelieve that the size of the space of desirable codevariants for autotuning likely will grow substantially asarchitectures change and software adapts in response.Today’s applications are currently undergoing rewrites andother adaptations to target emerging many-core, GPU,and heterogeneous architectures. New ways of organizingalgorithms, new data structures, and even fundamen-tally new algorithms with different numerical conver-gence and stability properties are appearing. Indeed, anew theory of communication-avoiding algorithms showshow to construct algorithms that do asymptotically lessdata movement, which are more efficient on currentarchitectures where communication costs are increasinglydominant.

Beyond HPC, autotuning technology is highly rele-vant for some emerging computation-intensive work-loads, such as deep learning and data analytics. In orderto improve the single-node performance, deep learningrequires highly optimized kernels for key computationaloperations. These include sparse matrix–vector, matrix–transpose–vector, matrix–vector–transpose, and matrix–matrix products. As the set of possible algorithms andimplementations continues to grow, tools for more easilygenerating members of this set become important.




R E F E R E N C E S[1] J. K. Hollingsworth, B. P. Miller, and J. Cargille,

“Dynamic program instrumentation for scalableperformance tools,” in Proc. Scalable High-Perform.Comput. Conf., May 1994, pp. 841–850.

[2] D. Boehme, “Caliper: Performance introspection forHPC software stacks,” in Proc. Supercomputing(SC), Salt Lake City, UT, USA, Nov. 2016,pp. 550–560.

[3] K. A. Huck, A. D. Malony, R. Bell, and A. Morris,“Design and implementation of a parallelperformance data management framework,” inProc. Int. Conf. Parallel Process. (ICPP), Jun. 2005,pp. 473–482.

[4] C. L. Lawson, R. J. Hanson, D. R. Kincaid, andF. T. Krogh, “Basic linear algebra subprograms forFortran usage,” ACM Trans. Math. Softw., vol. 5,no. 3, pp. 308–323, Sep. 1979.

[5] J. Dongarra, J. Du Croz, I. Duff, andS. Hammarling, “A set of level 3 basic linear algebrasubprograms,” ACM Trans. Math. Softw., vol. 16,no. 1, pp. 1–17, Mar. 1990.

[6] J. Bilmes, K. Asanovic, C. W. Chin, and J. Demmel,“Optimizing matrix multiply using PHiPAC: Aportable, high-performance, ANSI C codingmethodology,” in Proc. Int. Conf. Supercomput.,Vienna, Austria, Jul. 1997, pp. 253–260.

[7] R. C. Whaley and J. J. Dongarra, “Automaticallytuned linear algebra software,” in Proc. ACM/IEEEConf. Supercomput. (SC), 1998, pp. 1–27.

[8] J. Kurzak, S. Tomov, and J. Dongarra, “AutotuningGEMM kernels for the Fermi GPU,” IEEE Trans.Parallel Distrib. Syst., vol. 23, no. 11,pp. 2045–2057, Nov. 2012.

[9] M. Püschel, “SPIRAL: Code generation for DSPtransforms,” Proc. IEEE, vol. 93, no. 2, pp. 232–275,Feb. 2005.

[10] V. Eijkhout, P. Bientinesi, and R. van de Geijn,“Proof-driven derivation of Krylov solver libraries,”Texas Adv. Comput. Center, Univ. Texas Austin,Austin, TX, USA, Tech. Rep. TR-10-02, 2010.

[11] D. Fabregat-Traver and P. Bientinesi,“Application-tailored linear algebra algorithms: Asearch-based approach,” Int. J. High Perform.Comput. Appl., vol. 27, no. 4, pp. 425–438,Nov. 2013.

[12] E.-J. Im and K. A. Yelick, “Optimizing sparse matrixvector multiplication on SMPs,” in Proc. SIAM Conf.Parallel Process. Sci. Comput., San Antonio, TX,USA, Mar. 1999, pp. 1–9.

[13] R. Vuduc, J. W. Demmel, and K. A. Yelick, “OSKI: Alibrary of automatically tuned sparse matrixkernels,” J. Phys., Conf. Ser., vol. 16, no. 16,pp. 521–530, 2005.

[14] P. Sao, X. Liu, R. Vuduc, and X. Li, “A sparse directsolver for distributed memory Xeon phi-acceleratedsystems,” in Proc. Int. Parallel Distrib. Process. Symp.(IPDPS), Hyderabad, India, May 2015, pp. 71–81.

[15] J. Li, J. Choi, I. Perros, J. Sun, and R. Vuduc,“Model-driven sparse CP decomposition forhigher-order tensors,” in Proc. IEEE Int. ParallelDistrib. Process. Symp. (IPDPS), May/Jun. 2017,pp. 1048–1057.

[16] M. Frigo and S. G. Johnson, “FFTW: An adaptivesoftware architecture for the FFT,” in Proc. Int. Conf.Acoust., Speech, Signal Process., vol. 3, May 1998,pp. 1381–1384.

[17] M. Frigo and S. G. Johnson, “The design andimplementation of FFTW3,” Proc. IEEE, vol. 93,no. 2, pp. 216–231, Feb. 2005.

[18] G. Baumgartner, “Synthesis of high-performanceparallel programs for a class of ab initio quantumchemistry models,” Proc. IEEE, vol. 93, no. 2,pp. 276–292, Feb. 2005.

[19] Y. Tang, R. A. Chowdhury, B. C. Kuszmaul,C.-K. Luk, and C. E. Leiserson, “The pochoir stencilcompiler,” in Proc. Symp. Parallel AlgorithmsArchitect. (SPAA), Jun. 2011, pp. 117–128.

[20] M. Christen, O. Schenk, and H. Burkhart, “PATUS:A code generation and autotuning framework forparallel iterative stencil computations on modernmicroarchitectures,” in Proc. IEEE Parallel Distrib.Process. Symp. (IPDPS), Anchorage, AK, USA, May2011, pp. 676–687.

[21] M. Frigo, “A fast Fourier transform compiler,” inProc. ACM SIGPLAN Conf. Program. Lang. Des.Implementation, May 1999, vol. 34, no. 5,pp. 169–180.

[22] J. Bilmes, K. Asanovic, C.-W. Chin, and J. Demmel,“Author retrospective for optimizing matrixmultiply using PHiPAC: A portablehigh-performance ANSI C coding methodology,” inProc. 25th Anniversary Volume Int. Conf.Supercomput. (ICS), 2014, pp. 42–44.

[23] R. C. Whaley and J. Dongarra (2016). SC16 Test ofTime Award Winner: Official Citation. [Online].Available:http://sc16.supercomputing.org/conference-components/awards/test-time-award-page/

[24] M. Frigo and S. G. Johnson (1999). J. H. WilkinsonPrize for Numerical Software: Official List ofWinners. [Online]. Available:http://www.anl.gov/mcs/about-us/j-h-wilkinson-prize-numerical-software

[25] A. Heinecke, G. Henry, M. Hutchinson, andH. Pabst, “LIBXSMM: Accelerating small matrixmultiplications by runtime code generation,” inProc. ACM/IEEE Int. Conf. High-Perform. Comput.Netw., Storage Anal., Nov. 2016, pp. 981–991.

[26] C. Chen, “Model-guided empirical optimization formemory hierarchy,” Ph.D. dissertation, Univ.Southern California, Los Angeles, CA, USA, 2007.

[27] A. Tiwari, C. Chen, C. Jacqueline, M. Hall, andJ. K. Hollingsworth, “A scalable auto-tuningframework for compiler optimization,” in Proc. IEEEInt. Symp. Parallel Distrib. Process., Washington,DC, USA, May 2009, pp. 1–12.

[28] A. Hartono, B. Norris, and P. Sadayappan,“Annotation-based empirical performance tuningusing Orio,” in Proc. IPDPS, May 2009, pp. 1–11.

[29] Q. Yi, K. Seymour, H. You, R. Vuduc, andD. Quinlan, “POET: Parameterized optimizationsfor empirical tuning,” in Proc. IPDPS, Long Beach,CA, USA, Mar. 2007, pp. 1–8.

[30] A. Mametjanov, D. Lowell, C.-C. Ma, and B. Norris,“Autotuning stencil-based computations on GPUs,”in Proc. IEEE Cluster, Sep. 2012, pp. 266–274.

[31] C. Choudary, “Stencil-aware GPU optimization ofiterative solvers,” SIAM J. Sci. Comput., vol. 35,no. 5, pp. S209–S228, Oct. 2013.

[32] S. Donadio, “A language for the compactrepresentation of multiple program versions,” inProc. Workshop Lang. Compilers Parallel Comput.(LCPC), Oct. 2005, pp. 136–151.

[33] M. Hall, J. Chame, C. Chen, J. Shin, G. Rudy, andM. M. Khan, “Loop transformation recipes for codegeneration and auto-tuning,” in Proc. 22nd Int.Workshop Lang. Compilers Parallel Comput.,Oct. 2009, pp. 50–64.

[34] H. Takizawa, S. Hirasawa, Y. Hayashi, R. Egawa,and H. Kobayashi, “Xevolver: An XML-based codetranslation framework for supporting HPCapplication migration,” in Proc. 21st Int. Conf. HighPerform. Comput. (HiPC), Dec. 2014, pp. 1–11.

[35] J. Ansel, “PetaBricks: A language and compiler foralgorithmic choice,” in Proc. ACM SIGPLAN Conf.Program. Lang. Des. Implement. (PLDI). New York,NY, USA: ACM, 2009, pp. 38–49.

[36] M. Ren, J. Y. Park, M. Houston, A. Aiken, andW. J. Dally, “A tuning framework forsoftware-managed memory hierarchies,” in Proc.Int. Conf. Parallel Archit. Compilation Techn.,Oct. 2008, pp. 280–291.

[37] S. Muralidharan, M. Shantharam, M. Hall,M. Garland, and B. Catanzaro, “Nitro: A frameworkfor adaptive code variant tuning,” in Proc. IEEE28th Int. Parallel Distrib. Process. Symp. (IPDPS),May 2014, pp. 501–512.

[38] Y. Ding, J. Ansel, K. Veeramachaneni, X. Shen,U.-M. O’Reilly, and S. Amarasinghe, “Autotuningalgorithmic choice for input sensitivity,” in Proc.36th ACM SIGPLAN Conf. Program. Lang. Des.Implement. (PLDI), 2015, pp. 379–390.

[39] R. Nair, S.-L. Bernstein, E. Jessup, and B. Norris,“Generating customized sparse eigenvaluesolutions with Lighthouse,” in Proc. 9th Int.Multi-Conf. Comput. Global Inf. Technol., Seville,

Spain, Jun. 2014, pp. 1–4.[40] A. Tiwari and J. K. Hollingsworth, “Online adaptive

code generation and tuning,” in Proc. Int. Conf.Parallel Distrib. Process. Syst., May 2011,pp. 879–892.

[41] M. J. Voss and R. Eigemann, “High-level adaptiveprogram optimization with ADAPT,” in Proc. ACMPrinciples Pract. Parallel Program., Jun. 2001,pp. 93–102.

[42] I.-H. Chung and J. K. Hollingsworth, “A case studyusing automatic performance tuning for large-scalescientific programs,” in Proc. 15th IEEE Int. Conf.High Perform. Distrib. Comput., Jun. 2006,pp. 45–56.

[43] R. D. Hornung and J. A. Keasler, “The RAJAportability layer: Overview and status,” LawrenceLivermore Nat. Lab., Livermore, CA, USA, Tech.Rep. LLNL-TR-661403, Sep. 2014.

[44] H. C. Edwards, C. Trott, and D. Sunderland,“Kokkos: A manycore device performanceportability library for C++ HPC applications,” inProc. Workshop Program. Abstractions Data Locality.Livermore, CA, USA: Sandia National Laboratories,Mar. 2014, pp. 1–37.

[45] D. Beckingsale, O. Pearce, I. Laguna, andT. Gamblin, “Apollo: Fast, dynamic tuning fordata-dependent code,” in Proc. IEEE Int. ParallelDistrib. Process. Symp. (IPDPS), Orlando, FL, USA,May/Jun. 2017.

[46] J. Ragan-Kelley, C. Barnes, A. Adams, S. Paris,F. Durand, and S. Amarasinghe, “Halide: Alanguage and compiler for optimizing parallelism,locality, and recomputation in image processingpipelines,” in Proc. 34th ACM SIGPLAN Conf.Program. Lang. Des. Implement. (PLDI), 2013,pp. 519–530.

[47] K. J. Brown, “A heterogeneous parallel frameworkfor domain-specific languages,” in Proc. Int. Conf.Parallel Archit. Compilation Techn., Oct. 2011,pp. 89–100.

[48] S. Williams, J. Carter, L. Oliker, J. Shalf, andK. Yelick, “Optimization of a lattice Boltzmanncomputation on state-of-the-art multicoreplatforms,” J. Parallel Distrib. Comput., vol. 69,no. 9, pp. 762–777, Sep. 2009.

[49] K. Datta, S. Kamil, S. Williams, L. Oliker, J. Shalf,and K. Yelick, “Optimization and performancemodeling of stencil computations on modernmicroprocessors,” SIAM Rev., vol. 51, no. 1,pp. 129–159, 2009.

[50] S. Williams, L. Oliker, R. Vuduc, J. Shalf, K. Yelick,and J. Demmel, “Optimization of sparsematrix–vector multiplication on emergingmulticore platforms,” Parallel Comput., vol. 35,no. 3, pp. 178–194, Mar. 2009.

[51] J. Shin, M. W. Hall, J. Chame, C. Chen, andP. D. Hovland, “Autotuning and specialization:Speeding up matrix multiply for small matriceswith compiler technology,” in Proc. 4th Int.Workshop Autom. Perform. Tuning, Japan, 2009,pp. 353–370.

[52] K. Seymour, H. You, and J. Dongarra, “Acomparison of search heuristics for empirical codeoptimization,” in Proc. IEEE Int. Conf. ClusterComput., Sep./Oct. 2008, pp. 421–429.

[53] T. Kisuki, P. M. W. Knijnenburg, and M. F. P. O’Boyle,“Combined selection of tile sizes and unroll factorsusing iterative compilation,” in Proc. Int. Conf.Parallel Archit. Compilation Techn., Washington,DC, USA, Oct. 2000, pp. 237–246.

[54] B. Norris, A. Hartono, and W. Gropp, Annotationsfor Productivity and Performance Portability(Computational Science). Boca Raton, FL, USA:CRC, 2007, pp. 443–461.

[55] R. C. Whaley and J. J. Dongarra, “Automaticallytuned linear algebra software,” in Proc. ACM/IEEEConf. Supercomput. (SC), Washington, DC, USA,Nov. 1998, pp. 1–27.

[56] A. Qasem, K. Kennedy, and J. Mellor-Crummey,“Automatic tuning of whole applications usingdirect search and a performance-basedtransformation system,” J. Supercomput., vol. 36,no. 2, pp. 183–196, May 2006.




[57] V. Tabatabaee, A. Tiwari, and J. K. Hollingsworth,“Parallel parameter tuning for applications withperformance variability,” in Proc. ACM/IEEE Conf.Supercomput. (SC), Washington, DC, USA, 2005,p. 57.

[58] P. Balaprakash, S. M. Wild, and P. D. Hovland, “Anexperimental study of global and local searchalgorithms in empirical performance tuning,” inProc. 10th Int. Conf. Revised Sel. Papers HighPerform. Comput. Comput. Sci. (VECPAR). Springer,2013, pp. 261–269.

[59] T. Nelson, “Generating efficient tensor contractionsfor GPUs,” in Proc. 44th Int. Conf. Parallel Process.(ICPP), Sep. 2015, pp. 969–978.

[60] P. Balaprakash, S. M. Wild, and P. D. Hovland, “Cansearch algorithms save large-scale automaticperformance tuning?” in Proc. Int. Conf. Comput.Sci. (ICCS), vol. 4, 2011, pp. 2136–2145.

[61] J. Bergstra, N. Pinto, and D. Cox, “Machinelearning for predictive auto-tuning with boostedregression trees,” in Proc. Innov. Parallel Comput.(InPar), May 2012, pp. 1–9.

[62] S. Balay, W. D. Gropp, L. C. McInnes, andB. F. Smith, “Efficient management of parallelism inobject-oriented numerical software libraries,” inModern Software Tools for Scientific Computing,E. Arge, A. M. Bruaset, and H. P. Langtangen, Eds.Cambridge, MA, USA: Birkhäuser, 1997,pp. 163–202.

[63] S. Balay (2015). PETSc Web Page. [Online].

Available: http://www.mcs.anl.gov/petsc[64] S. Balay, “PETSc users manual,” Argonne Nat. Lab.,

Lemont, IL, USA, Tech. Rep. ANL-95/11, 2015.[Online]. Available: http://www.mcs.anl.gov/petsc

[65] M. A. Heroux, “An overview of the Trilinos project,”ACM Trans. Math. Softw., vol. 31, no. 3,pp. 397–423, Sep. 2005.

[66] K. Sood, B. Norris, and E. Jessup, “Lighthouse: Ataxonomy-based solver selection tool,” in Proc. 2ndWorkshop Softw. Eng. Parallel Syst. (SEPS),Pittsburgh, PA, USA, Oct. 2015, pp. 66–70.

[67] P. Motter, K. Sood, E. Jessup, and B. Norris,“Lighthouse: An automated solver selection tool,”in Proc. 3rd Int. Workshop Softw. Eng. High Perform.Comput. Comput. Sci. Eng. (SEHPCCSE), Austin, TX,USA, Nov. 2015, pp. 16–24.

[68] E. Jessup, P. Motter, B. Norris, and K. Sood,“Performance-based numerical solver selection inthe Lighthouse framework,” SIAM J. Sci. Comput.,vol. 38, no. 5, pp. S750–S771, 2016.

[69] A. Tiwari, “Auto-tuning full applications: A casestudy,” Int. J. High Perform. Comput. Appl., vol. 25,no. 3, pp. 286–294, Aug. 2011.

[70] J. Shin, M. W. Hall, J. Chame, C. Chen, P. F. Fischer,and P. D. Hovland, “Speeding up Nek5000 withautotuning and specialization,” in Proc. 24th ACMInt. Conf. Supercomput. (ICS), 2010, pp. 253–262.

[71] T. Nelson, “Generating efficient tensor contractionsfor GPUs,” in Proc. 44th Int. Conf. Parallel Process.,Sep. 2015, pp. 969–978.

[72] K. Ahmad, A. Venkat, and M. Hall, “OptimizingLOBPCG: Sparse matrix loop and datatransformations in action,” in Proc. 29th Int.Workshop Lang. Compilers Parallel Comput.,C. Ding, J. Criswell, and P. Wu, Eds.Springer-Verlag, 2016, pp. 218–231.

[73] H. M. Aktulga, A. Buluc, S. Williams, and C. Yang,“Optimizing sparse matrix-multiple vectorsmultiplication for nuclear configuration interactioncalculations,” in Proc. IEEE 28th Int. Parallel Distrib.Process. Symp., May 2014, pp. 1213–1222.

[74] S. Song and J. K. Hollingsworth,“Computation–communication overlap andparameter auto-tuning for scalable parallel 3-DFFT,” J. Comput. Sci., vol. 14, pp. 38–50,May 2016.

[75] T. Gamblin, “The Spack package manager: Bringingorder to HPC software chaos,” in Proc.Supercomput. (SC), Austin, TX, USA, Nov. 2015,pp. 1–12. [Online]. Available:http://tgamblin.github.io/pubs/spack-sc15.pdf

[76] A. Roy, P. Balaprakash, P. D. Hovland, andS. M. Wild, “Exploiting performance portability insearch algorithms for autotuning,” in Proc. IEEE Int.Parallel Distrib. Process. Symp. Workshops,May 2016, pp. 1535–1544.

[77] S. Muralidharan, A. Roy, M. Hall, M. Garland, andP. Rai, “Architecture-adaptive code variant tuning,”in Proc. 21st Int. Conf. Archit. Support Program.Lang. Oper. Syst. (ASPLOS), 2016, pp. 325–338.

A B O U T T H E A U T H O R S

Prasanna Balaprakash received the B.S.degree in computer science engineeringfrom the Periyar University, Salem, India,the M.S. degree in computer science fromthe Otto-von-Guericke University, Magde-burg, Germany, and the Ph.D. degree inengineering sciences from CoDE-IRIDIA (AILab), Université libre de Bruxelles, Brussels,Belgium.He was a Marie Curie Fellow and later an FNRS Aspirant at AI Lab.

Currently, he is a Computer Scientist with a joint appointment inthe Mathematics and Computer Science Division and the Leader-ship Computing Facility, Argonne National Laboratory. His researchinterests span the areas of artificial intelligence, machine learn-ing, optimization, and high-performance computing. Currently, hisresearch focus is on the automated design and development ofscalable algorithms for solving large-scale problems that arise inscientific data analysis and in automating application performancemodeling and tuning.

Jack Dongarra (Fellow, IEEE) holds anappointment at the University of Tennessee,Oak Ridge National Laboratory, and theUniversity of Manchester. He specializes innumerical algorithms in linear algebra, par-allel computing, use of advanced-computerarchitectures, programming methodology,and tools for parallel computers.Dr. Dongarra was awarded the IEEE Sid

Fernbach Award in 2004. In 2008, he was the recipient of thefirst IEEE Medal of Excellence in Scalable Computing; in 2010,he was the first recipient of the SIAM Special Interest Group onSupercomputing’s award for Career Achievement. In 2011, he wasthe recipient of the IEEE Charles Babbage Award, and in 2013 hereceived the ACM/IEEE Ken Kennedy Award. He is a Fellow of theAAAS, ACM, IEEE, and SIAM and a foreign member of the RussianAcademy of Science and a member of the U.S. National Academyof Engineering.

Todd Gamblin (Member, IEEE) receivedthe B.A. degrees in computer science andJapanese from Williams College, in 2002,and the M.S. and Ph.D. degrees in computerscience from the University of North Car-olina, Chapel Hill, in 2005 and 2009, respec-tively.He is a Computer Scientist in the Center

for Applied Scientific Computing, LawrenceLivermore National Laboratory, where he has been since 2008.His research focuses on scalable tools for measuring, analyzing,and visualizing parallel performance data. He is also the creatorof Spack, a popular HPC package management tool.

Mary Hall (Senior Member, IEEE) receivedthe B.A., M.S., and Ph.D. degrees incomputer science, all from Rice Univer-sity.Currently, she is a Professor in the School

of Computing at the University of Utah. Herresearch focuses on compiler technology forexploiting performance-enhancing featuresof a variety of computer architectures:automatic parallelization for multicores and GPUs, superword-level parallelism, processing-in-memory architectures, andFPGAs.




Jeffrey K. Hollingsworth (Senior Member,IEEE) received the B.S. degree in electricalengineering from the University of Califor-nia, Berkeley, CA, USA, and the M.S. andPh.D. degrees in computer sciences from theUniversity of Wisconsin.He is currently serving as Interm Chief

Information Officer of the University of Mary-land. He is a Professor in the ComputerScience Department, University of Maryland, College Park. In hisresearch, he seeks to develop a unified framework to understandthe performance of large systems and focuses on performancemeasurement and autotuning. He was Editor-in-Chief of the journalParallel Computing, was general chair of the SC12 Conference, andis Chair of ACM SIGHPC.

Boyana Norris received the B.S. degreefrom Wake Forest University and the Ph.D.degree from the University of Illinois atUrbana-Champaign, both in computer sci-ence.She is an Associate Professor in the Com-

puter and Information Science Departmentat the University of Oregon. Her researchin high-performance computing focuses onmethodologies and tools for performance reasoning and automatedoptimization of scientific applications, while ensuring continued orbetter usability of HPC tools and libraries and improving developerproductivity.

Richard Vuduc (Member, IEEE) receivedthe B.S. degree in computer science fromCornell University, Ithaca, NY, USA, andthe Ph.D. degree in computer science fromthe University of California, Berkeley, CA,USA.He is an Associate Professor in the School

of Computational Science and Engineering,Georgia Institute of Technology, Atlanta, GA,USA. His research lab, The HPC Garage (@hpcgarage), is interestedin high-performance computing, with an emphasis on algorithms,performance analysis, and performance engineering. From 2014 to2016, he served as an Associate Editor for the IEEE TRANSACTIONS

ON PARALLEL AND DISTRIBUTED SYSTEMS.


Date post:	23-Jan-2022
Category:	Documents
Upload:	others
View:	4 times
Download:	0 times

Autotuning in High- Performance Computing Applications

Documents