RETROSPECTIVE: Automatic Loop Interchange Randy Allen Catalytic Compilers 1900 Embarcadero Rd, #206 Palo Alto, CA. 94043 [email protected]Ken Kennedy Department of Computer Science Rice University Houston, TX. 77251 [email protected]Retrospectives provide a rare and interesting opportunity to reflect upon the past and recall (or more accurately after a couple of decades, speculate upon) our state of mind and understandings at earlier times. “Automatic Loop Interchange” was published almost 20 years ago at a midpoint in research on data dependence and program transformations. This paper pulled together the work of many predecessors [9,10,11,15] into a simple, clean theory, providing a checkpoint on earlier predictions on the power and applicability of data dependence. At the same time, the paper was published just as the field of data dependence entered a “golden age”. It is our hope that this paper helped catalyze the following research into data dependence theory and applications. A 1978-79 sabbatical at IBM catalyzed our initial interest in automatic vectorization. We started development with a version of the Parafrase system built at the University of Illinois by Dave Kuck and his group (including Michael Wolfe) [9,15], but we began work on an entirely new system in 1981 to provide a better platform for our research on multilevel code generation [8] (eventually published in 1987 [3]). The new system became known as the Parallel Fortran Converter (PFC). At the time we started the PFC project, few programmers had access to vectorizing compilers that used data dependence. Vector units and vectorizing compilers were employed exclusively on expensive high-end machines or specialized array processors, which were available to only a small percentage of the general programming public. Despite this limited access, vectorizing compilers had already earned the informal nicknames “paralyzers” and “terrorizers” due to their large compile times and often less-than-optimal output. We began our effort with modest expectations. PFC was deliberately structured as a source-to-source translator, primarily because we believed the algorithms that we wanted to employ would require more compile-time than could be justified in a production compiler. We also doubted the power of data dependence, and expected that we would need to employ techniques from artificial intelligence to achieve satisfactory results. This paper marked a point in PFC’s development where our early assumptions had been proved wrong. What PFC and this paper had shown was that a fairly simple set of program transformations based on a unified underlying theory could provide effective restructuring without requiring unacceptable compile times. 20 Years of the ACM/SIGPLAN Conference on Programming Language Design and Implementation (1979-1999): A Selection, 2003. Copyright 2003 ACM 1-58113-623-4 $5.00 Foundation Although this paper is entitled “Automatic Loop Interchange”, it is far broader in scope. As an introduction to interchange, the paper also covers a wide spectrum of dependence-based theory and transformations. This work is built on the efforts of many others, and we would be remiss if we did not acknowledge at least some of those efforts – acknowledging all of them would quickly blow our page limits. The earliest papers on dependence-based program transformations include papers by Lamport [10,11] and Kuck [9]. Lamport developed a form of loop interchange for use in vectorization, as well as the wavefront method for parallelization, an early form of what came to be called loop skewing. As indicated earlier, we also had access to the Parafrase system and the associated body of research. In particular, Michael Wolfe’s Master’s thesis focused on loop interchange [15], a topic that he developed further in later works [16,17]. Our own work on the subject began with our multilevel code generation strategy [8, 1, 2, 3], which we implemented in the summer of 1981. Real implementations often provide incredible insights into the weaknesses of theoretical approaches; this was definitely true in the case of PFC. The code generation strategy proved extremely effective in practice and was far more efficient in terms of compile-time than we had anticipated. 1 However, a real implementation quickly showed us that loop interchange was the key missing piece. While PFC performed well in terms of the vectorization it detected, we quickly saw that loop interchange was the key incremental transformation. The practical strategy presented in the paper (“innermosting” loops that carried no dependence, testing loops that carried dependences for interchange only to the next deeper position) evolved out of discussions with Randy Scarborough, Joe Warren, and others in the PFC project. The strategy reported in this paper was implemented in the PFC system. Although we reported no experimental results in the paper, a later study reviewed in our book [4] showed that PFC was able to do extremely well on the Callahan, Dongarra, and Levine vectorization tests [6]. 1 At that time, we had to pay for computer time by the CPU- minute. The first time that we tried a large test case (roughly 1000 lines of code), Ken insisted that we limit the CPU time to 10 minutes (which was still several thousand dollars of computer time) to avoid blowing our research budget. We didn’t expect the test case to complete in the time limit; when it took only 40 seconds, we assumed that PFC had crashed processing the input. It took us a day of wading through the output to verify that it had in fact completely and correctly processed the test. ACM SIGPLAN 75 Best of PLDI 1979-1999
Retrospectives provide a rare and interesting opportunity to reflect upon the past and recall (or more accurately after a couple of decades, speculate upon) our state of mind and understandings at earlier times. “Automatic Loop Interchange” was published almost 20 years ago at a midpoint in research on data dependence and program transformations. This paper pulled together the work of many predecessors [9,10,11,15] into a simple, clean theory, providing a checkpoint on earlier predictions on the power and applicability of data dependence. At the same time, the paper was published just as the field of data dependence entered a “golden age”. It is our hope that this paper helped catalyze the following research into data dependence theory and applications. A 1978-79 sabbatical at IBM catalyzed our initial interest in automatic vectorization. We started development with a version of the Parafrase system built at the University of Illinois by Dave Kuck and his group (including Michael Wolfe) [9,15], but we began work on an entirely new system in 1981 to provide a better platform for our research on multilevel code generation [8] (eventually published in 1987 [3]). The new system became known as the Parallel Fortran Converter (PFC). At the time we started the PFC project, few programmers had access to vectorizing compilers that used data dependence. Vector units and vectorizing compilers were employed exclusively on expensive high-end machines or specialized array processors, which were available to only a small percentage of the general programming public. Despite this limited access, vectorizing compilers had already earned the informal nicknames “paralyzers” and “terrorizers” due to their large compile times and often less-than-optimal output. We began our effort with modest expectations. PFC was deliberately structured as a source-to-source translator, primarily because we believed the algorithms that we wanted to employ would require more compile-time than could be justified in a production compiler. We also doubted the power of data dependence, and expected that we would need to employ techniques from artificial intelligence to achieve satisfactory results. This paper marked a point in PFC’s development where our early assumptions had been proved wrong. What PFC and this paper had shown was that a fairly simple set of program transformations based on a unified underlying theory could provide effective restructuring without requiring unacceptable compile times. 20 Years of the ACM/SIGPLAN Conference on Programming Language Design and Implementation (1979-1999): A Selection, 2003. Copyright 2003 ACM 1-58113-623-4 $5.00
Foundation Although this paper is entitled “Automatic Loop Interchange”, it is far broader in scope. As an introduction to interchange, the paper also covers a wide spectrum of dependence-based theory and transformations. This work is built on the efforts of many others, and we would be remiss if we did not acknowledge at least some of those efforts – acknowledging all of them would quickly blow our page limits. The earliest papers on dependence-based program transformations include papers by Lamport [10,11] and Kuck [9]. Lamport developed a form of loop interchange for use in vectorization, as well as the wavefront method for parallelization, an early form of what came to be called loop skewing. As indicated earlier, we also had access to the Parafrase system and the associated body of research. In particular, Michael Wolfe’s Master’s thesis focused on loop interchange [15], a topic that he developed further in later works [16,17]. Our own work on the subject began with our multilevel code generation strategy [8, 1, 2, 3], which we implemented in the summer of 1981. Real implementations often provide incredible insights into the weaknesses of theoretical approaches; this was definitely true in the case of PFC. The code generation strategy proved extremely effective in practice and was far more efficient in terms of compile-time than we had anticipated.1 However, a real implementation quickly showed us that loop interchange was the key missing piece. While PFC performed well in terms of the vectorization it detected, we quickly saw that loop interchange was the key incremental transformation. The practical strategy presented in the paper (“innermosting” loops that carried no dependence, testing loops that carried dependences for interchange only to the next deeper position) evolved out of discussions with Randy Scarborough, Joe Warren, and others in the PFC project. The strategy reported in this paper was implemented in the PFC system. Although we reported no experimental results in the paper, a later study reviewed in our book [4] showed that PFC was able to do extremely well on the Callahan, Dongarra, and Levine vectorization tests [6].
1 At that time, we had to pay for computer time by the CPU-minute. The first time that we tried a large test case (roughly 1000 lines of code), Ken insisted that we limit the CPU time to 10 minutes (which was still several thousand dollars of computer time) to avoid blowing our research budget. We didn’t expect the test case to complete in the time limit; when it took only 40 seconds, we assumed that PFC had crashed processing the input. It took us a day of wading through the output to verify that it had in fact completely and correctly processed the test.
ACM SIGPLAN 75 Best of PLDI 1979-1999
Impact The approaches to dependence and loop interchange presented in this paper were soon incorporated into a number of commercial compilers. We are directly aware of the implementations in the IBM compiler for the 3090 Vector Feature [13] and the Convex vectorizing compiler, and were involved in the implementation of the Ardent restructuring compilers. Beyond the immediate practical impact, “Automatic Loop Interchange” also established interchange as a fundamental transformation in all advanced optimizing compilers: vectorizing, parallelizing, and even scalar. While many previous papers had focused on dependences as execution constraints that limit reordering, this paper (in a section devoted to other applications of dependence) also pointed out the dual aspect: dependences represented reused memory locations. Accordingly, dependence provided a basis for optimizing for memory hierarchies by moving the most frequently accessed memory locations into the fastest elements of the hierarchy. Later research would prove loop interchange to be as important for moving dependences into inner loops (thereby optimizing memory reuse) as it had proven to be for moving dependences out of inner loops (as was necessary for vectorizing loops). Particularly important exemplars of this research are the papers by Callahan, Carr, and Kennedy on register optimization [6] and by Wolf and Lam on cache blocking [14]. Both papers are included in this volume. Practical implementations that included this aspect of dependence include the Ardent compiler [5].
Future Applications Looking back over the past 18 years, we doubt that we would have predicted the impact of loop interchange on the compiler literature. Although our own work and the work of others went on to more powerful transformation strategies based on direction and distance matrices [4, 14, 16, 17], this work was one of the first to establish that powerful and effective program transformations could be implemented in practical compiler systems. Of course, one reason for the growth in importance of this work is the increased use of parallelism in computer architecture and the increasing disparity between CPU and memory speeds. Looking to the future, we believe these factors are only going to increase in the design of computer systems, making these compiler techniques even more relevant. Memory hierarchies in particular are increasingly dominating computation times, and automatic loop interchange is a key transformation for exploiting that hierarchy. While loop interchange has been thoroughly explored in the context of restructuring compilers, there are other contexts which have not been so thoroughly explored. For instance, given the intimate relationship between dependence and loop iterations, it is natural to assume that dependence and loop interchange should have as important a role to play in the design of pipelined architectures as it does in exploiting pipelined architectures.
Acknowledgements As was the case at the time the paper was published, this work has progressed over the years only by the efforts and collaborations of others far too numerous to list here. However, we would be remiss if we did not acknowledge the contributionsof Randy Scarborough, Joe Warren, Horace Flatt, and all thegraduate students who worked on PFC.
Bibliography 1. J.R. Allen. Dependence analysis for subscripted variables
and its application to program transformations. Ph.D dissertation, Department of Mathematical Sciences, Rice University, May, 1983.
2. J. R. Allen and K. Kennedy. PFC: a program to convert Fortran to parallel form. In Supercomputers: Design and Applications, K. Hwang, editor, pages 186–203. IEEE Computer Society Press, August 1984.
3. J. R. Allen and K. Kennedy. Automatic translation of Fortran programs to vector form. ACM Transactions on Programming Languages and Systems, 9(4):491–542, October 1987.
4. R. Allen and K. Kennedy. Optimizing Compilers for Modern Architectures. Morgan Kaufmann, 2002.
5. R. Allen. Unifying vectorization, parallelization, and optimization: the Ardent compiler. In Proceedings of the Third International Conference on Supercomputing, 1988.
6. D. Callahan, S. Carr, and K. Kennedy. Improving register allocation for subscripted variables. In PLDI ’90 (also included in this volume).
7. D. Callahan, J. Dongarra, and D. Levine. Vectorizing compilers: A test suite and results. In Proceedings of Supercomputing ‘88, Orlando, FL, 1988.
8. K. Kennedy. Automatic translation of Fortran programs to vector form. Rice Technical Report 476-029-4, Department of Mathematical Sciences, Rice University, 1980.
9. D. Kuck, R. Kuhn, D. Padua, B. Leasure, and M. J. Wolfe. Dependence graphs and compiler optimizations. In Conference Record of the Eighth Annual ACM Symposium on the Principles of Programming Languages, Williamsburg, VA, January 1981.
10. L. Lamport. The parallel execution of DO loops. Communications of the ACM, 17(2):83–93, February 1974.
11. L. Lamport. The coordinate method for the parallel execution of iterative DO loops. Technical Report CA-7608-0221, SRI, Menlo Park, CA, August 1976, revised October 1981.
12. D. A. Padua and M. J. Wolfe. Advanced compiler optimizations for supercomputers. Communications of the ACM, 29(12):1184–1201, December 1986.
13. R. G. Scarborough and H. G. Kolsky. A vectorizing FORTRAN compiler. IBM Journal of Research and Development, March 1986.
14. M. E. Wolf and M. Lam. A data locality optimizing algorithm. In PLDI ’91 (also included in this volume).
15. M. J. Wolfe. Techniques for improving the inherent parallelism in programs. Master’s thesis, Dept.of Computer Science, University of Illinois at Urbana-Champaign, July 1978.
16. M. J. Wolfe. Advanced loop interchanging. In Proceedings of the 1986 International Conference on Parallel Processing, St. Charles, IL, August 1986.
17. M. J. Wolfe. High Performance Compilers for Parallel Computing. Addison-Wesley, Redwood City, CA, 1996.
