Abstract In December 2009 the 768-bit, 232-digit numberRSA-768 was factored using the number field sieve. Over-all, the computational challenge would take more than 1700years on a single, standard core. In the article we presentthe heterogeneous computing approach, involving different
T. Kleinjung · J.W. Bos · A.K. Lenstra (�) · D.A. OsvikEPFL IC LACAL, Station 14, 1015 Lausanne, Switzerlande-mail: [email protected]
K. AokiNTT, 3-9-11 Midori-cho, Musashino-shi, Tokyo, 180-8585, Japan
S. ContiniMacquarie University, Sydney, Australia
J. FrankeDepartment of Mathematics, University of Bonn, Beringstraße 1,53115 Bonn, Germany
E. ThoméINRIA CNRS LORIA, Équipe CARAMEL, bâtiment A, 615 ruedu jardin botanique, 54602 Villers-lès- Nancy Cedex, France
P. Jermini · M. ThiémardEPFL AI DIT, CP 121, 1015 Lausanne, Switzerland
P. LeylandBrnikat Ltd, 19a Hauxton Rd, Little Shelford, Cambridge,CB22 5HJ, UK
P.L. MontgomeryMicrosoft Research, One Microsoft Way, Redmond, WA 98052,USA
P.L. Montgomery · A. TimofeevCWI, P.O. Box 94079, 1090 GB Amsterdam, The Netherlands
H. StockingerSwiss Institute of Bioinformatics, Vital-IT Group,1015 Lausanne, Switzerland
compute clusters and Grid computing environments, used tosolve this problem.
Keywords RSA · HPC · Integer factorization
1 Introduction
Scientific calculations in domains such as physics (fluiddynamics, high-energy physics etc.), chemistry (quantumchemistry, molecular modeling, etc.), biology (large scalegenomic or proteomics projects), and climate modeling of-ten require massive amounts of computing power. If the kindof problem calls for tightly coupled massively parallel com-puting, it is often well suited for supercomputers and standsa good chance to attract funding both to get access to theproper hardware and to develop suitable software. In con-trast, embarrassingly parallel applications, where any num-ber of stand-alone devices will do, are often not welcomeon classical supercomputers but are instead relegated tohigh-performance computing clusters, Grid infrastructures,or desktop computing environments such as BOINC or Con-dor.
The latter category includes cryptographic applications.Although they are not less CPU-demanding than other sci-entific applications, they are not considered to be of muchinterest by the HPC community. Despite this disregard,many cryptographic problems are computationally, mathe-matically and algorithmically challenging and non-trivial,and practically relevant for users of cryptography (i.e., al-most everyone these days). One of these problems is inte-ger factorization, the subject of this paper: we present theheterogeneous computational infrastructure that was used toset a new integer factorization record by factoring the chal-lenge number RSA-768, a 768-bit, 232-digit composite in-teger [11].
We used the approach commonly used to factor large in-tegers since the late 1980s. Thus, we did not aim for a sin-gle, homogeneous computing environment or single super-computer, but used a heterogeneous compute environmentconsisting of several cluster and Grid environments (in Aus-tralia, Japan, and many European countries), different op-erating systems, job execution environments and technicalpersonnel. Several technical as well as non-technical reasonshave led us to a such a heterogeneous approach:
– Single Grid infrastructures such as the European Grid in-frastructure: only a limited number of resources (for alimited time) in the European Grid environment have beendedicated to the RSA-factorization problem. Therefore,even though using a Grid-only environment is promis-ing and convenient, we had to look into alternative ap-proaches.
– Individual clusters at different sites are available butare managed in different ways (i.e., different local re-source management systems, different file systems, dif-ferent hardware, different site expertise with respect tothe RSA application, etc.). Again, standardized Grid pro-tocols and middleware would have helped to addressthe problem in a more homogeneous way but the clus-ters we used are not part of a single Grid environ-ment.
– A diversified, heterogeneous environment better repre-sents the Internet as a whole than any organized computerinfrastructure. It thus allows a more reliable estimate ofthe havoc miscreants could cause by large surreptitiouscryptanalytic efforts.
In brief, the RSA-768 challenge has the following fea-tures and computational stages. The first main stage is themost CPU intensive and embarrassingly parallel part ofthe computation and also created most data. Compressedabout 5 TB had to be transferred over the Internet, at a rateof about 10 GB per day, to a central location where, onceenough data had been collected, the second main stage wasprepared. This stage is not embarrassingly parallel and wastraditionally done on a single supercomputer or, later, on asingle tightly coupled cluster. Lacking access to sufficientlylarge clusters for a long enough period of time, we had toadopt a somewhat more complicated approach that allowsusage of a limited number of tightly coupled clusters at dif-ferent locations. As a result, roughly 100 GB of data had tobe distributed, using the Internet again, to the contributingclusters (restricting to those in France, Japan, and Switzer-land). They worked independently, with the exception of anintermediate step that had to be done on a single tightly cou-pled cluster with 1 TB of RAM. This implied, yet again, asubstantial data exchange. In summary, the RSA-768 chal-lenge was solved using a truly heterogeneous infrastructure
with no communication among the contributors nor compu-tational cores except to break the computation into indepen-dent tasks.
Given the embarrassingly parallel nature of the first mainstage, the most CPU intensive part of the computation, wecould have used one of the tools that allow contributionsby volunteers on the Internet. That is how it was donein the late 1980s and early 1990s, using crude email andFTP-based precursors of currently popular tools. Indeed,an ongoing integer factorization BOINC project [14] doesjust that. Having managed several such proof-of-conceptprojects from 1988 until about 1995, we found that a lotof time goes into addressing simple concerns of the contrib-utors, in particular if considerable RAM resources are re-quired (as was the case for RSA-768 with at least one butpreferably two GB RAM per core). We chose to limit ourproject to a limited set of dedicated and knowledgeable re-searchers who could be expected to resolve occasional is-sues themselves.
Section 2 gives the cryptographic motivation, Sect. 3presents the computational steps in more detail, and Sect. 4describes the heterogeneous compute infrastructure used.
2 Cryptographic background and motivation
Cryptography is ubiquitous on the Internet. Authenticationmethods used by browsers, Grid computing applications,and websites accessed through HTTPS commonly rely onX.509 certificates based on the RSA cryptosystem. Thismeans that the security depends on the hardness of factoringan appropriately chosen integer, typically of 1024 or 2048bits: factoring that integer could undermine the security ofthat particular instance of the HTTPS protocol.
A 1024-bit integer has more than 300 decimal digits, a2048-bit one twice as many. Dealing with such large num-bers makes protocols unwieldy. Smaller numbers would bemore efficient—but they are easier to factor and give less se-curity. One would like to use the smallest key size that givesan acceptable level of security. Once a certain size has beenpicked it is hard to modify the choice. But integer factoringconstantly gets easier, not just because computers keep get-ting faster but also because factoring methods keep gettingbetter. So far this goes at a fairly steady pace. However, dis-covery of an efficient factoring method cannot be ruled out:it is conceivable that from one moment to the next the entireinformation security infrastructure collapses.1 This wouldaffect much more than just web-security, since most of the
1Integer factorization is easy on a quantum computer [16]. Quantumcomputers have not been realized yet. Estimates and opinions on thissubject vary.
Cluster Comput
public and private sectors use the same cryptographic meth-ods as HTTPS does. A “disaster” of this sort has not oc-curred yet—at least, not that we are aware of—and most ofus simply hope that it will not happen either.
The steady progress is taken into account in crypto-graphic standards that prescribe key sizes for RSA. At thispoint in time we are on the verge of an important transi-tion: the USA National Institute of Standards and Technol-ogy (NIST) recommends phasing out 1024-bit RSA by theend of the year 2010 and to adopt 2048-bit RSA or other sys-tems of comparable security. This does not mean that by theend of 2010 integers of 1024 bits can suddenly be factored. Itmeans that the security of 1024-bit RSA is perceived to be-come too low and that, indeed, several years down the road,factoring 1024-bit integers may become feasible.
How does our effort, factoring a 768-bit RSA challengekey, fit in this picture? We now know what effort sufficedto factor a 768-bit integer. Interestingly, it turned out to bean order of magnitude easier than predicted by some in thefield [15]. Combined with a theoretical analysis we can nowmore accurately predict what would be required to factor asingle 1024-bit RSA challenge. Unless an integer factoringbreakthrough occurs, we are convinced that an effort on ascale similar to ours will have no chance of success withinthe next five years. After that, all bets are off [11]. Thus,NIST rightly encourages phasing out 1024-bit RSA but thereis no need to rush into a costly, overhasty security upgrade.There is almost certainly no risk in a more economical grad-ual adoption of the new standards, as long as the transitionis complete by the year 2014.
Furthermore, our result shows that an effort of this sortcan be pulled off even if there is no uniformity in the plat-forms used or in the way they communicate. This requiresan extra degree of prudence when selecting cryptographickey sizes, as the enormous computational power of Internetas a whole, or a substantial fraction thereof, can in principlebe harnessed for similar cryptanalytic calculations.
A project of this sort is scientifically interesting in its ownright as well. When trying to solve larger problems thereare always new challenges that must be dealt with. For in-stance, for a previous large scale effort [1], when dividingthe second main step over a number of independent tightlycoupled clusters, we had not realized that the faster clusterswould finish their task much earlier than the slower ones,quite simply because we had never ran this step for such along time in such a heterogeneous environment. Faced withthe “threat” of long idle times on some clusters, a new algo-rithmic twist was developed allowing total flexibility in tasksizes, eliminating all idle times. It considerably facilitatedmanagement of the present project where the new approachwas used for the first time. This project, in turn, triggeredalgorithmic advances for new types of processors, adaptingnot just to multicores but also to the decreasing amount of
RAM per core. This will prove useful in later projects andwill greatly influence the feasibility of a 1024-bit factoringattempt.
3 The computational challenge
When faced with a factoring problem, one first checks forsmall factors. For RSA challenges this step can be omit-ted, since the RSA challenge numbers are constructed as theproduct of two primes of about the same size and thereforethey have no small factor. The fastest known algorithm tofactor RSA challenges is the number field sieve (NFS, [12]),which works by combining relations, as illustrated below.The two main steps mentioned in Sect. 1 are the most CPUintensive steps of NFS: in the first step relations are gen-erated, in the second step they are combined. In this sec-tion we present a more complete outline of NFS, concen-trating on the computational effort and data sizes for RSA-768, while avoiding all underlying mathematical details. Inbrief, we describe the five main steps of the overall work-flow.
We give a simple example to show how relations are com-bined to factor an integer while avoiding the intricacies in-volved in the NFS. For the integer 143 a relation would begiven by 172 ≡ 3 mod 143 because the difference 172 − 3 isan integer multiple of 143. Similarly, 192 ≡ 3 · 52 mod 143is a relation. These relations can be combined into the re-lation 172 · 192 ≡ 32 · 52 mod 143 with squares on bothsides. The square roots 17 · 19 and 3 · 5 of both sides fol-low immediately. The greatest common divisor of 143 andthe difference 17 · 19 − 3 · 5 of the square roots turns outto be 11, a factor of 143. Relations for NFS are more com-plex and involve algebraic integers. Defining those requiresproper polynomials, which are selected in the first step ofNFS:
Preparatory step: polynomial selection The runtime ofNFS depends strongly on the parameter choice. The mostimportant choice is that of a pair of irreducible polynomi-als f , g that define two algebraic number fields. For RSA-768 we fixed degree(f ) = 6 and degree(g) = 1, implyingthat the number field defined by g is the field Q of the ra-tional numbers. The best current method to find good f
and g is a mathematically rather sophisticated, embarrass-ingly parallel search. A good pair was found in 2005 already,after three months on 80 AMD Opteron cores in Bonn.A comparable effort at EPFL, in 2007, did not turn up a bet-ter pair. Overall, more than 2 ·1018 pairs were considered, ata rate of 1.6 billion pairs per core per second. Although thisis a considerable computation, as it would have required aday of computing on a 15 000 core cluster, it is dwarfed bythe other steps.
Cluster Comput
First main step: sieving In this step many relations aresought: co-prime pairs of integers a, b such that f (a, b) ·g(a, b) has no large prime factors. How many relations areneeded depends on the size of those prime factors. For RSA-768 we used the bound 240. We analyzed that enough rela-tions could be found by searching through |a| < 6.3 · 1011
and 0 < b < 1.4 · 107. This implies that for more than 1019
co-prime pairs a, b the value f (a, b) · g(a, b) had to betested for divisibility by the almost 38 billion primes < 240.Per prime p this can be done for many a, b pairs simultane-ously using sieving: if a polynomial value such as f (a, b) isa multiple of p, then so is f (a +mp,b +np) for integers m
and n.The sieving can be distributed, in an embarrassingly par-
allel fashion, by assigning disjoint ranges of b-values todifferent contributors. Given a b-value one just sieves all|a| < 6.3 · 1011. This straightforward approach was used inthe earliest distributed NFS factoring efforts. A more effi-cient and still embarrassingly parallel strategy is to assigndisjoint ranges of primes q to different contributors, and tolimit the search, given such a special q , to all relevant a, b
pairs for which f (a, b) is divisible by q . Each special q re-sults in a number of different sieving tasks that varies fromzero to degree(f ) (i.e., six, for RSA-768). This approach,which is a bit harder to program, has gained popularity sincethe mid 1990s. We used it for RSA-768 and we could fullyinspect 15 to 20 million a, b pairs per second on a 2.2 GHzcore with 2 GB RAM. Sieving task were distributed amongthe contributors depending on their available computer re-sources.
Overall, about 465 million sieving tasks were processed,for special q values between 108 and 1.11 ·1010. An averagesieving task took about 100 seconds on a core as above, andresulted in about 134 relations at about 150 bytes per rela-tion. Sieving started in the summer of 2007 and lasted foralmost two years. With about 1500 core years, we achieveda sustained performance equivalent to more than 700 cores,full time for two years. Table 1 gives a breakdown of theranges of special q values processed by the different contrib-utors. Section 4 presents more details of the infrastructuresused. Including duplicates, in total more than 64 billion re-lations were generated. They were collected at EPFL, withseveral backups, also off-campus.
The main input for a processor contributing to the sievingis the range of special q values to be processed. The numberof sieving tasks per range behaves roughly as the numberof primes in it. Thus, it slowly drops off with increasing q
values, and for a range [L,U ] can be estimated as Uln(U)−1 −
Lln(L)−1 . For a range of length 1000 with L ≈ 109 this resultsin about 48 sieving tasks, which is reduced to about 43 forL ≈ 1010. Such ranges can typically be processed in lessthan two hours.
Compressed storage of the relations, along with the fac-torizations of the f (a, b) · g(a, b)-values, took 5 TB. This
amount of storage is by no means exceptional, and shouldnot be hard to deal with. Nevertheless, storage problemscaused most stress while sieving for RSA-768, mostly dueto the lack of reliability of the storage devices. Not just disksfailed (with RAID servers as a first line of defense), disk cas-ings failed as well, with unforeseeable consequences for thedisks and RAID servers. In the course of the sieving we de-cided to hedge our bets by spreading the risk over a varietyof manufacturers and vendors. Additionally, human errorsare unavoidable and rigid rules had to be enforced to mini-mize the consequences.
Otherwise, sieving is the least stressful step of NFS, as itis not just embarrassingly parallel but also tolerant to slop-piness and errors. All that counts is if ultimately enoughrelations will be found, for the rest one mostly needs pa-tience. It does not matter what special q was used to finda relation, and it does not matter much—except for a mi-nor loss of efficiency, and unless it occurs systematically—if not all special q values in a range are properly processedor if occasionally some of the data generated gets lost orcorrupted (as the correctness of a relation can easily be ver-ified at the central repository: anything that is not correct issimply discarded). A crash of one or more processors doesnot affect the results of any of the other processors, and atask that may be left unfinished due to some mishap can bereassigned to another processor or it can be dropped alto-gether.
Intermediate step: filtering After duplicate removal, use-less relations are removed. These include, for instance, re-lations for which f (a, b) contains a prime factor that doesnot occur in any other relation. This can only convenientlybe done if all data reside at a single location. The survivingrelations are used to build an over-square bit-matrix withrows determined by the exponent-vectors of the primes inthe remaining f · g-values, or combinations thereof. Whilebuilding this bit-matrix, many choices can be made. Be-cause dependencies among the rows will be determinedin the next step, it pays off to aim for a low dimensionand overall weight (i.e., number of non-zero matrix en-tries).
For RSA-768, the 64 billion relations resulted in 48 bil-lion non-duplicates. Several matrices were built, the best ofwhich had 193 million rows and 28 billion non-zero entries.It required about 105 GB of disk space. The entire processto convert the raw relations into a matrix took about twoweeks of computing on a 304-core cluster at EPFL: rela-tively speaking quite modest but rather cumbersome as largeamounts of data had to be moved around.
Second main step: matrix Although the sieving step re-quires more CPU time, the matrix step is considered to bethe most challenging step of current large scale factoring ef-forts. Gaussian elimination was used for factoring related
Cluster Comput
Table 1 For each range of special q values, the contributor that sievedthat range is listed, along with the amount of RAM used for the sievingprogram, the number of relations found, and the approximate numberof relations that was found per task. The number of relations per taskdecreases with increasing special q values. Independently, the numberof tasks per fixed length range decreases because the number of primesper range decreases. Together these effects contribute to the overall
drop in the number of relations found per fixed length range (as exem-plified by the two arrows), though for some machines it turned out tobe faster to produce fewer relations per range. Note also that, on aver-age, fewer relations are found per task if less RAM is available: exceptfor the low range of special q values, we therefore preferred to use ma-chines with at least 2 GB RAM per core to sieve the more productiveranges
q-range Contributor RAM Number of Percentage Relations Percentage
(millions) relations found of relations per task of tasks
matrices until the early 1990s. It was abandoned in favorof the block Lanczos algorithm [6] which requires muchless time and memory due to the sparseness of the inputmatrix. A disadvantage of block Lanczos (which it shareswith Gaussian elimination) is that it does not allow indepen-dent parallelization: it must be run on a single tightly cou-pled massively parallel machine. We are still in the processof evaluating the feasibility of doing this step using blockLanczos at a single location.
Because of this disadvantage, we preferred block Wiede-mann [7]. Though not embarrassingly parallel, the compu-tation can be split up into a limited number of chunks. Eachchunk can be processed on a tightly coupled cluster, inde-pendently of the other chunks each of which may simulta-neously be processed at some other location.
More precisely, block Wiedemann works in three stages:a first stage that can be split up as above, a brief centralstage that needs to be done at one location, and a final stagethat is less work than the first stage and that can be split upinto any number of chunks if enough checkpoints are keptfrom the first stage. The first and final stages both consistof iterations of matrix × vector multiplications, where thematrix is the fixed, sparse bit-matrix resulting from the fil-tering step, and where the (bit-)vector is constantly updated(as the result of the previous multiplication). It is possible touse k different initial bit-vectors and to reduce the numberof multiplications per bit-vector by a factor of k, as long asthe total number of multiplications by bit-vectors remainsconstant. This comes at various penalties, though, and com-plicates the brief central stage if k gets large.
For RSA-768 we used k = 8 · 64 = 512 and 8 chunkseach of which processed 64 bit-vectors at a time. Per chunk,565 000 matrix × vector multiplications had to be done inthe first stage, 380 000 in the third stage, for our matrixof dimension 193 million with 28 billion non-zero entries.Running a first or third stage chunk required 180 GB RAM.Table 3 in Appendix lists the various clusters used for thefirst and third stage, along with the time required per mul-tiplication per chunk. Obviously, the timings vary consider-ably depending on the type of processor, number of cores,and type of interconnect. For instance, on 12 dual AMD2427 nodes (hex-core, thus 144 cores, with 16 GB RAMper node) with InfiniBand, a multiplication takes about 4.5seconds. This implies that on 48 such nodes (576 cores—56 such nodes with 672 cores were installed at EPFL whilethe first stage was underway) all eight chunks for stages oneand three could have been completed in about 100 days, forabout 160 core years of computing. The central stage tooka bit more than 17 hours on the 56 freshly installed nodes,using all available 896 GB RAM (except for a short periodwhen a terabyte was needed and swapping occurred), butjust 224 of the 672 available cores. On the variety of clus-ters that was actually used the entire block Wiedemann steptook 119 days.
Unlike sieving, no errors can be tolerated during the ma-trix step. The iterations thus included frequent checkpointsto ensure that the computation was still on-track. We expe-rienced no glitches. In the original distributed block Wiede-mann all chunks consisted of an equal amount of work, i.e.,the same number of multiplications on matrices and vectorsof identical sizes. We used a more flexible version of the al-gorithm, so that faster jobs can do more multiplications andslower ones fewer, as long as the same overall number ofmultiplications as before is reached.
Finishing up: square root Finding out if the dependenciesas produced by the matrix step are correct is probably themost nerve-racking part of any large scale factoring project.Also from a mathematical point of view, deriving a factor-ization from a dependency is one of the more exciting steps.Computationally speaking, however, it is usually the leastchallenging step. For RSA-768 it took about one core dayand resulted in the following factorization.
RSA-768 = 12301866845301177551304949583849627207
72853569595334792197322452151726400507
26365751874520219978646938995647494277
40638459251925573263034537315482685079
17026122142913461670429214311602221240
47927473779408066535141959745985690214
3413
= 33478071698956898786044169848212690817
70479498371376856891243138898288379387
80022876147116525317430877378144679994
89 · 3674604366679959042824463379962795
26322791581643430876426760322838157396
66511279233373417143396810270092798736
308917.
The correctness of the result, once obtained after 1700 coreyears of computing, can be verified in a fraction of a second.
Table 2 gives the overall workflow, along with the per-centages contributed (for the sieving measured in differentways).
4 Heterogeneous compute infrastructure
In this section we describe the heterogeneous environmentused for the factorization of RSA-768, with a focus on themanagement of the sieving step.
Cluster Comput
Table 2 Workflow of the project. The sieving percentages are onlyrough indications for the overall sieving contributions because tasksfor larger special q values are less productive, and as a consequence
also faster to process: the truth is biased toward the relation contri-bution percentage. The last two rows contain approximations for coreyears (“cy”) spent and dates (yyyy:mm:dd) of the calculation
As set forth in Sect. 3, sieving consists of processing arange of special q values, where for each special q value atmost six sieving tasks have to be performed. Given a range,this is carried out by a C program. This program, calledlasieve, resulted from many years of research, develop-ment and refinements at the university of Bonn. All collab-orators got statically linked versions of lasieve, gearedtoward their hardware (processor type and cache size) andoperating systems. The number of relations found per taskdrops off with increasing special q values. One thereforetries to completely process all smaller ranges before mov-ing to larger ones, leaving as few unprocessed gaps as pos-sible.
At the highest level, EPFL distributed relatively large,disjoint ranges of special q values among the collabora-tors, depending on the specifics of the cluster(s) or machinesto be used. The way a range is processed depends on howlasieve is run, the cluster usage agreements, and the jobscheduler. In any case, a large range assigned to a site mustbe partitioned into smaller subranges, each of which can beprocessed in a reasonable amount of time by a CPU corerunning lasieve: as mentioned, a range of length 1000takes about two hours to process. The naive approach toassign subranges is to do so upfront at the job scheduler’slevel before any particular CPU core has been allocated toprocess that subrange using lasieve. It allows for man-ual range partitioning and assignment. This works if, bar-ring exceptional irregularities, one may assume that all jobs,once put in the queue to be executed, will eventually betaken into execution and that, when taken into execution,they will finish their allotted range. This situation may ap-ply if one is the sole user or owner of a desktop machine
or cluster, or if otherwise favorable access conditions havebeen granted to the compute resources. It applied to somecontributors.
Even so, several set-ups used for the sieving used an au-tomated approach where range assignment is postponed tothe moment that a CPU core is ready to start sieving. Itavoids range fragmentation caused by the apparently un-avoidable fact of life that in some environments there are al-ways jobs that disappear from the queue without ever havingbeen taken into execution. Nevertheless, and in either case,it may be desirable to conduct post-mortems of occasionalcrashes. This would involve cumbersome analysis of partialoutput files to extract (and reassign) previously assigned butunfinished ranges. Several such systems were used (and aredescribed below) that are semi-automated in the sense thatranges were assigned automatically, but that make the im-plicit assumption that range fragmentation will be kept to aminimum, i.e., that normally speaking assigned ranges willbe fully processed and will not be left unfinished. We stressagain that the existing systems that we used are heteroge-neous and do not share a common software layer. Addition-ally, the systems were not always available at or for the sametime, and resource allocation and availability was not guar-anteed neither at the beginning nor during the computationalruns.
This assumption, which is based on a 100% completionmodel of assigned ranges, can certainly not always be made.Traditionally, sieving jobs are only run on processors thatwould otherwise be idle. For example, in [13], the morethan 20 year old, first collaborative sieving effort that weare aware of, usage is cited of a “machine idle” tool to iden-tify machines that have not recently been used and that thus
Cluster Comput
may be added to the pool of sievers. However, sieving jobswere terminated instantaneously as soon as a machine wasreclaimed, for instance by hitting a key. See also [10]. In thecluster job scheduling system OAR [4] the possibility to ex-ploit otherwise idle resources in a similarly volatile way iscreated by best-effort jobs. With such jobs, partial process-ing of assigned ranges is systematic, making extraction ofparts of ranges that are left unfinished mandatory, and re-sulting in range fragmentation that quickly becomes unman-ageable for humans. A convenient way to fully automaterange management—including reclaiming ranges from in-terrupted jobs—was implemented at INRIA and is describedbelow.
The matrix jobs require much closer supervision thansieving jobs. Gaps cannot be tolerated, and work left un-finished by crashed jobs has to be completed starting fromthe most recent checkpoint. Thus, preferably the matrix isnot done using best-effort types of jobs but using super-computers or relatively large dedicated (sub)clusters for ex-tended periods of time. This is what we did for the RSA-768 project, simultaneously using various clusters, all withmanually managed jobs, only a small percentage of whichwere best-effort jobs. In particular during stage 3 of the ma-trix step the three participants contributing to that part ofthe calculation frequently discussed task assignments, withsome clusters taking over jobs previously assigned to others.The matrix step is not further discussed below. Some detailson how and where it was run are listed in Tables 2 and 3.
4.1 AC3
The Australian Centre for Advanced Computing and Com-munications provides high performance computing plat-forms for academic and research staff at eight Australianuniversities. A selection of machines is available for staffto apply for system units of computing time. We were allo-cated computing resources equivalent to full-time use of 16nodes of the machine Barossa, a Dell Beowulf cluster hav-ing 155 3 GHz dual processor Pentium 4 nodes with 2 GBRAM per core. Of this memory, each node reserves somefor the operating system, the batch queue system, and videosharing. As a result, only about 1.7 GB are available forsubmitted jobs. Although less than the ideal 2 GB, we sub-mitted jobs restricted to 1.7 GB RAM and to 2 GB of vir-tual memory, and these jobs ran nearly as efficiently as ma-chines that had the full 2 GB RAM. So, swapping was min-imal.
Submission of jobs to Barossa is via the PBS batch queuesystem. In addition to the memory restrictions, a job is onlyallowed to run for two days or else an automated programkills the violating submissions. From a few experiments,we determined how many special q values could be han-dled safely under the two day limit. A simple shell script
was used to submit new ranges while keeping track of whatranges had already been done. Running of this shell scriptwas done manually on a daily basis. Moreover, uploadingof the data to EPFL and dealing with the rare occurrencesof failed jobs was also done manually. Participation wascut short when the Australian participant (Scott Contini)left Macquarie University for a full-time position in indus-try.
4.2 CWI
At the Centrum Wiskunde & Informatica we utilized work-stations outside the usual office hours. All workstations runa recent version of Fedora Linux. The home and project di-rectories are hosted on the NFS2 file server located at SARA(the Academic Computing Centre Amsterdam), over a net-work based on UTP switched gigabit Ethernet.
All workstations that participated are x86-64 machineswith varying numbers of cores, clock rates, cache sizes andamounts of memory per core. At the outset, most weresingle-core machines with 1 GB RAM, with a small num-ber of Intel dual-core machines. During the sieving, almostall single-core machines were replaced by dual-core ones,whereas some dual-cores were replaced by quad-cores with2 GB RAM per core. We had 120 to 180 cores at our dis-posal, depending on hardware upgrades and on users willingto share their workstations.
After getting a large special q range from EPFL, we useda script to generate jobs that invoke lasieve on subrangesthat a single core can complete within three to five nights ora weekend. All jobs were placed in the input queue whichis located at the NFS file server accessible from every work-station. The jobs were managed and run as follows:
factord. On each participating workstation, the shellscript factord was invoked by crontab every evening.It manages the supply of jobs in the following manner:
1. checking existence in the machine’s working directoryof a checkpoint file of a previous job, and if so re-invoking that job (using sieving task as describedbelow);
2. fetching a new job from the input queue if a checkpointfile is not present;
3. moving output produced by jobs that completed theirrange to the output queue;
4. terminating if the input queue is empty;5. sending a termination signal to lasieve early in the
morning on working days.
Early on in the project, the simultaneous start of manysieving jobs crashed the automounter daemon on some
2In this section NFS stands for Network File System instead of NumberField Sieve.
Cluster Comput
single-core machines, thereby preventing lasieve tostart. This was solved by randomly spreading the startingtimes over a period of half an hour. When invoked, fac-tord reschedules its next start. The script is used to man-age other factorization projects as well.sieving task. This is a shell script that ensures theproper start of lasieve. If a checkpoint file exists inthe machine’s sieving working directory, the script resumeslasieve with the old configuration, starting from the lastused special q . Otherwise, sieving task determinesthe number of available cores and RAM per core, in or-der to set proper input parameters for lasieve. On manymulti-core machines we utilized all but one core, keep-ing one core available for applications by the workstation’sowner.
Monitoring. A monitoring tool checked regular progress ofall jobs. If a job is found to be stalled, for instance dueto a user program or hardware failure, the tool moves allrelations from the machine’s working directory to the out-put queue and reassigns the remaining special q values toa new job which is put in the input queue. Once the hostis available for sieving again, it fetches a new job from theinput queue as described above.
4.3 EGEE
The infrastructure provided by the Enabling Grids for E-SciencE (EGEE, cf. [8]) is the biggest production Grid in-frastructure in the world. It is open to various types ofscientific domains, applications and users. Typically, sci-entific applications are organized in Virtual Organizations(VOs) and are shared among several users. Since there wasno VO available that would suit our factoring attempt, wefirst used an existing VO and later created our own cryptoVO.
As we demonstrated in [17] the gLite [9] Grid mid-dleware that underlies EGEE’s job submission and execu-tion, though suitable for embarrassingly parallel jobs, fo-cuses on optimizing throughput for many users and applica-tions rather than for a single application or user. This is dueto a complex interaction of the Grid’s meta-scheduler andeach site’s local resource management system, and is com-pounded by job queue latencies affecting perceived perfor-mance, scheduler failures that cause jobs to vanish (as notedabove), and heterogeneous hardware causing diverse run-times. We therefore adopted the approach proposed in [17]which integrates in the gLite Grid middleware a runtime-sensitive BOINC-like system with a task server, as illus-trated in Fig. 1. Compared to the traditional way EGEE jobsare handled, the main advantage of our approach is that itadapts automatically to the different runtimes required onthe heterogeneous EGEE worker nodes, thereby maximiz-ing throughput. The EGEE infrastructure, thus adapted, was
successfully used for several months, processing up to athousand ranges in parallel at more than 20 sites across Eu-rope.
The overall workflow included the following softwarecomponents.
siever-submit.pl. This perl script uses the gLite jobsubmission command line interface to submit siever-worker.pl jobs to the gLite resource broker (Step 1ain Fig. 1), which for each job selects a suitable workernode (depending on required RAM) and submits it therefor execution (Step 1b in Fig. 1). As long as a certain con-figurable minimum number of running jobs is not reached(we used from 100 to 1000 parallel jobs; once running, jobsmay abort due to failure or because they exceed their max-imum runtime), the script keeps submitting new jobs. Ad-ditionally, the script monitors how many jobs have finished(Step 6 in Fig. 1) and displays the status on a webpage(Step 7 in Fig. 1).siever-worker.pl. This is an (in principle) everlast-
ing perl script that runs on a worker node and that
1. attempts to obtain a range of special q values by sub-mitting an HTTP request to the task server (steps 2 and3 in Fig. 1);
2. terminates if no range was received;3. runs lasieve on the range obtained (Step 4 in Fig. 1);4. upon completion of the range, notifies the task server
and transfers the results of the calculation to a Grid stor-age element (step 5a and 5b in Fig. 1);
5. returns to Step 1.
Because several EGEE job submission systems limit jobruntimes to 24 hours, the script (and thus its most recentlyspawned lasieve job) may be terminated, giving rise toassigned but unfinished ranges.Taskserver.pl. The task server is initially loaded
(Step 0 in Fig. 1) with a special q range, partitioned insubranges of length 1000. If notification of completion of arange is not received within some fixed period of time (say,one day) after it was assigned, the task server returns theentire range to the pool. No attempts were made to avoidrecomputation of data by analyzing partial output files toextract unprocessed subranges.The task server has an HTTP interface for task assignmentand management and, since the task server keeps track ofcompleted ranges, to determine which output files havebeen uploaded to the Grid storage element. Internally, thetask server uses a relational database management systemback-end to manage tasks (start date, maximum allowedruntime to avoid zombie tasks, end date, etc.).gLite StorageElement. This is a conventional, Stor-
age Resource Manager (SRM) based Grid storage systemthat securely and reliably stores result files.
Cluster Comput
Fig. 1 Job submission andexecution infrastructure arebased on gLite (componentsindicated by italic font). Thecontrolling scriptsiever-submit.pl generatessieving jobs which are executedon nodes of the various EGEEclusters
4.4 EPFL
EPFL DIT Of the various computing resources providedby EPFL’s Domaine IT (Central IT services), the Callistocluster and the campus grid “Greedy” contributed to thesieving. Callisto is a general-purpose cluster, usable byEPFL researchers; Greedy is EPFL’s desktop grid (alsoknown as a campus grid), meant to enable recovering un-used CPU power. Table 3 lists some of the hardware speci-fications for both systems.
Callisto is served by two front-end nodes, also actingas file servers for an 18 TB General Parallel File System(GPFS). Behind the two front-ends sit 128 dual quad-corecompute nodes, interconnected via a fast Dual Data Rate(DDR) InfiniBand fabric. All the compute nodes are run-ning SuSE Linux 10.2, and job scheduling is done with PBS(“Portable Batch System”) Pro 10. For the sake of energyconsumption and ease of administration, the compute nodesare in “blade” format: a single chassis can host up to 14blades, leading to higher electrical efficiency and less ca-ble clutter behind the racks. Furthermore, in an effort to re-lieve the air-cooling infrastructure, the cluster is installed inwater-cooled racks, with almost no heat dissipation into themachine room air. At the start of the sieving, access to Cal-listo was free, though closely regulated with a focus on par-allel jobs that can profit from the fast interconnect. In 2009Callisto’s access policies were changed, by requiring labo-ratories wishing to use the cluster to give a partial financialcontribution toward its purchase.
Greedy is the second axis of the computing resourceslandscape of DIT (the third being a massively parallel super-computer), with an emphasis on grid computing and single-core jobs. Access to Greedy is free to all members of EPFL.The goal of Greedy is to federate unused CPU power acrosscampus, by harvesting otherwise unused cycles of classroomand office PCs at EPFL. In order to not bother the user of
the machine with continuous computations (fan noise canbe bothersome in an office environment), grid jobs can runonly during nights and weekends, when the probability ofhaving someone working on the machine is low. Addition-ally, if user activity is detected while a grid job is running,the grid job will be suspended.
The software stack used is Condor [5], a “High Through-put Computing” middleware. All policies governing jobstartups and suspensions are made with Condor mech-anisms, without relying on external tools. Besides twocentrally-managed servers controlling the grid infrastruc-ture, more than 1000 compute cores are available on thegrid, mainly from classrooms. Greedy is a highly hetero-geneous environment, with operating systems ranging fromLinux on 32-bit machines to Windows 7 on 64-bit ones,and a variety of combinations of CPU types and memoryamounts. Due to this heterogeneity, mechanisms are put inplace that select the proper platform for a given set of jobs,so that they run on the operating system/CPU combinationthey were compiled for.
EPFL LACAL EPFL’s Laboratory for Cryptologic Algo-rithms has a variety of clusters at its disposal. Two of theseclusters (Lacal140 and Lacal304) were used for the siev-ing. A third (Lacal672) was used only for the matrix, as itwas purchased when the sieving was already complete. Ta-ble 3 lists some hardware specifications for these three clus-ters.
Lacal140 is hooked up to a front-end with a 4.2 TB NFSfile system and a 6.1 TB Parallel Virtual File System (ver-sion 2) provided by 8 IO servers. It is integrated in EPFL’sPleiades2 HPC Linux cluster, running SuSE Linux 10.2 andusing Torque with Maui for job scheduling. Lacal304 ranGentoo Linux, and used a single front-end server that actedas a file server for a 1.5 TB NFS file system. Formerly lo-cated at EPFL’s Sciences de Base clusterroom, Lacal304
Cluster Comput
was partially dismantled with the arrival of the new clusterLacal672 (cf. below) and 24 of its nodes now serve as thefreely accessible general purpose 192-core cluster VEGA atEPFL DIT.
The 56 Lacal672 nodes are connected to a single front-end server that acted before as Lacal304’s file server (forthe 1.5 TB NFS file system that was used by Lacal304).It was installed, at EPFL’s Sciences de Base clusterroom,while stage 1 of block Wiedemann was already in progress.
Sieving at EPFL With the exception of the Greedy pool,Paul Leyland’s cabalc and cabaldwere used to run siev-ing jobs on EPFL’s DIT and LACAL clusters. This is fullydescribed in Sect. 4.6.
On the Greedy pool Condor jobs are submitted with para-meters for machines with 1 GB RAM. Because most of theworkstations on the grid run a Windows operating system,special binaries were created that do not use our fast assem-bly routines. As a consequence, processing a range of length2000 would take five to six hours, i.e., a bit slower than usualbut typically less than a night. Therefore, jobs with ranges oflength 2000 were submitted to the grid, in batches of 5000 asthat would produce at most 10 gigabytes of data. Transfer-ring the output to the storage facilities at LACAL was donemanually.
4.5 INRIA
The Aladdin-Grid’5000 (“g5k”) is an HPC grid funded byseveral French research institutions, including INRIA, andintended for experimental research. Started in 2004 it con-sists of about 5000 CPU cores (taking the latest hardwareupgrades into account there are currently more than 6400cores), spread over nine sites across France. Each site hostsup to five clusters of identical nodes. The g5k clusters usedfor this project are listed in Table 3. Nodes at the same sitehave access to a shared NFS volume, but no global NFSfilesystem is shared across the sites.
Access to g5k is not exclusively limited to experimentalresearch projects. Long running, resource-hungry applica-tions such as sieving are allowed as well, as long as theycomply with the platform’s policies for this type of jobs.Thus we agreed to limit our jobs to at most 25% of a site’sresources at any given time, running at the lowest priority.In the job submission system OAR [4] used by g5k theseare best-effort jobs, as mentioned above.
Premature termination of best-effort jobs is a normalevent. If it occurs it affects all nodes allocated to the job.Notification of termination may never be received. Further-more, as alluded to above, due to scheduler errors it cannotbe guaranteed that all queued jobs are eventually taken intoexecution: sometimes jobs vanish. To deal with the rangefragmentation that would result from premature termination,
and to avoid fragmentation due to jobs disappearing fromthe queue, a framework consisting of simple shell and perlscripts was designed that resulted in a very effective and userfriendly range management system. It was successfully usedfor about a year, contributing substantially to the sieving ef-fort.
The workflow described below, as it applied to each ofthe g5k clusters that we used, is superficially similar to theEGEE workflow. The details are quite different.
OAR jobs. The scheduler allocates best-effort OAR jobs tothe nodes. These jobs may differ in the number of nodestargeted, but are otherwise identical and carry no infor-mation about the calculation to be performed. The num-ber of OAR jobs that can be submitted at the same timeis limited because, due to our best-effort constraint, in to-tal never more than 25% of the nodes may be used, andbecause the job scheduler performs suboptimally if thereare many jobs in the queue (recent software upgrades haveimproved it). More down-to-earth, the web-based grid oc-cupancy visualization tool uses one line per job irrespectiveof the job’s size: if many sieving jobs are displayed nega-tive feedback can be expected. To deal with these issues,and to make sure that we always had jobs small enoughto “fit in the holes”, we submitted OAR jobs that allocaten/2, n/4, n/8, n/16, and n/16 nodes, where n is a quar-ter of the number of nodes at the site, targeting a total of n
nodes.An OAR job starts one core job on each CPU core ofthe nodes it had been given access to. Upon (expected)interruption of OAR jobs, a new one needs to be sub-mitted to sustain the throughput. The required function-ality of resubmitting interrupted jobs is provided by theOAR scheduler: so-called “idempotent” jobs, if left un-finished, may be restarted with the same command lines.Given our generic OAR jobs it thus sufficed to set theirtime limit to infinity (actually, one week), with the resultthat the scheduler made them persistent. As a result wedid not have to rely on scripts that automatically submitjobs and that, in our experience, often lack the robustnessthey should have (due to communication glitches and timedrifts).
Core jobs. All core jobs, over all nodes and all OAR jobson the same cluster, are identical. A core job is an (in prin-ciple) everlasting shell script that
1. attempts to obtain a range of special q values that doesnot intersect with any other range that has been com-pleted or that is currently under execution:
As different core jobs may make concurrent re-quests for ranges of special q values, range al-location must be atomic. Because file renaming(moving) on an NFS partition is atomic, rangesmay be claimed by a core job by trying to move a
Cluster Comput
file containing a range from the queue/directoryto the inprogress/directory. Per file the moveis guaranteed to succeed for exactly one core job,which gets the claimed range. Core jobs that failto move a file sleep for a couple of seconds beforetrying again.The queue/directory contains ranges to beprocessed encoded in names of otherwise emptyfiles, allowing for convenient sequential process-ing of the available ranges assuming files areclaimed in lexicographic order. Obviously, differ-ent clusters receive non-intersecting ranges.
2. terminates if no range could be obtained;3. runs lasieve on the range obtained;4. upon completion of the range, marks the output as
clean:
If a call to lasieve terminates because it fin-ished the assigned range, the core job com-presses the output created in the working/directory, moves it to the results/directory,and removes the corresponding file from the in-progress/directory, thereby marking that out-put as clean.
5. returns to Step 1.
Although a core job is not meant to terminate (except onrange starvation), it dies as soon as the scheduler decidesto abort the best-effort OAR job that spawned it. Abrupttermination during execution of lasieve was handled asdescribed below.
Watchdog job. The working/directory will contain par-tial output files of interrupted lasieve jobs, along withstill active output files. The watchdog job identifies out-put files that have not been touched for longer than rea-sonable if its lasieve job were still alive (say, for 15minutes, which is 5 to 10 times more than the expected de-lay between subsequent writes). It analyses each of thesepartial output files, returns the unprocessed part of therange to the queue/directory, renames and compressesthe output file (possibly after truncation) so its name re-flects the processed part of the range, moves it to theresults/directory, and removes the corresponding filefrom the inprogress/directory. Due to the lightweightapproach of encoding ranges in names of empty files thefile system could easily cope with the range fragmenta-tion.This approach makes sure that, eventually, all specialq values assigned to the cluster are processed, with-out human supervision. All that needed to be done waskeeping an eye on the queue/directory to make surethat there was an adequate supply of ranges. The I/Oand CPU footprint of the watchdog job are not sig-nificant, so it could be run on the submission front-end.
Data movement. Storage nodes on g5k are not meant tohost large amounts of data. Results were therefore reg-ularly copied from g5k to INRIA Nancy, where sev-eral partitions totaling 13 TB were used for storage andbackups. Relations were assembled to larger files cor-responding to ranges of length e.g. 106 (about half aGB compressed), checked for correctness, and copied toEPFL.
4.6 Leyland
In comparison with the computation as a whole, Leyland’ssieving contribution was relatively minor. At most 25 ma-chines, most of them dual-core systems, were in use at anyone time. Accordingly a relatively simple client/server har-ness was used to allocate special q ranges and a simple scriptused to automate uploading the results to an sftp site locatedat EPFL. Monitoring of progress, detection of error condi-tions and recovery from them was performed manually. Thescripts running on the client and server side are named ca-balc and cabald, respectively, because they were devel-oped for the factorization of 2773 + 1 in 2000 by a teamusing the nom-de-plume ‘The Cabal’ [3], some of whomcontributed to the factorization of RSA-768. The scripts, de-scribed in more detail below, were also used for the sievingon the clusters at EPFL DIT and LACAL.
cabalc. cabalc uses a configuration file to specify theIP address of the machine running its cabald; the porton which to communicate; and a prototypical command tobe run. It can execute an arbitrary command with parame-ters derived from a pair of numbers provided by cabald,allowing it to correctly run lasieve for any range of spe-cial q values.Upon start, cabalc clears a Boolean ‘work-to-do’ vari-able, reads the server’s address and port from the con-figuration file and then enters an endless loop. There, itfirst attempts to open communications to the server. Ifnothing is forthcoming, cabalc waits for a few sec-onds and tries again. If ‘work-to-do’ is clear a request ismade of the server for a special q range. The range isstored and ‘work-to-do’ set. The command given in ca-balc’s configuration file is then run with proper com-mand line arguments to process the newly received range.When that sub-process completes, cabalc returns to thestart of its endless loop. This time around, ‘work-to-do’ isset so the saved initial and final values of the completedrange are returned to cabald before a new task is re-quested.cabald. cabald maintains a configuration file which
contains a list of special q values which have been allo-cated to clients; one or more pool lines to specify un-allocated special q values; a list of zero or more frag-ments; a single value, blocksize, which specifies the
Cluster Comput
maximum special q range to be allocated to each client;and the network port on which it communicates to itsclients. Initially, there are no fragments and a singlepool containing a large special q range. A final set oflines contains information about which special q rangeshave been allocated to clients by earlier invocations of ca-bald.When cabald starts it reads its configuration file and cre-ates a data structure which contains one or more ranges(lower and upper limits) of special q values which havenot yet been allocated. Under normal circumstances, thiswould be a single range given by a pool line. Very occa-sionally, a second such line would be added to the con-figuration file when the existing pool was close to ex-haustion. A more frequent occurrence would be after oneor more clients had crashed. In this situation, the ca-bald process would be stopped, the unsieved special q
ranges extracted from partial output files and added to theserver’s configuration file as fragment lines. Any corre-sponding allocation lines in the configuration file wouldbe deleted. Upon restart, cabald also places the frag-ment data into the unallocated tasks data structure. ca-bald then opens a log file for appending status messagesand enters an endless loop waiting for cabalc client re-quests.On receipt of a client communication, the returned specialq values are used to update the unallocated tasks struc-ture. A new range of special q values, of size at mostblocksize, is then sent to the client. Allocation is madefrom the pool(s) only when all the fragments havebeen exhausted. The log file is then updated with an en-try which records the IP address of the client, the spe-cial q values, if any, returned by the client and the spe-cial q range just allocated. Finally, the cabald config-uration file is re-written so that the current state of thepool and/or fragments is available for subsequent runsof cabald.In practice, cabald was very stable. It never crashed un-expectedly and was stopped only for scheduled systemshutdowns or for maintenance of its configuration file whenfragments or a new pool were added.
Monitoring. As noted above, cabald and cabalc pro-vide neither detection of errors nor uploading of outputdata. The latter was performed by an uploader script whichcompressed all but the most recently modified lasieveoutput file (on the assumption that the latter was still beingwritten by an active lasieve); uploaded the result to afixed directory of a sftp server at EPFL; and then movedthe compressed files to another directory where they couldbe recovered if necessary and yet not interfere with subse-quent activity.At sporadic intervals, usually once a day or so, the ma-chines supposed to be sieving would be examined to see
whether they were in fact doing so. A trivial script waswritten to contact all machines in the set of clients andto determine whether lasieve was running the cor-rect number of times (a multi-cored system usually ranseveral copies). If a client failed to respond or if theywere not sieving the situation would be investigated byhand. First, the uploader script would be run. Any re-maining output files were examined to determine the spe-cial q at which the siever failed. Finally, cabalc wasrestarted.
Despite not being fully automated, cabalc and cabaldbetween them allowed one person (Leyland) to manage sev-eral dozen siever instances with little effort. That the samescripts also worked satisfactorily to manage hundreds ofsieving jobs at EPFL is probably due to the fact that therewe restricted ourselves to stable resources fully dedicated tothe sieving.
4.7 NTT
Nippon Telegraph and Telephone Corporation provided thefollowing computational resources that were fully dedicatedto the sieving:
The nodes are connected with gigabit Ethernet and eachnode is equipped with a local disk. To manage sieving as-signments we used two perl scripts that were also used dur-ing the sieving for M1039 [1]: ds2c on the client side andds2 on the server side.
ds2c. For each client on which it is running, the scriptds2c requests a special q range from the server, and runslasieve while recording its standard input, error and re-turn values. After lasieve finishes its assigned range,ds2c sends all resulting data to the server and requestsa new range. When ds2c cannot connect to the server,lasieve is invoked with a range of special q values thatis randomly chosen from a range previously communicatedby the server.
3These nodes got more RAM for the matrix step (cf. Table 3), resultingin 5 GB RAM for most nodes (13 nodes got 8 GB RAM). Furtherdetails can be found in [2].4One of these nodes broke down during the sieving. It was not repaired.
Cluster Comput
ds2. The server script ds2 has an interface that allows ahuman operator to provide a new range of special q val-ues, typically of length 1 000 000. Upon request from aclient, ds2 assigns to the client a subrange, typically oflength 1000 as a range of that size can be processed ina few hours. If a client does not report back within, say,8 hours, its range is reassigned. At any time the opera-tor may change priority of range assignments. Data cor-responding to a range that is reported back are stored.The server may also receive data for ranges other thanthose it assigned; occasionally, correctness of such spuri-ous data is verified manually. Logging mechanisms are inplace to allow recovery from mishaps (or scheduled main-tenance).
Although these two scripts can deal with many exceptions,they are unable to detect a full disk. Every working daymanually invoked scripts and commands are therefore runto confirm client node status, to merge any duplicate assign-ments, to roughly confirm the consistency of all data storedby ds2, to compress the data, and to send them to EPFL.
4.8 University of Bonn
At the University of Bonn sieving took place at only onelocation, the Himalaya cluster at the Institute for Numeri-cal Simulation. On this cluster jobs have to be submittedvia a queueing system. This was done using a simple C-program. It checked periodically how many sieving jobs arein the queue and, if this number is below a certain thresh-old, it submitted new jobs. All problems and inconsisten-cies, caused by jobs that were never taken into execution,jobs that crashed, etc., were resolved manually.
5 Conclusion
We described the heterogeneous hardware resources and di-verse management tools used during a period of about twoyears at many different locations to solve a cryptanalyticchallenge. The computational effort required, though largegiven the resources available, was considerable though notexceptionally large: it would require a couple of weeks usingthe full “Ranger” supercomputer at the University of Texasat Austin.
Our result is a good indication for the size cryptana-lytic effort that can successfully be undertaken in a moreor less acceptable amount of time by a rather loosely cou-pled, widely scattered and mostly academic team of vol-unteers. Pulling off a substantially larger effort in com-parable or less time would require tighter managementor more funding than customary in academic cryptana-lytic circles. A greater appreciation of the HPC commu-nity for cryptanalytic activities could change this pictureovernight.
Acknowledgements This work was supported by the Swiss NationalScience Foundation under grant numbers 200021-119776 and 206021-128727 and by the Netherlands Organization for Scientific Research(NWO) as project 617.023.613. Experiments presented in this paperwere carried out using the Grid’5000 experimental testbed, being de-veloped under the INRIA ALADDIN development action with sup-port from CNRS, RENATER and several universities as well as otherfunding bodies (see https://www.grid5000.fr). Condor middleware wasused on EPFL’s Greedy pool. We acknowledge the help of Cyril Bou-vier during filtering and merging experiments.
Appendix: Clusters used and block Wiedemann timings
1. Aoki, K., Franke, J., Kleinjung, T., Lenstra, A.K., Osvik, D.A.:A kilobit special number field sieve factorization. In: Asiarcypt.LNCS, vol. 4833, pp. 1–12. Springer, Berlin (2007)
2. Aoki, K., Shimoyama, T., Ueda, H.: Experiments on the linearalgebra step in the number field sieve. In: IWSEC. LNCS, vol.4752, pp. 58–73. Springer, Berlin (2007)
3. Cabal factorization of 2773 + 1. http://www.mail-archive.com/[email protected]/msg05260.html
10. Gordon, D.M., McCurley, K.S.: Massively parallel computationof discrete logarithms. In: Crypto. LNCS, vol. 740, pp. 312–323.Springer, Berlin (1992)
11. Kleinjung, T., Aoki, K., Franke, J., Lenstra, A.K., Thomé, E.,Bos, J.W., Gaudry, P., Kruppa, A., Montgomery, P.L., Osvik, D.A.,te Riele, H.J.J., Timofeev, A., Zimmermann, P.: Factorization of a768-bit RSA modulus. In: Crypto. LNCS, vol. 6223, pp. 333–350.Springer, Berlin (2010)
12. Lenstra, A.K., Lenstra, H.W. Jr.: The Development of the NumberField Sieve. LNM, vol. 1554. Springer, Berlin (1993)
13. Lenstra, A.K., Manasse, M.S.: Factoring by electronic mail. In:Eurocrypt. LNCS, vol. 434, pp. 355–371. Springer, Berlin (1989)
14. Nfs@home. http://escatter11.fullerton.edu/nfs15. The rsa factoring challenge faq. http://www.rsa.com/rsalabs/
node.asp?id=209416. Shor, P.W.: Algorithms for quantum computation: discrete loga-
rithms and factoring. In: Proceedings of the 35th Annual Sympo-sium on Foundations of Computer Science, pp. 124–134. IEEEComput. Soc., Los Alamitos (1994)
