Classification of . . . Levels of Parallelism Quantitative . . . Network Topologies Design of Parallel . . . Page 1 of 64 Introduction to Parallel Programming Ralf-Peter Mundani Ioan Lucian Muntean ATHENS Course on Parallel Numerical Simulation Munich, March 20-24, 2006 Introduction to Parallel Programming March 20 & 21, 2006 Ralf-Peter Mundani & Ioan Lucian Muntean Department of Computer Science – Chair V Technische Universität München, Germany
Transcript
Classification of . . .
Levels of Parallelism
Quantitative . . .
Network Topologies
Design of Parallel . . .
Page 1 of 64
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
ATHENS Course on
Parallel Numerical Simulation
Munich, March 20-24, 2006
Introduction to Parallel Programming
March 20 & 21, 2006
Ralf-Peter Mundani & Ioan Lucian MunteanDepartment of Computer Science – Chair VTechnische Universität München, Germany
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
1. Classification of Parallel Computers
Parallel Computers
• parallel computers consist of a set of processing elements that can collaborate in acoordinated and (partially) simultaneous way in order to solve a joint task
• possible appearances of such processing elements (in a broader sense):
– specialized units (the steps of a vector pipeline or the vector pipeline of a vectorcomputer’s vector unit, for example)
– parallel features in modern monoprocessors (superscalar processor architec-ture, fine-grain parallelism via instruction pipelining, cooperation of CPU, buscontrol, DMA unit, and graphics card, VLIW processors (very long instructionword), multi-threading processor technology, for example)
– several uniform arithmetical units (the processing elements of an array com-puter, for example)
– processors or processing nodes of a multiprocessor computer (i.e. the actualparallel computers in a narrower sense)
– complete stand-alone computers, connected via a LAN (work station or PCclusters as virtual parallel computers)
– parallel computers or clusters connected via a remote network (WAN) (so-called metacomputers)
• target machines in the following: multi- and specialized processors as well as clusters(i.e. the so-called high-performance architectures or supercomputers)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Commercial Parallel Computers• manufacturers: starting from 1983, big players and small start-ups• names have been coming and going rapidly• see the table below for producers of commercial parallel computers based on micro-
processor CPUs between 1984 and 1993 and their status 2003 (taken from [Qui03])• in addition to that: several manufacturers of vector computers, non-standard archi-
tectures (e.g. Thinking Machines and their Connection Machines starting from 1986and ending in the nineties)
Company Country Year Status in 2001Sequent U.S. 1984 Acquired by IBMIntel U.S. 1984 Out of the businessMeiko U.K. 1985 BankruptnCUBE U.S. 1985 Out of the businessParsytec Germany 1985 Out of the businessAlliant U.S. 1985 BankruptEncore U.S. 1986 Out of the businessFloating Point Systems U.S. 1986 Acquired by SunMyrias Canada 1987 Out of the businessAmetek U.S. 1987 Out of the businessSilicon Graphics U.S. 1988 ActiveC-DAC India 1991 ActiveKendall Square Research U.S. 1992 BankruptIBM U.S. 1993 ActiveNEC U.S. 1993 ActiveSun Micrososystems U.S. 1993 ActiveCray Research U.S. 1993 Active (as Cray Inc.)
"Out of the business" means the company is no longer selling general-purpose parallel computer systems.
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Supercomputers
• supercomputing or high-performance scientific computing as the most important applica-tion of the big number crunchers
• national initiatives due to huge budget requirements
– ASCI – Adavanced Strategic Computing Initiative in the US
* in the sequel of the nuclear testing moratorium in 1992/1993
* decision: develop, build, and install a series of 5 supercomputers of up to$100 million each in the US
* start: ASCI Red (1997, Intel-based, Sandia National Laboratory, the world’sfirst TFLOPS computer)
* then: ASCI Blue Pacific (1998, Lawrence Livermore National Lab), ASCIBlue Mountain, ASCI White, ...
– meanwhile new High-End Computing memorandum in the US 2004
– federal Bundeshöchstleistungsrechner initiative in Germany
* decision in the midth nineties
* 3 federal supercomputing centres in Germany (München, Stuttgart, andJülich), one new installation each year, the newest one to be among thetop 10 of the world
• overview and state of the art: Top500 list (every six months), see http://www.top500.org
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
BlueGene/L – World’s No. 1 from 2004 -
• installed in 2004 in Lawrence Livermore National Laboratory, United States.• 2 processors, 5.6 GFlop/s, 512 MB memory per node• 65,536 (131,072) Nodes (Processors) achieved through extreme scalability, 367 TFlop/s
theoretical peak, 32 TB memory, 400 TB aggregate global disk• Dual PowerPC 440 microprocessor technology• 1,024 x 1-Gb/s Ethernet external networking• delivered bandwidth is 0.684 TB/s, cost effective for molecular dynamics and turbu-
lence• about 2 MW power needed for computer and cooling, about 4.7 billion joules of heat
is generated per hour• more than 5000 cables are present in the machine and the aggregate cable length
counts to more than 12 miles• Area of computer = 2,500 sq.ft (232 sq.m approx.)• Linpack Benchmark = 70.72 TFlop/s in Nov. 2004, 136.8 TFlop/s in Jun. 2005 and
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Flynn’s Classes
• SISD:
– the classical monoprocessor following von Neumann’s principle
• SIMD:
– array computers: consist of a large number (65536 and more) of uniform pro-cessing elements arranged in a regular way, which – under central control – allapply the same instruction to some part of the data each, simultaneously
– vector computers: consist of at least one vector pipeline (functional unit de-signed as a pipeline for processing vectors of floating point numbers)
• MISD:
– a pipeline of multiple independently executing functional units operating on asingle stream of data, forwarding results from one functional unit to the next
– not very popular class (mainly for special applications such as Digital SignalProcessing)
– example: systolic array – a network of primitive processing elements that "pump"data (for example, a hardware priority queue with constant-complexity opera-tions can be built out of primitive three-number sorting elements)
• MIMD:
– multiprocessor systems, i.e. the classical parallel computers
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
A Hybrid Type: DSM/VSM
• central issues:
– scalability: How simple is it to add new nodes (processors) to the system?
– programming model: How complicated is programming?
– portability: How simple is portation/migration, i.e. the transfer from one pro-cessor to another one, if executability and functionality shall be preserved?
– load distribution: How difficult is it to obtain a uniform distribution of the workload among the processors?
• message-coupled systems are advantageous concerning scalability, memory-coupledsystems are typically better with respect to the other aspects
• idea: combine advantages of both
• DSM (distributed shared memory) or VSM (virtual shared memory):
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
2. Levels of ParallelismGranularity
• The decision which type of parallel architecture is best-suited for a given parallelprogram strongly depends on the character and, especially, on the granularity ofparallelism.
• some remarks on granularity:
– qualitative meaning: the level on which work is done in parallel
– we distinguish coarse-grain and fine-grain parallelism
– quantitative meaning: ratio of computational effort and communication or syn-chronization effort (roughly speaking the number of instructions between twonecessary steps of communication)
• starting point of the following considerations: a parallel program (explicit parallelism)
• typically, five different levels are identified:
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Block, Instruction, and Sub-Instruction Level
• block level:
– here, the units running in parallel are blocks of instructions or light-weight pro-cesses (threads, see Section 2.1)
– smaller number of instructions, which share the address space with other blocks
– examples: threads according to POSIX standard in multi-threading operatingsystems, loops in numerical programs
– communication via shared variables and synchronization mechanisms
• instruction level:
– parallel execution of machine instructions
– optimizing compilers can increase this potential by modifying the order of thecommands (better exploitation of superscalar architecture and pipelining mech-anisms)
• sub-instruction level:
– instructions are subdivided still further in units that can be executed in parallelor via overlapping
– examples: pipelining in superscalar processors, vector operations
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Techniques of Parallel Work
• the different levels of parallelism have methods of parallel work in the hardware astheir counterparts
• objective: best exploitation of the inherent parallel potential
• levels of parallel work:
– computer coupling: useful for program level only, sometimes also for processlevel
– processor coupling:
* message-coupling for program and process level
* memory-coupling for program, process, and block level
– parallel work within the processor architecture: instruction pipelining, super-scalar organization, VLIW and so on for instruction level only, eventually forsub-instruction level
– SIMD techniques: concerning the sub-instruction level in vector and array com-puters
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Notions of Time in the Execution of Instructions
• not only simple instruction time, but more detailed considerations instead:
– execution time T of a parallel program:time between start of the execution on the first participating processor and endof all computations on the last participating processor
– computation time Tcomp of a parallel program:part of the execution time used for computations
– communication time Tcomm of a parallel program:part of the execution time used for send and receive operations
– idle time Tidle of a parallel program:part of execution time used for waiting (for sending or receiving)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Average Parallelism
• total work during a parallel computation:
W (p) := l ·pX
i=1
i · ti ,
where
– l: performance of one single processor
– p: number of processors
– ti: time when exactly i processors are busy
• average parallelism:
A(p) :=
Ppi=1 i · tiPp
i=1 ti=
1
l· W (p)Pp
i=1 ti
• for A(p), there exist several theoretical estimates (typically quite pessimistic), whichwere often used as arguments against massively parallel systems
• example: estimate of Minsky (1971):
– problem class: in the first step, p processors can be used, in the second steponly p/2, and so on (today considered as of no big relevance)
– example: parallel addition of 2p numbers on p processors
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Scalability, Overhead, Parallel Index
• scalability:
– objective: adding more processors to the system shall reduce the executiontime significantly (efficiency close to 1) without necessary program modifica-tions
– scalability requires a sufficient problem size: although one porter may carryone suitcase in a minute, sixty won’t do it in a second – however, 60 porters willcarry 60 suitcases in one minute!
– therefore often scaled problem analysis: with increasing number of processors,increase problem size, too (then, efficiency 1 means constant execution times)
• overhead R(p) for the parallelization:
– definition:R(p) :=
P (p)
P (1), bound: 1 ≤ R(p)
– describes the additional number of operations for organization, synchroniza-tion, and communication
• parallel index I(p):
– definitionI(p) =
P (p)
T (p)
– measure for the average degree of parallelism, counts the number of paralleloperations per time unit
– relative acceleration (taking into account (parallel) overhead)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Amdahl’s Law
• the probably most important and most famous estimate for the speed-up
• underlying model:
– each program consists of parallelizable parts and of parts that can be executedonly in a sequential way; let the sequential part be s, 0 ≤ s ≤ 1
– then, the following holds for execution time and speed-up:
T (p) = T (1) · 1− s
p+ T (1) · s; S(p) =
T (1)
T (p)=
11−s
p+ s
– thus, we get Amdahl’s Law:
S(p) ≤ 1
s
• meaning:
– the sequential part can have a dramatic impact on the speed-up
– example: even for just one per cent (s = 0.01), more than a speed-up of 100is impossible – even on massively parallel computers with p much bigger than100!
– therefore central effort of all (parallel) algorithmics: keep s small!
– this is possible: about 75% of all LINPACK routines fulfil s < 0.1
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Model of Gustafson
• alternative model for speed-up prediction or estimation
• underlying idea:
– normalize the execution time on the parallel machine to 1
– there: non-parallelizable part σ
– hence execution time on the monoprocessor:
T (1) = σ + p · (1− σ)
– this results in a speed-up of
S(p) = σ + p · (1− σ) = p + σ · (1− p)
• difference to Amdahl:
– sequential part – with respect to execution time on one processor – is not con-stant, but gets smaller with increasing p:
s(p) =σ
σ + p · (1− σ)∈ ]0, 1[
– often more realistic, because more processors are used especially for largerproblem size, and here parallelizable parts typically increase (more computa-tions, less declarations ...)
– in Gustafson’s model, speed-up is not bounded for increasing p
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
CCR
• important quantity measuring the success of a parallelization: communication-computation-ratio (CCR)
– gives the relation of pure communication time and pure computing time
– a small CCR is favourable (a lot of computations, only a small amount of com-munication)
– typically: CCR decreases with increasing problem size
• example (see exercises):
– consider a full N ×N -matrix
– consider the following iterative method: in each step, each matrix element isreplaced by the average of its eight neighbour values (analogous definition atthe boundary)
– for the update of each row, we need the two neighbouring rows
– p processors, decompose the matrix in p blocks of N/p rows
– computing time: 8N ·N/p
– communication time: 2(p− 1) ·N– hence, CCR is (p2 − p)/(4N)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
4. Network Topologies
Valuation Criteria
• interconnection network: medium of communication of the processors or nodes, re-spectively
• topology: way of arrangement or connections of the processor nodes (modulo sym-metries)
• different criteria to judge the quality of a network topology:
– complexity, costs: overall hardware costs to realize the net, that is basicallynumber of connecting lines, of network cards, and of switches and so on
– connectivity of a node: number of connections that exist between this node andother nodes (important cost factor)
– diameter: maximum distance (number of direct connections) between two nodes
– regularity of the network: extent of deviations in the local net quantities (con-nectivity, for example)
– length of lines: physical length of the connections (a room, a building; shortdistances are advantageous)
– blocking: does an existing communication between two nodes block the com-munication of other pairs of nodes (optimum: blocking-free networks)?
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Static Networks (cont’d)
• three-dimensional topologies:
– cube: connectivity is three, diameter is three
• four-dimensional topologies:
– hypercube: connectivity 4, in the general n-dimensional case n; constructionprinciple: connect in two n-dimensional hypercubes the respective nodes toconstruct the n + 1-dimensional hypercube
– dual (hyper)cube: processors on the edges; results in a better connectivity 2
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Dynamic Networks
• bus topologies: as simple bus the most frequently used structure for symmetric multi-processors; poor scalability; multiple buses are fault tolerant, but rare
• Modern and more powerful dynamic topologies are based upon switches, i.e. con-nection elements with n inputs and n outputs; all CPUs or memory elements etc. areconnected to it, and the switch provides the one-to-one connections internally.
• internal structure of general switches: built as a cascadic network of several elemen-tary switches (each with a small number of input and outputs)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Dynamic Networks (cont’d)
• crossbar switch:
– hardware component which can be switched such that, in a set of input andoutput slots, all possible disjoint pairs can communicate simultaneously withoutany blocking
– for processor-processor coupling (message-coupled systems) or for processor-memory-coupling (memory-coupled systems)
– depending on the state of the switch elements the different pairs can communi-cate
– excellent performance, but very high hardware costs (especially for a largernumber of nodes to be connected)
– example: the Earth Simulator consists of 640 nodes, all of which are directlyconnected via a 640-port crossbar switch
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
5. Design of Parallel Programs
Design Patterns – Ways of Parallelizing
• main problem: how can we parallelize, and what has to be parallelized?
• larger differences between the sequential program and its parallel version are possi-ble
• we meet several design patterns again and again:
– function parallelism: processing of different components of the program suchas blocks or procedures is done in parallelexample: assembly-line production in automotive industry
– data parallelism: parts of programs or whole programs are applied to differentpartitions of the overall data in parallelexample: solve a partial differential equation numerically with a domain decom-position method
– competition parallelism: all processors solve the same problem, but with differ-ent strategies or algorithmsexample: determine the fastest of k given sorting algorithms
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Data Parallelism
• each processor deals with a part of the data only, but covers all functional subtasks
• characteristics:
– assumption: underlying uniform data structure (array, e.g.)
– each processor covers the whole algorithm or program, but with a reduced input
– extreme example: array computer
– advantages: one program for all processors, often good parallelization proper-ties
• structure of data-parallel programs:
– static: compiler defines parallelization and order (simple organization, but noflexibility if runtimes of the jobs differ or change)
– dynamic: control of the parallel processing is done dynamically during runtime,according to program organization or runtime system (more complex organiza-tion, but more flexible with respect to local changes of the load)
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Dynamic Structure of the Program
• allows for a permanent dynamic load balancing
• different options of organization:
– order placing (master-slave):
* master nodes places orders to slave nodes: this requires two differenttypes of programs or program parts, at least (for the master and for theslaves)
* often, the master process is a bottleneck (especially for large numbers ofslaves)
– order fetching:
* there is a set of tasks not processed so far (bag of tasks) where idle pro-cessors look for new tasks to deal with
* bag must be accessible to all tasks, hence primarily used in the memory-coupled case
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Important Problem of Parallel Programming
• data access does not happen with some constant speed:
– this observation already holds for monoprocessors: registers, cache, mainmemory and so on
– except for true shared memory, we now have an additional level in memoryhierarchy:local memory can be accessed in shorter time than remote memory (holds fordirect remote access and for message-coupling)
– ratio can be up to 1:100 or 1:100000
• therefore: organize program such that data access can be realized locally as often aspossible
• this is a fundamental principle of parallel programming
• a lot of consequences for parallel algorithms, which often have to be modified com-pared with their sequential counterparts:
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Tools for the Development of Parallel Programs
• the use of specific tools during the different steps is crucial
• such tools are available for the late phases, primarily
• objective: get from a manual to a (semi-) automated program development
• especially for the subdivision, we need estimates of various quantities of the program(resources needed, memory requirements, communication profile)
• examples of development tools:
– simulators
– profilers: estimate the runtime of (parts of) programs already during compilation
– monitors: observe the program’s execution (hardware monitors, software mon-itors, ...; important: small influence on measurements);note that this notion of a monitor has nothing to do with the one we studied inthe context of synchronization!
– parallel debuggers: facilitate the detection of errors in parallel programs
– program flow visualizers: tools for graphical interpretation
Introduction to Parallel ProgrammingRalf-Peter Mundani
Ioan Lucian Muntean
Alternative Phase Model
• strongly heuristic proceeding
• four phases in the development of parallel programs:
– partitioning: subdivision of program and data into smaller partsprimary objective: allow for as much parallelism as possible, without taking intoaccount practical restrictions of the hardware and so on
– communication: define communication requirements and appropriate commu-nication hardware
– bundling: evaluation of the first two phases’ results with respect to performanceand costs
– mapping: assign tasks to processors (often difficult balancing between optimumprocessor loads and minimum communication costs); can be realized staticallyby the compiler or dynamically during runtime via load balancing
• focus of the first two phases: detection of parallelism, good scalability
• focus of the last two phases: locality, parallel efficiency
