SWE 594 1
CMPE 49B Spec. Topics in CMPE: Multi-core Programming
picture of ASCI WHITE,
the most powerfulcomputer in the world in 2001
SWE 594 2
Von Neumann Architecture
CPU RAM Device Device
• sequential computer
BUS
SWE 594 3
Memory Hierarchy
Registers
Cache
Real Memory
Disk
CD
Fast
Slow
SWE 594 4
History of Computer Architecture
• 4 Generations (identified by logic technology)
1. Tubes
2. Transistors
3. Integrated Circuits
4. VLSI (very large scale integration)
SWE 594 5
PERFORMANCE TRENDS
SWE 594 6
PERFORMANCE TRENDS
• Traditional mainframe/supercomputer performance 25% increase per year
• But … microprocessor performance 50% increase per year since mid 80’s.
SWE 594 7
Moore’s Law
• “Transistor density doubles every 18 months”
• Moore is co-founder of Intel.
• 60 % increase per year
• Exponential growth
• PC costs decline.
• PCs are building bricks of all future systems.
SWE 594 8
VLSI Generation
SWE 594 9
Bit Level Parallelism(upto mid 80’s)
• 4 bit microprocessors replaced by 8 bit, 16 bit, 32 bit etc.
• doubling the width of the datapath reduces the number of cycles required to perform a full 32-bit operation
• mid 80’s reap benefits of this kind of parallelism (full 32-bit word operations combined with the use of caches)
SWE 594 10
Instruction Level Parallelism(mid 80’s to mid 90’s)
• Basic steps in instruction processing (instruction decode, integer arithmetic, address calculations, could be performed in a single cycle)
• Pipelined instruction processing
• Reduced instruction set (RISC)
• Superscalar execution
• Branch prediction
SWE 594 11
Thread/Process Level Parallelism(mid 90’s to present)
• On average control transfers occur roughly once in five instructions, so exploiting instruction level parallelism at a larger scale is not possible
• Use multiple independent “threads” or processes
• Concurrently running threads, processes
SWE 594 12
Evolution of the Infrastructure
• Electronic Accounting Machine Era: 1930-1950
• General Purpose Mainframe and Minicomputer Era: 1959-Present
• Personal Computer Era: 1981 – Present
• Client/Server Era: 1983 – Present
• Enterprise Internet Computing Era: 1992- Present
SWE 594 13
Sequential vs Parallel Processing
• physical limits reached
• easy to program
• expensive supercomputers
• “raw” power unlimited
• more memory, multiple cache
• made up of COTS, so cheap
• difficult to program
SWE 594 14
What is Multi-Core Programming ?
• Answer: It is basically parallel programming on a single computer box (e.g. a desktop, a notebook, a blade)
SWE 594 15
Amdahl’s Law• The serial percentage of a program is fixed. So speed-up obtained by
employing parallel processing is bounded.
• Lead to pessimism in in the parallel processing community and prevented development of parallel machines for a long time.
Speedup = 1
s + 1-s
P
• In the limit:
Spedup = 1/s s
SWE 594 16
Gustafson’s Law• Serial percentage is dependent on the number of
processors/input.
• Broke/disproved Amdahl’s law.
• Demonstrated achieving more than 1000 fold speedup using 1024 processors.
• Justified parallel processing
SWE 594 17
Grand Challenge Applications
• Important scientific & engineering problems identified by U.S. High Performance Computing & Communications Program (’92)
SWE 594 18
Flynn’s Taxonomy
• classifies computer architectures according to:
1. Number of instruction streams it can process at a time
2. Number of data elements on which it can operate simultaneously
Data Streams
Single Multiple
Single
Multiple
Instruction Streams
SISD SIMD
MIMDMISD
SWE 594 19
SPMD Model (Single Program Multiple Data)
• Each processor executes the same program asynchronously
• Synchronization takes place only when processors need to exchange data
• SPMD is extension of SIMD (relax synchronized instruction execution)
• SPMD is restriction of MIMD (use only one source/object)
SWE 594 20
Parallel Processing Terminology• Embarassingly Parallel:
-applications which are trivial to parallelize
-large amounts of independent computation
-Little communication
•Data Parallelism:
-model of parallel computing in which a single operation can be applied to all data elements simultaneously
-amenable to SIMD or SPMD style of computation
•Control Parallelism:
-many different operations may be executed concurrently
-require MIMD/SPMD style of computation
SWE 594 21
Parallel Processing Terminology• Scalability:
- If the size of problem is increased, number of processors that can be effectively used can be increased (i.e. there is no limit on parallelism).
- Cost of scalable algorithm grows slowly as input size and the number of processors are increased.
- Data parallel algorithms are more scalable than control parallel alorithms
• Granularity:
- fine grain machines: employ massive number of weak processors each with small memory
- coarse grain machines: smaller number of powerful processors each with large amounts of memory
SWE 594 22
Shared Memory Machines
Shared Address Space
process(thread)
process(thread)
process(thread)
process(thread)
process(thread)
•Memory is globally shared, therefore processes (threads) see single address space
•Coordination of accesses to locations done by use of locks provided by thread libraries
•Example Machines: Sequent, Alliant, SUN Ultra, Dual/Quad Board Pentium PC
•Example Thread Libraries: POSIX threads, Linux threads.
SWE 594 23
Shared Memory Machines• can be classified as:
-UMA: uniform memory access
-NUMA: nonuniform memory access
based on the amount of time a processor takes to access local and global memory.
Inter-connectionnetwork/or BUS
Inter-connection
network
Inter-connection
network
P
P
..
P
M
M
..
M
P
M
P
M
..
P
M
P
M
P
M
..
P
M
M
M
M
..
M
(a)(b) (c)
SWE 594 24
Distributed Memory Machines
Network
process
process
process
process
processM
M
M
M
M
•Each processor has its own local memory (not directly accessible by others)
•Processors communicate by passing messages to each other
•Example Machines: IBM SP2, Intel Paragon, COWs (cluster of workstations)
•Example Message Passing Libraries: PVM, MPI
SWE 594 25
Beowulf Clusters
•Use COTS, ordinary PCs and networking equipment
•Has the best price/performance ratio
PC cluster
SWE 594 26
Multi-Core Computing
• A multi-core microprocessor is one which combines two or more
independent processors into a single package, often a single integrated circuit.
• A dual-core device contains only two independent microprocessors.
SWE 594 27
Comparison of Different Architectures
CPU State
CacheExecution
unit
Single Core Architecture
SWE 594 28
Comparison of Different Architectures
CPU State
CacheExecution
unit
Multiprocessor
CPU State
CacheExecution
unit
SWE 594 29
Comparison of Different Architectures
CPU State
CacheExecution
unit
Hyper-Threading Technology
CPU State
SWE 594 30
Comparison of Different Architectures
CPU State
CacheExecution
unit
Multi-Core Architecture
CPU State
CacheExecution
unit
SWE 594 31
Comparison of Different Architectures
CPU State
Executionunit
Multi-Core Architecture with Shared Cache
CPU State
Cache
Executionunit
SWE 594 32
Comparison of Different Architectures
Multi-Core with Hyper-Threading Technology
CPU State
CacheExecution
unit
CPU State CPU State
CacheExecution
unit
CPU State
SWE 594 33
SWE 594 34
Rank Site System Processors Rmax Rpeak
1
DOE/NNSA/LLNL eServer Blue Gene Solution
131072 280600 367000United States IBM
2
IBM Thomas J. Watson Research Center eServer Blue Gene Solution
40960 91290 114688United States IBM
3
DOE/NNSA/LLNL eServer pSeries p5 575 1.9 GHz
12208 75760 92781United States IBM
4
NASA/Ames Research Center/NAS SGI Altix 1.5 GHz, Voltaire Infiniband
10160 51870 60960United States SGI
5
Commissariat a l'Energie Atomique (CEA)NovaScale 5160, Itanium2 1.6 GHz, Quadrics
8704 42900 55705.6France Bull SA
6
Sandia National Laboratories PowerEdge 1850, 3.6 GHz, Infiniband
9024 38270 64972.8United States Dell
7
GSIC Center, Tokyo Institute of TechnologySun Fire X64 Cluster, Opteron 2.4/2.6 GHz, Infiniband
10368 38180 49868.8Japan NEC/Sun
8
Forschungszentrum Juelich (FZJ) eServer Blue Gene Solution
16384 37330 45875Germany IBM
9
Sandia National Laboratories Red Storm Cray XT3, 2.0 GHz
10880 36190 43520United States Cray Inc.
10
The Earth Simulator Center Earth-Simulator
5120 35860 40960Japan NEC
Top 10 Most Powerful Computers in the World (as of 6/2006)
SWE 594 35
Most Powerful Computers in the World (as of 11/2007)
SWE 594 36
Top 500 Lists
• http://www.top500.org/list/2007/11
• http://www.top500.org/list/2007/06
• ……..
SWE 594 37
Application Areas in Top 500 List
SWE 594 38
Top 500 Statistics
• http://www.top500.org/stats
SWE 594 39
Grid Computing• provide access to computing power and various resources
just like accessing electrical power from electrical grid
• Allows coupling of geographically distributed resources
• Provide inexpensive access to resources irrespective of their physical location or access point
• Internet & dedicated networks can be used to interconnect distributed computational resources and present them as a single unified resource
• Resources: supercomputers, clusters, storage systems, data resources, special devices
SWE 594 40
Grid Computing
• the GRID is, in effect, a set of software tools, which when combined with hardware, would let users tap processing power off the Internet as easily as the electrical power can be drawn from the electricty grid.
• Examples of Grids:
-TeraGrid (USA) : http://www.teragrid.org
-EGEE Grid (Europe) : http://www.eu-egee.org/
- TR-Grid (Turkey) : http://www.grid.org.tr/
- Sun Grid Compute Utility (Commercial, pay-per-use) http://www.network.com/
SWE 594
GRID COMPUTING
Power Grid Compute Grid
SWE 594 42
ArcheologyAstronomyAstrophysicsCivil ProtectionComp. ChemistryEarth SciencesFinanceFusionGeophysicsHigh Energy PhysicsLife SciencesMultimediaMaterial Sciences…
>250 sites48 countries>50,000 CPUs>20 PetaBytes>10,000 users>150 VOs>150,000 jobs/day
SWE 594 43
Cloud Computing
•Style of computing in which IT-related capabilities are provided “as a service”,allowing users to access technology-enabled services from the Internet ("in the cloud") without knowledge of, expertise with, or control over the technology infrastructure that supports them.
•General concept that incorporates software as a service (SaaS), Web 2.0 and other recent, well-known technology trends, in which the common theme is reliance on the Internet for satisfying the computing needs of the users.
SWE 594
Cloud Computing
• Virtualisation provides separation between infrastructure and user runtime environment
• Users specify virtual images as their deployment building blocks
• Pay-as-you-go allows users to use the service when they want and only pay for what they use
• Elasticity of the cloud allows users to start simple and explore more complex deployment over time
• Simple interface allows easy integration with existing systems
44
SWE 594
Cloud computing is about much more than technological capabilities.
Technology is the mechanism, but, as in any shift in business, the driver is economics.
Nicholas Carr,The author of “The Big Switch”
SWE 594
We want to pay only for what we useAnd we want to control it accurately.
Better Economics
SWE 594
Facing New Challenges
• Complexity of modern IT infrastructures: physical servers, virtual machines, clusters, Grids, geographical distribution
• Cost of electricity• Credit crunch• Further pressures to reduce costs• Openness to the acceptable security concept
SWE 594
SWE 594
The Grid/Cloud
Advantages
• Lower cost
• Access to larger infrastructure
– Faster calculations
– More storage
• Speed
– Faster calculations
– Easier provisioning
Disadvantages
• Very complicated
• Security
• Lack of confidence
– Trust
– Compatibility
SWE 594
Issue Classic Grid Computing Cloud computing
Why we need it?(The Problem)
To enable the R&D community to achieve its research goals in reasonable time.Computation over large data sets, or of paralleizable compute-intensive applications.
Reduce IT costs.On-demand scalability for all applications, including research, development and business applications.
Main Target Market
First - AcademiaSecond – certain industries
Mainly Industry
Business Model – Where the money comes from?
AcademiaSponsor-based (Mainly government money).
Industry paysInternal Implementations.
Hosted by commercial companies, paid-for by users. Based on the economies of scale and expertise. Only pay for what you need, when you need it: (On- Demand + Pay per Use).
Grid and Clouds
SWE 594
Example Cloud: Amazon Web Services
• EC2 (Elastic Computing Cloud) is the computing service of Amazon
– Based on hardware virtualisation
– Users request virtual machine instances, pointing to an image (public or private) stored in S3
– Users have full control over each instance (e.g. access as root, if required)
– Requests can be issued via SOAP and REST
51
SWE 594
Example Cloud: Amazon Web Services
•S3 (Simple Storage Service) is a service for storing and accessing data on the Amazon cloud– From a user’s point-of-view, S3 is independent
from the other Amazon services– Data is built in a hierarchical fashion, grouped in
buckets (i.e. containers) and objects– Data is accessible via various protocols
•Elastic Block Store– Locally mounted storage– Highly available
52
SWE 594
Example Cloud: Amazon Web Services
• Other AWS services:
– SQS (Simple Queue Service)
– SimpleDB
– Billing services: DevPay
– Elastic IP (Static IPs for Dynamic Cloud Computing)
– Multiple Locations
53
SWE 594
Example Cloud: Amazon Web Services
• Pricing information
http://aws.amazon.com/ec2/
54
SWE 594
“By 2012 ,80 percent of Fortune 1000 companies
will pay for some cloud computing service ,
And
30 percent of them will pay for
cloud computing infrastructure.”
Gartner, 2008
Cloud Market
SWE 594
EC2 – “Google of the Clouds”
According to Vogels (Amazon CTO), 370,000 developers have registered for Amazon Web Services since their start in 2002, and the company now spends more bandwidth on the developers than it does on e-commerce. http://www.theregister.co.uk/2008/06/26/amazon_trumpets_web_services/
In the last two months of 2007 usage of Amazon Web Services grew by 40%
$131 million revenues in Q1 from AWS 60,000 customersThe majority of usage comes from banks, pharmaceuticals and other large corporations
SWE 594
• - CIOs -> Do more with Less (Energy costs / Recession will boost it)
- Lower cost for Scalability
- Enterprise IT budget - Spending 80% on MAINTENANCE
- In average, we utilize only 15% of our computing resources capacity
- Peak Times economy
- The Enterprise IT is not its core business
- Psychology of Internet/Cloud trust (SalesForce, Gmail, Internet banking, etc.)
- Ideal for Developers
Why Now? (Economy)
SWE 594 58
Models of Parallel Computers
1. Message Passing Model
- Distributed memory
- Multicomputer
2. Shared Memory Model
- Multiprocessor
- Multi-core
3. Theoretical Model
- PRAM
• New architectures: combination of 1 and 2.
SWE 594 59
Theoretical PRAM Model
• Used by parallel algorithm designers
• Algorithm designers do not want to worry about low level details: They want to concentrate on algorithmic details
• Extends classic RAM model
• Consist of :
– Control unit (common clock), synchronous
– Global shared memory
– Unbounded set of processors, each with its private own memory
SWE 594 60
Theoretical PRAM Model
• Some characteristics
– Each processor has a unique identifier, mypid=0,1,2,…
– All processors operate synhronously under the control of a common clock
– In each unit of time, each procesor is allowed to execute an instruction or stay idle
SWE 594 61
Various PRAM Models
weakest
strongest
EREW (exlusive read / exclusive write)
CREW (concurrent read / exclusive write)
CRCW (concurrent read / concurrent write)
Common (must write the same value)
Arbitrary (one processor is chosen arbitrarily)
Priority (processor with the lowest index writes)
(how write conflicts to the same memory location are handled)
SWE 594 62
Algorithmic Performance Parameters
• Notation
Input size
Time Complexity of the best sequential algorithm
Number of processors
Time complexity of the parallel algorithm when run on P processors
Time complexity of the parallel algorithm when run on 1 processors
SWE 594 63
Algorithmic Performance Parameters
• Speed-Up
• Efficiency
SWE 594 64
Algorithmic Performance Parameters
• Work = Processors X Time
– Informally: How much time a parallel algorithm will take to simulate on a serial machine
– Formally:
SWE 594 65
Algorithmic Performance Parameters
• Work Efficient:
– Informally: a work efficient parallel algorithm does no more work than the best serial algorithm
– Formally: a work efficient algorithm satisfies:
SWE 594 66
Algorithmic Performance Parameters
• Scalability:
– Informally, scalability implies that if the size of the problem is increased, the number of processors effectively used can be increased (i.e. there is no limit on parallelism)
– Formally, scalability means:
SWE 594 67
Algorithmic Performance Parameters
• Some remarks:
– Cost of scalable algorithm grows slowly as input size and the number of procesors are increased
– Level of ‘control parallelism’ is usually a constant independent of problem size
– Level of ‘data parallelism’ is an increasing function of problem size
– Data parallel algorithms are more scalable than control parallel algorithms
SWE 594 68
Goals in Designing Parallel Algorithms
• Scalability:
– Algorithm cost grows slowly, preferably in a polylogarithmic manner
• Work Efficient:
– We do not want to waste CPU cycles
– May be an important point when we are worried about power consumption or ‘money’ paid for CPU usage
SWE 594
•Array of N numbers can be summed in log(N) steps using
Summing N numbers in Parallel
x1 x2 x3 x4 x5 x6 x7 x8
x1+x2 x2 x3+x4 x4 x5+x6 x6 x7+x8 x8
x1+..+x4 x2 x3+x4 x4 x5+..+x8 x6 x7+x8 x8
x1+..+x8 x2 x3+x4 x4 x5+..+x8 x6 x7+x8 x8
step 1
step 2
step 3
result
N/2 processors
SWE 594
Prefix Summing N numbers in Parallel
x1 x2 x3 x4 x5 x6 x7 x8
x1+x2 x2+x3 x3+x4 x4+x5 x5+x6 x6+x7 x7+x8 x8
x1+..+x4 x2+..+x4 x3+..+x6 x4+..+x7 x5+..+x8 x6+..+x8 x7+x8 x8
x1+..+x8 x2+..+x8 x3+..+x8 x4+..+x8 x5+..+x8 x6+..+x8 x7+x8 x8
step 1
step 2
step 3
•Computing partial sums of an array of N numbers can be done in log(N) steps using N processors
SWE 594
Prefix Paradigm for Parallel Algorithm Design
•Prefix computation forms a paradigm for parallel algorithm development, just like other well known paradigms such as:
– divide and conquer, dynamic programming, etc.
•Prefix Paradigm: – If possible, transform your problem to prefix type computation– Apply the efficient logarithmic prefix computation
•Examples of Problems solved by Prefix Paradigm:
– Solving linear recurrence equations– Tridiagonal Solver– Problems on trees– Adaptive triangular mesh refinement
SWE 594
Solving Linear Recurrence Equations
• Given the linear recurrence equation:
• we can rewrite it as:
• if we expand it, we get the solution in terms of partial products of coefficients and the initial values z1 and z0 :
• use prefix to compute partial products
z a z b zi i i i i 1 2
z
z
a b z
zi
i
i i i
i
1
1
21 0
z
z
a b a b a b a b z
zi
i
i i i i i i
1
1 1 2 2 2 2 1
01 0 1 0 1 0 1 0...
SWE 594
Pointer Jumping Technique
•A linked list of N numbers can be prefix-summed in log(N)steps using N processors
step 1
step 3
x1 x2 x3 x4 x5 x6 x7 x8
x1+..+x4 x2+..+x5 x3+..+x6 x4+..+x7 x5+..+x8 x6+x7 x7+x8 x8
step 2
x1+.x2 x2+x3 x3+x4 x4+x5 x5+x6 x6+x7 x7+x8 x8
x1+..+x8 x2+..+x8 x3+..+x8 x4+..+x8 x5+..+x8 x6+..+x8 x7+x8 x8
SWE 594
Euler Tour Technique
b d
a
c
f ge
h i
Tree Problems:
•Preorder numbering•Postorder numbering•Number of Descendants•Level of each node
•To solve such problems, first transform the tree by linearizing it into a linked-list and then apply the prefix computation
SWE 594
Computing Level of Each Node by Euler Tour Technique
b d
a
c
f ge
h i
1
-1
1
-11
-1 1
1
-11
-1
-1
1 -11
-1 weight assignment:
1 -1
level(v) = pw(<v,parent(v)>)level(root) = 0
w(<u,v>)
pw(<u,v>)
igba d a c a g h g b f b e b a1-1 -1 1 -1 -1 -1 1 -1 1 1 -1 1 -1 1 1
1212123232101010
initial weights:
prefix:
SWE 594
Computing Number of Descendants by Euler Tour Technique
b d
a
c
f ge
h i
0
1
0
10
1 0
0
10
1
1
0 10
1 weight assignment:
0 1
# of descendants(v) = pw(<parent(v),v>) - pw(<v,parent(v)>) # of descendants(root) = n
w(<u,v>)
pw(<u,v>)
igba d a c a g h g b f b e b a01 1 0 1 1 1 0 1 0 0 1 0 1 0 0
0011222334566778
initial weights:
prefix:
SWE 594
Preorder Numbering by Euler Tour Technique
b d
a
c
f ge
h i
1
0
1
01
0 1
1
01
0
0
1 01
0 weight assignment:
1 0
preorder(v) = 1 + pw(<v,parent(v)>)preorder(root) = 1
w(<u,v>)
pw(<u,v>)
igba d a c a g h g b f b e b a10 0 1 0 0 0 1 0 1 1 0 1 0 1 1
1223345566667788
initial weights:
prefix:
1
2
34
5
6 7
8 9
SWE 594
Postorder Numbering by Euler Tour Technique
b d
a
c
f ge
h i
0
1
0
10
1 0
0
10
1
1
0 10
1 weight assignment:
0 1
postorder(v) = pw(<parent(v),v>)
postorder(root) = n
w(<u,v>)
pw(<u,v>)
igba d a c a g h g b f b e b a01 1 0 1 1 1 0 1 0 0 1 0 1 0 0
0011222334566778
initial weights:
prefix:
9
6
1 25
3 4
78