ECE 1747: Parallel Programming
Basics of Parallel Architectures:Shared-Memory Machines
Two Parallel Architectures
• Shared memory machines.• Distributed memory machines.
Shared Memory: Logical View
proc1 proc2 proc3 procN
Shared memory space
Shared Memory Machines
• Small number of processors: shared memory with coherent caches (SMP).
• Larger number of processors: distributed shared memory with coherent caches (CC-NUMA).
SMPs
• 2- or 4-processors PCs are now commodity.• Good price/performance ratio.• Memory sometimes bottleneck (see later).• Typical price (8-node): ~ $20-40k.
Physical Implementation
proc1 proc2 proc3 procN
Shared memory
cache1 cache2 cache3 cacheN
bus
Shared Memory Machines
• Small number of processors: shared memory with coherent caches (SMP).
• Larger number of processors: distributed shared memory with coherent caches (CC-NUMA).
CC-NUMA: Physical Implementation
proc1 proc2 proc3 procN
mem2 mem3 memNmem1
cache2cache1 cacheNcache3
inter-connect
Caches in Multiprocessors
• Suffer from the coherence problem:– same line appears in two or more caches– one processor writes word in line– other processors now can read stale data
• Leads to need for a coherence protocol– avoids coherence problems
• Many exist, will just look at simple one.
What is coherence?
• What does it mean to be shared?• Intuitively, read last value written.• Notion is not well-defined in a system
without a global clock.
The Notion of “last written” in a Multi-processor System
w(x)
w(x)
r(x)
r(x)
P0
P1
P2
P3
The Notion of “last written” in a Single-machine System
w(x) w(x) r(x) r(x)
Coherence: a Clean Definition
• Is achieved by referring back to the single machine case.
• Called sequential consistency.
Sequential Consistency (SC)
• Memory is sequentially consistent if and only if it behaves “as if” the processors were executing in a time-shared fashion on a single machine.
Returning to our Example
w(x)
w(x)
r(x)
r(x)
P0
P1
P2
P3
Another Way of Defining SC
• All memory references of a single process execute in program order.
• All writes are globally ordered.
SC: Example 1
w(x,1) w(y,1)
r(x) r(y)
Initial values of x,y are 0.
What are possible final values?
SC: Example 2
w(x,1) w(y,1)
r(y) r(x)
SC: Example 3
w(x,1)
w(y,1)
r(y) r(x)
SC: Example 4
w(x,1)
w(x,2)
r(x)
r(x)
Implementation
• Many ways of implementing SC.• In fact, sometimes stronger conditions.• Will look at a simple one: MSI protocol.
Physical Implementation
proc1 proc2 proc3 procN
Shared memory
cache1 cache2 cache3 cacheN
bus
Fundamental Assumption
• The bus is a reliable, ordered broadcast bus.– Every message sent by a processor is received
by all other processors in the same order.• Also called a snooping bus
– Processors (or caches) snoop on the bus.
States of a Cache Line
• Invalid• Shared
– read-only, one of many cached copies• Modified
– read-write, sole valid copy
Processor Transactions
• processor read(x)• processor write(x)
Bus Transactions
• bus read(x) – asks for copy with no intent to modify
• bus read-exclusive(x)– asks for copy with intent to modify
State Diagram: Step 0
I S M
State Diagram: Step 1
I S M
PrRd/BuRd
State Diagram: Step 2
I S M
PrRd/BuRdPrRd/-
State Diagram: Step 3
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
State Diagram: Step 4
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
State Diagram: Step 5
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
PrWr/-
State Diagram: Step 6
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
PrWr/-
BuRd/Flush
State Diagram: Step 7
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
PrWr/-
BuRd/Flush
BuRd/-
State Diagram: Step 8
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
PrWr/-
BuRd/Flush
BuRd/-
BuRdX/-
State Diagram: Step 9
I S M
PrRd/BuRdPrRd/-
PrWr/BuRdX
PrWr/BuRdX
PrWr/-
BuRd/Flush
BuRd/-
BuRdX/-
BuRdX/Flush
In Reality
• Most machines use a slightly more complicated protocol (4 states instead of 3).
• See architecture books (MESI protocol).
Problem: False Sharing
• Occurs when two or more processors access different data in same cache line, and at least one of them writes.
• Leads to ping-pong effect.
False Sharing: Example (1 of 3)
#pragma omp parallel for schedule(cyclic)for( i=0; i<n; i++ )
a[i] = b[i];
• Let’s assume: – p = 2– element of a takes 4 words– cache line has 32 words
False Sharing: Example (2 of 3)
a[0] a[1] a[2] a[3] a[4] a[5] a[6] a[7]
cache line
Written by processor 0Written by processor 1
False Sharing: Example (3 of 3)
P0
P1
a[0]
a[1]
a[2] a[4]
a[3] a[5]
...inv data
Summary
• Sequential consistency.• Bus-based coherence protocols.• False sharing.
Algorithms for Scalable Synchronization on Shared-
Memory Multiprocessors
J.M. Mellor-Crummey, M.L. Scott(MCS Locks)
Introduction
• Busy-waiting techniques – heavily used in synchronization on shared memory MPs
• Two general categories: locks and barriers– Locks ensure mutual exclusion– Barriers provide phase separation in an
application
Problem
• Busy-waiting synchronization constructs tend to:– Have significant impact on network traffic due
to cache invalidations– Contention leads to poor scalability
• Main cause: spinning on remote variables
The Proposed Solution
• Minimize access to remote variables• Instead, spin on local variables• Claim:
– It can be done all in software (no need for fancy and costly hardware support)
– Spinning on local variables will minimize contention, allow for good scalability, and good performance
Spin Lock 1: Test-and-Set Lock
• Repeatedly test-and-set a boolean flag indicating whether the lock is held
• Problem: contention for the flag (read-modify-write instructions are expensive)– Causes lots of network traffic, especially on
cache-coherent architectures (because of cache invalidations)
• Variation: test-and-test-and-set – less traffic
Test-and-test with Backoff Lock• Pause between successive test-and-set (“backoff”)• T&S with backoff idea:
while test&set (L) fails {pause (delay);delay = delay * 2;
}
Spin Lock 2: The Ticket Lock
• 2 counters (nr_requests, and nr_releases)• Lock acquire: fetch-and-increment on the
nr_requests counter, waits until its “ticket” is equal to the value of the nr_releases counter
• Lock release: increment of the nr_releases counter
Spin Lock 2: The Ticket Lock
• Advantage over T&S: polls with read operations only
• Still generates lots of traffic and contention• Can further improve by using backoff
Array-Based Queueing Locks
• Each CPU spins on a different location, in a distinct cache line
• Each CPU clears the lock for its successor (sets it from must-wait to has-lock)
• Lock-acquire while (slots[my_place] == must-wait);
• Lock-release slots[(my_place + 1) % P] = has-lock;
List-Based Queueing Locks (MCS Locks)
• Spins on local flag variables only• Requires a small constant amount of space
per lock
List-Based Queueing Locks (MCS Locks)
• CPUs are all in a linked list: upon release by current CPU, lock is acquired by its successor
• Spinning is on local flag• Lock points at tail of queue (null if not held)• Compare-and-swap allows for detection if it
is the only processor in queue and atomic removal of self from the queue
List-Based Queueing Locks (MCS Locks)
• Spin in acquire_lock waits for lock to become free
• Spin in release_lock compensates for the time window between fetch-and-store and assignment to predecessor->next in acquire_lock
• If no compare_and_swap – cumbersome
The MCS Tree-Based Barrier
• Uses a pair of P (nr. of CPUs) trees: arrival tree, and wakeup tree
• Arrival tree: each node has 4 children • Wakeup tree: binary tree
– Fastest way to wake up all P processors
Hardware Description• BBN Butterfly 1 – DSM multiprocessor
– Supports up to 256 CPUs, 80 used in experiments– Atomic primitives allow fetch_and_add,
fetch_and_store (swap), test_and_set• Sequent Symmetry Model B – cache-coherent,
shared-bus multiprocessor– Supports up to 30 CPUs, 18 used in experiments– Snooping cache-coherence protocol
• Neither supports compare-and-swap
Measurement Technique
• Results averaged over 10k (Butterfly) or 100k (Symmetry) acquisitions
• For 1 CPU, time represents latency between acquire and release of lock
• Otherwise, time represents time elapsed between successive acquisitions
58
Spin Locks on Butterfly
59
Spin Locks on Butterfly
Spin Locks on Butterfly
• Anderson’s fares poorer because the Butterfly lacks coherent caches, and CPUs may spin on statically unpredictable locations – which may not be local
• T&S with exponential backoff, Ticket lock with proportional backoff, MCS all scale very well, with slopes of 0.0025, 0.0021 and 0.00025 μs respectively
61
Spin Locks on Symmetry
62
Spin Locks on Symmetry
63
Latency and Impact of Spin Locks
Latency and Impact of Spin Locks
• Latency results are poor on Butterfly because:– Atomic operations are inordinately expensive in
comparison to non-atomic ones– 16-bit atomic primitives on Butterfly cannot
manipulate 24-bit pointers
65
Barriers on Butterfly
66
Barriers on Butterfly
67
Barriers on Symmetry
Barriers on Symmetry
• Different results from Butterfly because:– More CPUs can spin on same location (own
copy in local cache)– Distributing writes across different memory
modules yields no benefit because the bus serializes all communication
Conclusions
• Criteria for evaluating spin locks:– Scalability and induced network load– Single-processor latency– Space requirements– Fairness– Implementability with available atomic
operations
Conclusions
• MCS lock algorithm scales best, together with array-based queueing on cache-coherent machines
• T&S and Ticket Locks with proper backoffs also scale well, but incur more network load
• Anderson and G&T: prohibitive space requirements for large numbers of CPUs
Conclusions
• MCS, array-based, and Ticket Locks guarantee fairness (FIFO)
• MCS benefits significantly from existence of compare-and-swap
• MCS is best when contention expected: excellent scaling, FIFO ordering, least interconnect contention, low space reqs.