Synchronization Transformationsfor
Parallel Computing
Pedro Dinizand
Martin Rinard
Department of Computer ScienceUniversity of California, Santa Barbara
http://www.cs.ucsb.edu/~{pedro,martin}
Motivation
Parallel Computing Becomes Dominant Form of Computation
Parallel Machines Require Parallel Software
Parallel Constructs Require New Analysis and Optimization Techniques
Our GoalEliminate Synchronization Overhead
Talk Outline
• Motivation
• Model of Computation
• Synchronization Optimization Algorithm
• Applications Experience
• Dynamic Feedback
• Related Work
• Conclusions
Model of Computation
• Parallel Programs• Serial Phases• Parallel Phases
• Single Address Space
• Atomic Operations on Shared Data• Mutual Exclusion Locks• Acquire Constructs• Release Constructs
Acq
S1MutualExclusionRegion
Rel
Reducing Synchronization Overhead
Acq
S1
S2
Rel
S3
Rel
Acq
Synchronization Optimization
Idea:Replace Computations that Repeatedly Acquire and Release the Same Lock with a Computation that Acquires and Releases the Lock Only Once
Result:Reduction in the Number of
Executed Acquire and Release Constructs
Mechanism:Lock Movement Transformations and
Lock Cancellation Transformations
Lock Cancellation
Acquire Lock Movement
Release Lock Movement
Synchronization Optimization Algorithm
Overview:
• Find Two Mutual Exclusion Regions With the Same Lock
• Expand Mutual Exclusion Regions Using Lock Movement Transformations Until They are Adjacent
• Coalesce Using Lock Cancellation Transformation to Form a Single Larger Mutual Exclusion Region
Interprocedural Control Flow Graph
Acquire Movement Paths
Release Movement Paths
Migration Paths and Meeting Edge
Intersection of Paths
Compensation Nodes
Final Result
Synchronization Optimization Trade-Off
• Advantage: • Reduces Number of Executed Acquires and Releases• Reduces Acquire and Release Overhead
• Disadvantage: May Introduce False Exclusion• Multiple Processors Attempt to Acquire Same Lock• Processor Holding the Lock is Executing Code that
was Originally in No Mutual Exclusion Region
False Exclusion Policy
Goal: Limit Potential Severity of False Exclusion
Mechanism: Constrain the Application of Basic
Transformations
• Original: Never Apply Transformations• Bounded: Apply Transformations only on
Cycle-Free Subgraphs of ICFG
• Aggressive: Always apply Transformations
Experimental Results
• Automatic Parallelizing Compiler Based on Commutativity Analysis [PLDI’96]
• Set of Complete Scientific Applications (C++ subset)• Barnes-Hut N-Body Solver (1500 lines of Code)• Liquid Water Simulation Code (1850 lines of Code)• Seismic Modeling String Code (2050 lines of Code)
• Different False Exclusion Policies
• Performance of Generated Parallel Code on Stanford DASH Shared-Memory Multiprocessor
Lock Overhead
0
20
40
60
Perc
enta
ge L
ock
Ove
rhea
d
Barnes-Hut (16K Particles)
Original
Bounded
Aggressive
Percentage of Time that the Single Processor Execution Spends Acquiring and Releasing Mutual Exculsion Locks
0
20
40
60
Perc
enta
ge L
ock
Ove
rhea
d
Water (512 Molecules)
Original
BoundedAggressive
0
20
40
60
Perc
enta
ge L
ock
Ove
rhea
d
String (Big Well Model)
OriginalAggressive
Contention OverheadC
onte
ntio
n Pe
rcen
tage
Percentage of Time that Processors Spend Waiting to Acquire Locks Held by Other Processors
100
0
25
50
75
0 4 8 12 16Processors
Barnes-Hut (16K Bodies)
0
25
50
75
100
0 4 8 12 16Processors
Water (512 Molecules)
0
25
50
75
100
0 4 8 12 16Processors
String (Big Well Model)
OriginalBoundedAggressive
0
2
4
6
8
10
12
14
16
Spe
edup
0 2 4 6 8 10 12 14 16Number of Processors
Ideal
Aggressive
Bounded
Original
Barnes-Hut (16384 bodies)
Performance Results : Barnes-Hut
Performance Results: Water
Ideal
Aggressive
Bounded
Original
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Spe
edup
Number of Processors
Water (512 Molecules)
Performance Results: String
String (Big Well Model)
Spe
edup
Number of Processors
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Ideal
Original
Aggressive
Choosing Best Policy
• Best False Exclusion Policy May Depend On• Topology of Data Structures• Dynamic Schedule Of Computation
• Information Required to Choose Best Policy Unavailable at Compile Time
• Complications• Different Phases May Have Different Best Policy• In Same Phase, Best Policy May Change Over Time
Solution: Dynamic Feedback
• Generated Code Consists of• Sampling Phases: Measure Performance of Different
Policies• Production Phases : Use Best Policy From Sampling
Phase
• Periodically Resample to Discover Changes in Best Policy
• Guaranteed Performance Bounds
Dynamic Feedback
AggressiveOriginalBounded
Time
Ove
rhea
d
Sampling Phase Production Phase Sampling Phase
AggressiveCodeVersion
Dynamic Feedback : Barnes-Hut
0
2
4
6
8
10
12
14
16
Spe
edup
0 2 4 6 8 10
12
14
16Number of Processors
Ideal
Aggressive
Dynamic Feedback
Bounded
Original
Barnes-Hut (16384 bodies)
Dynamic Feedback : Water
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Spe
edup
Number of Processors
Ideal
Bounded
Original
Aggressive
Dynamic Feedback
Water (512 Molecules)
Dynamic Feedback : String
String (BigWell Model)
0
2
4
6
8
10
12
14
16
0 2 4 6 8 10 12 14 16
Spe
edup
Number of Processors
Ideal
Original
Aggressive
Dynamic Feedback
Related Work
• Parallel Loop Optimizations (e.g. [Tseng:PPoPP95])
• Array-based Scientific Computations• Barriers vs. Cheaper Mechanisms
• Concurrent Object-Oriented Programs (e.g. [PZC:POPL95])
• Merge Access Regions for Invocations of Exclusive Methods
• Concurrent Constraint Programming• Bring Together Ask and Tell Constructs
• Efficient Synchronization Algorithms• Efficient Implementations of Synchronization
Primitives
Conclusions
• Synchronization Optimizations• Basic Synchronization Transformations for Locks• Synchronization Optimization Algorithm
• Integrated into Prototype Parallelizing Compiler• Object-Based Programs with Dynamic Data Structures• Commutativity Analysis
• Experimental Results• Optimizations Have a Significant Performance Impact• With Optimizations, Applications Perform Well
• Dynamic Feedback
