Parallel Program DesignTasks, Critical Path

Ned Nedialkov

McMaster UniversityCanada

SE 3F03January 2015

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Outline

Foster’s Methodology

Tasks

Degrees of concurrency

Critical path

Examples

c© 2013–15 Ned Nedialkov 2/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Foster’s Methodology

1. Partitioning. Divide the computation into small tasks thatcan be done in parallel

2. Communication. Determine the communications betweenthe tasks

3. Aggregation (or agglomeration). Combine tasks andcommunications into larger tasks

4. Mapping. Assign the combined tasks to processes (orthreads)

From Ian Foster, Designing and Building Parallel Programs,http://www.mcs.anl.gov/~itf/dbpp/

c© 2013–15 Ned Nedialkov 3/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Tasks

I We can decompose a computation into smaller parts ortasks

I The goal is to determine which can be executed in parallelI Fine-grained decomposition: many small tasksI Coarse-grained decomposition: small number of large

tasksI Task-dependency graph

I directed acyclic graphI nodes are tasksI there is an edge between task A and task B if B must be

executed after A

c© 2013–15 Ned Nedialkov 4/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Degrees of concurrency

I Maximum degree of concurrency: the maximum number oftasks that can be executed in parallel

I Average degree of concurrency: the average number oftasks that can be executed in parallel

I The average degree of concurrency is a more usefulmeasure

c© 2013–15 Ned Nedialkov 5/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Critical path

I Start nodes: nodes with no incoming edgesI Finish nodes: nodes with no outgoing edgesI Critical path: the longest directed path between any pair of

start and finish nodesI Critical path length: sum of the weights of the nodes on a

critical pathI

ave degree of concurrency =total amount of workcritical path length

c© 2013–15 Ned Nedialkov 6/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

ExampleTask-dependency graph

I maximum degree of concurrency is 6I critical path length is 5I total amount of work is 14 (assuming each task takes one

unit of time)I average degree of concurrency is 14/5 = 2.8

c© 2013–15 Ned Nedialkov 7/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 2 processes and the order in which thetasks are executed

I First number is the process numberI The speedup is 14/7 = 2

0,1 0,30,21,1 1,2

1,3 1,4

1,5

0,7

0,6

1,6

0,4

1,7 0,5

c© 2013–15 Ned Nedialkov 8/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 3 processesI The speedup is 14/5 = 2.8

1,3 2,10,12,3 1,1

1,41,2

2,2

0,5

0,4

2,4

0,2

1,5 0,3

c© 2013–15 Ned Nedialkov 9/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I An assignment to 4 processesI The speedup is 14/5 = 2.8

1,2 0,12,12,2 3,1

1,3 0,2

1,1

0,5

0,4

2,3

3,2

1,4 0,3

I What is the speedup on 8 processes?

c© 2013–15 Ned Nedialkov 10/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

Example: LU decomposition

I We want to compute the LU decomposition of a matrix AI Assume A consists of 3× 3 blocksI We wantA11 A12 A13

A21 A22 A23A31 A32 A33

→L11 0 0

L21 L22 0L31 L32 L33

︸ ︷︷ ︸

L

U11 U12 U130 U22 U230 0 U33

︸ ︷︷ ︸

U

c© 2013–15 Ned Nedialkov 11/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I We need to determine the Lij and Uij

I Consider L times the first column of UI We have

A11 = L11U11 compute L11 and U11 (1)

A21 = L21U11 L21 = A21U−111 (2)

A31 = L31U11 L31 = A31U−111 (3)

I From multiplying the first row of L times U, we obtain

A12 = L11U12 U12 = L−111 A21 (4)

A13 = L11U13 U13 = L−111 A13 (5)

c© 2013–15 Ned Nedialkov 12/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I Then we write

A22 = L21U12 + L22U22 A′22 = A22 − L21U12 = L22U22 (6)

compute L22 and U22 (7)

A23 = L21U13 + L22U23 A′23 = A23 − L21U13 = L22U23 (8)

U23 = L−122 A′23 (9)

A32 = L31U12 + L32U22 A′32 = A32 − L31U12 = L32U22 (10)

L32 = A′32U−122 (11)

A33 = L31U13 + L32U23 + L33U33 A′33 = A33 − L31U13 (12)

A′′33 = A′33 − L32U23 = L33U33 (13)

compute L33 and U22 (14)

c© 2013–15 Ned Nedialkov 13/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

I Using the right column in (1–14), we can decompose thecomputation in the following tasks

1. A11 = L11U11 (compute LU factorization)2. L21 = A21U−1

11

3. L31 = A31U−111

4. U12 = L−111 A21

5. U13 = L−111 A13

6. A′22 = A22 − L21U12

7. A′32 = A32 − L31U12

8. A′23 = A23 − L21U13

9. A′33 = A33 − L31U13

10. A′22 = L22U22 (compute LU factorization)11. L32 = A′32U−1

22

12. U23 = L−122 A′23

13. A′′33 = A′33 − L32U23

14. A′′33 = L33U33 (compute LU factorization)

c© 2013–15 Ned Nedialkov 14/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

1

2 345

6 78 9

10

11

13

12

14

Figure: Task-dependency graph

c© 2013–15 Ned Nedialkov 15/16

Foster’s Methodology Tasks Degrees of concurrency Critical path Examples

1

2 345

6 78 9

10

11

13

12

14

P0

P0 P1 P2 P3

P0 P1 P2 P3

P0

P0P1

P0

P0

Figure: Task distribution on 4 processes. Maximum speedup is 2

c© 2013–15 Ned Nedialkov 16/16