Lecture 6: Deadlocks - IItusharkrishna.ece.gatech.edu/.../sites/175/2016/10/L06-Deadlocks-II.pdf ·...

Lecture 6:Deadlocks - IITushar Krishna

Assistant ProfessorSchool of Electrical and Computer EngineeringGeorgia Institute of Technology

[email protected]

ECE 8823 A / CS 8803 - ICNInterconnection NetworksSpring 2017http://tusharkrishna.ece.gatech.edu/teaching/icn_s17/

Recap: Resource Dependence

January 30, 2017ICN | Spring 2017 | L06: Deadlocks-II © Tushar Krishna, School of ECE, Georgia Tech

2

A B

u v w x

C D

Hold (i..e, wait for release)Wait to acquire

Agents

Resources

Resource A is dependent on resource B if it is possible for A to be held-byan agent X and it is also possible for X to wait-for B

u v w x

Resource Dependence Graph

To avoid deadlocks, resource dependence graph should be acyclic

Recap: Network Deadlock§ Agent = packet or flit, Resource = buffer

§ Packet A holds buffer u (in 1) and wants buffer v (in 2)§ Packet B holds buffer v (in 2) and wants buffer w (in 3)§ Packet C holds buffer w (in 3) and wants buffer x (in 0)§ Packet D holds buffer x (in 0) and wants buffer u (in 1)


3

0 1

23

A

BC

Du

vw

x

We will use the following convention: channel == buffer at end of channel

Channel 01 blocked => Buffer at 1 not free

Channel 01 depends on Channel 12 => Buffer at 1 depends on Buffer at 2

Deadlock Avoidance¡Eliminate cycles in Resource Dependency Graph¡ Resource Ordering¡ Enforce a partial/total order on the resources, and insist that

an agent acquire the resources in ascending order¡ Deadlock avoided since a cycle must contain at least one

agent holding a higher numbered resource waiting for a lower-numbered resource which is not allowed by the ordering allocation

¡ Implementation¡ Restrict certain routes so that a higher numbered resource

cannot wait for a lower numbered resource¡ Partition the buffers at each node such that they belong to

different resource classes. A packet only any route can only acquire buffers in ascending order of resource class


4

Example: Channel Dependency Graph for Mesh


5

F

A B

E D

C

Acyclic CDG

Disable certain edges

FEED

DC CBBE EF FA

AFBA

EBDECDBCAB

Cyclic CDG

FEED

DC CBBE EF FA

AFBA

EBDECDBCAB

This is the West-first turn model!

Acyclic CDG for a Ring


6

F

A B

E D

C

FE

ED DC

CB

BAAF

CD

DE EF

FA

ABBC

Option 3

Problem?

Route from E to C disabled(E to D) and (D to C) allowed

Route from B to D disabled

Acceptable CDGE to C no longer minimal

CDG

Acyclic CDG for a Large Ring


7

H

C D

G F

E

I

B

J

A

G, H, I have to take non-minimal paths to reach E!

D, C, B have to take non-minimal paths to reach F

Problem?

Suppose two channels


8

F

A B

E D

C

FE0

ED0 DC0

CB0

BA0AF0

CD0

DE0 EF0

FA0

AB0BC0

ED1 DC1

FE1

AF1 BA1

CB1

DE1 EF1

CD1

AF1 BA1

FA1

Dateline

DatelineDateline

Need not be physical channels


9

FE0

ED0 DC0

CB0

BA0AF0

CD0

DE0 EF0

FA0

AB0BC0

ED1 DC1

FE1

AF1 BA1

CB1

DE1 EF1

CD1

AF1 BA1

FA1

Dateline

DatelineDateline

Need at least 2 classes of buffers - called“Virtual Channels”Start in VC in Class0After Dateline, jump to VC in Class1

F

A B

E D

C

Deadlock Avoidance¡Eliminate cycles in Resource Dependency Graph¡ Resource Ordering¡ Enforce a partial/total order on the resources, and insist that

an agent acquire the resources in ascending order¡ Deadlock avoided since a cycle must contain at least one

agent holding a higher numbered resource waiting for a lower-numbered resource which is not allowed by the ordering allocation

¡ Implementation¡ Restrict certain routes so that a higher numbered resource

cannot wait for a lower numbered resource¡ Partition the buffers at each node such that they belong to

different resource classes. A packet only any route can only acquire buffers in ascending order of resource class


10

Using VCs for Deadlock Avoidance¡Ring¡Use VC from class 0 before dateline¡Use VC from class 1 after dateline

¡Fully-Oblivious (e.g., O1turn)¡Use VC 0 for XY, VC 1 for YX

¡Fully-Adaptive Routing (no turns restricted)¡Use VC from class 0 before turning¡Use VC from class 1 after turning

¡Valiant’s Routing Algorithm¡DOR over VC in class 0 from source till intermediate¡DOR over VC in class 1 from intermediate to destination


11

VC utilization


12

F

A B

E D

C

Problem?Ring with Dateline

Mesh with O1Turn

VC from Class 1 never used!

Packet on Ring never crosses datelinePacket on Mesh does not make any turns

Class 0

Class 1

Solution: Overlapping Resource Classes

Class 0Class 1

As long at least one buffer per class, can guarantee deadlock freedom

Deadlock Avoidance¡So far, we said deadlock is avoided if cycles

eliminated in Channel Dependence Graph¡ Remove cycles via turn restriction¡ Convert cyclic CDG into a spiral using VCs¡ Called extended CDG


13

FE0

ED0 DC0

CB0

BA0AF0

CD0

DE0 EF0

FA0

AB0BC0

ED1 DC1

FE1

AF1 BA1

CB1

DE1 EF1

CD1

AF1 BA1

FA1

Deadlock Avoidance¡So far, we said deadlock is avoided if cycles

eliminated in Channel Dependence Graph¡ Remove cycles via turn restriction¡ Convert cyclic CDG into a spiral using VCs¡ Called extended CDG

¡However, it is possible for a (extended) CDG to have cycles and still be deadlock-free (Duato*, 1993)¡ As long as the cycle connects to some sub-graph within

the (extended) CDG that is acyclic¡ Known as the escape path or escape VC


14

*José Duato. A new theory of deadlock-free adaptive routing in wormhole networks.IEEE Transactions on Parallel and Distributed Systems, 4(12):1320–1331, December 1993.

CDG for Escape VCs


15

FE0

ED0 DC0

CB0

BA0AF0

ED1 DC1

FE1

AF1 BA1

CB1

F

A B

E D

C

Escape VC

Acyclic Escape VC

Why escape VCs work¡ Intuitively, at least one packet in the cycle has an

option to take an acyclic route¡ Packets should not wait on any specific channel¡ If allocation is fair, escape VCs guaranteed to show up

eventually

¡Use of escape channels by a message is not unidirectional¡ If a message enters the escape network it can move

back to the adaptive network, and vice versa, if minimal* routes¡ *for non-minimal routes, message has to continue on

escape VC once it gets in, without going back to the adaptive VCs


16

Example¡Consider a 2D Mesh with 8 VCs and minimal

routing¡VC 1-7 can use any arbitrary minimal routing¡ Cyclic CDG

¡VC 0 (escape VC) is restricted to DOR (provides escape path)¡ Acyclic CDG

¡As long as a packet can allocate all VCs fairly, there will always be an escape path available in case the network deadlocks


17

Example


18

Escape VC

West-first

Regular VCs

Deadlock-free escape path

Rules for getting in/out of escape VCs¡The escape VC should

always makes forward progress!¡A flit that is going NW or

SW should never enter a router from the S or N port in escape VC, else SàW or NàW turn is inevitable¡How to guarantee this?¡ When selecting VC at

previous router¡ Lab 3!


19

VC 1 (escape VC)West-first

VC 0 (regular VC)

X?

Ejection not shown

Deadlock Avoidance¡Routing¡ Turn Model in a Mesh¡ Acyclic CDG

¡Buffer Assignment¡ Acquire VCs in ascending order¡ Acyclic extended CDG

¡ Escape VCs¡ Cyclic CDG + Acyclic sub-graph

¡Flow Control¡ Control traffic flow such that deadlock does not occur¡ Cyclic CDG + injection control to avoid dependence cycle


20

Bubble Flow Control


21

F

A B

E D

C

Ring Traversal Rule: traverse if one bubble free

V. Puente et al. The adaptive bubble router. Journal of Parallel and Distributed Computing, 2001.

Bubble Flow Control


22


F

A B

E D

CV. Puente et al. The adaptive bubble router. Journal of Parallel and Distributed Computing, 2001.

Should it inject?BFC Injection Rule:only inject if 2 bubbles free.

Bubble Flow Control


23


F

A B

E D

C

BFC Injection Rule:only inject if 2 bubbles free.


Problem?

Bubble Flow Control


24


BFC Injection Rule:only inject if 2 bubbles free.


F

A B

E D

C

Not allowed to inject!Even though no deadlock

Critical Bubble Flow Control


25


CBFC Injection Rule:only inject if not critical bubble.

F

A B

E D

C

Allowed to inject!

L. Chen et al., “Critical Bubble Scheme: An Efficient Implementationof Globally Aware Network Flow Control,” IPDPS 2011

Critical Bubble

Critical Bubble Flow Control


26


CBFC Injection Rule:only inject if not critical bubble.

F

A B

E D

CL. Chen et al., “Critical Bubble Scheme: An Efficient Implementationof Globally Aware Network Flow Control,” IPDPS 2011

Critical Bubble

How does critical bubble move?If flit moves into critical bubble, its own buffer becomes new critical bubble

Dealing with Deadlocks¡Avoidance¡Guarantee that the network will never deadlock¡Almost all modern networks use deadlock avoidance

¡Recovery¡Detect deadlock and correct


27

Deadlock Recovery¡When might this make sense?¡ Cannot accept performance degradation required to avoid

deadlocks¡ Average-case performance more important

¡ Two phases¡ Detection: ¡ E.g., timeouts attached with each resource¡ Can lead to false positives

¡ Recovery:¡ Regressive – remove packets/connections that are deadlocked¡ E.g., drop packets after timeout

¡ Progressive – recover without removing packets/connections¡ E.g. shared escape buffer to drain deadlocked packets


28

K. V. Anjan and T. M. Pinkiston, “An efficient fully adaptive deadlock recovery scheme: DISHA”, ISCA 1995

Another kind of deadlock: Protocol Deadlock


29

Cache 0

Directoryreqreqreqreq

req

req

req

resp

resp

req

req

out

out

in

in

req

Cache / Directory can process a request only if there is space in its output queue to send a response

Need separate Virtual Channels* for requests and responses(called Virtual Networks)

Responses should always be drained (“consumption assumption”)

Deadlock, even though network is deadlock-free

Date post:	28-May-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

Lecture 6: Deadlocks - IItusharkrishna.ece.gatech.edu/.../sites/175/2016/10/L06-Deadlocks-II.pdf ·...

Documents