Gaussian Interconnections for On-Chip Networks

Ramón Beivide and Enrique VallejoUniversity of Cantabria, Spainenrique@atc.unican.es

R. Beivide, E. Vallejo 2Microgrid Workshop

Index Introduction: Why a Topology? Dense Gaussian Networks and other topologies Different layouts Routing:

ideas: Adaptive routing, deadlock avoidance, fault tolerance Unicast routing Broadcast Routing

Perfect placement of resources Expansibility:

Increasing number of nodes in a Gaussian network Hierarchical Gaussian networks

Some ideas about cache coherence

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 3Microgrid Workshop

Introduction

Future trends: many PE on a chip Possible interconections: bus, MIN, direct

network Bus-based interconnections do not scale – they

do not provide a sufficient bandwith when there are many PEs. MIN hard to implement in a chip.

Direct networks with a given topology: The way to connect different routers in the chip

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 4Microgrid Workshop

Mesh Network

Number of Nodes N:

N = b x b = b2

Diameter k:

k = (b-1) + (b-1) = 2b-2

i 1+i 2+i -1+i -2+i

2i 1+2i 2+2i -1+2i -2+2i

0 1 2 -1 -2

-i 1-i 2-i -1-i -2-i

-2i 1-2i 2-2i -1-2i -2-2i

b

b

Nb

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 5Microgrid Workshop

Number of Nodes N:

N = (b+1)2 + b2

Diameter k= b + b = 2b 0 1 2 -1 -2 3 -3

i 1+i 2+i -1+i -2+i

2i 1+2i -1+2i

-i 1-i 2-i -1-i -2-i

-2i 1-2i -1-2i

3i

-3i

2 b+1

Diamond Network

2112

Nb

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 6Microgrid Workshop

Torus Network

Number of Nodes N:

N = b x b = b2

Diameter k = b -1

Nb i 1+i 2+i -1+i -2+i

2i 1+2i 2+2i -1+2i -2+2i

0 1 2 -1 -2

-i 1-i 2-i -1-i -2-i

-2i 1-2i 2-2i -1-2i -2-2i

b

b

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 7Microgrid Workshop

0 1 2 -1 -2 3 -3

i 1+i 2+i 2+i 2+i

2i 1+2i 2+i

-i 2+i 2+i 2+i 2+i

-2i 2+i 2+i

3i

-3i

2b+1

Number of Nodes N:

N = (b+1)2 + b2

Diameter k = b

2112

Nb

Dense Gaussian Network

• Same # links as torus, withperipheral links.• Lower mean distance andDiameter.

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 8Microgrid Workshop

Topological properties comparative

Topology Nodes Diameter Aprox. Diam.

Average Distance

Aprox. Aver. Dist

2-D Mesh

2-D Torus

Dense Gaussian

2WN

22 W N 12

12

2

NWWW

2N

122 2 kkN k

13121

2

Nkk

32N

2N

32 N

N222 W3

2W2WN

Lower average distance and diameter

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 9Microgrid Workshop

Area comparativeGaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 10Microgrid Workshop

3i -2-i -1-i 3 2 -i 1

2i 1+2i 1-2i -1+2i 2-i -2i 1-i

i 1+i 2+i -1+i -2+i -3i 1-2i

0 -1 -2 -3

Different Layouts

2314

20191718

0

232422

21

5

131514

78 6

101211 9

16

Different layouts for the same network:•Mesh-like layout•Without peripheral links, bounded link length

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 11Microgrid Workshop

Routing ideas Adaptive routing: in-fligh packets can choose their

(minimal) path from info in the Routing Record (jumps in each direction), depending on congestion or other parameters.

Deadlock avoidance: Bubble routing proposed as a cost-effective deadlock avoidance mechanism (used in IBM’s Blue Gene). Only 2 virtual channels needed per link.

Fault-tolerant routing: Inmunet proposed as a fast, efficient mechanism to detect link failures and restore network performance.

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 12Microgrid Workshop

Unicast Routing

0 1 2 -1 -2 3 -3

1+i 2+i -1+i -2+i

1+2i -1+2i

-i 1-i 2-i -2-i

-2i 1-2i -1-2i

3i

-3i

Route from a to b:Routing record generatedFrom the difference: dest-source (x, y)

-1-i

i

Example:i – (-1-i) = 1+2i (x=1, y=2)1 jump to the right, 2 jumps up Movement from sourcenode to the origin (node 0)generates routing record.

Example 2: The movement makesthe arrow fall outside the originalnetwork Peripheral links used

Translations from surroundingreplicas of the network areconsidered, to obtain an optimal RR

2i

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 13Microgrid Workshop

P 1 2 -1 -2 3 -3

i 1+i 2+i -1+i -2+i

2i -1+2i

-i 1-i 2-i -1-i -2-i

-2i 1-2i -1-2i

3i

-3i

1+2i

NW NE

SW SE

Broadcast Routing

• Triangle-based broadcast• Minimum number of steps• The same for any node (due to peripheral links)• Balanced use of resources• Simple routing based on labels (see abstract)

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 14Microgrid Workshop

Perfect placement of resources

Resource distribution overthe network.

All nodes have resources within a given distance (example: distance 1)

Resource example: I/O portsProcessing elementsMemory banks...

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 15Microgrid Workshop

Expansibility: Increasing # nodes

Increasing Gaussian network: Network can be expanded with the number of nodes necessary to increase diameter in 1 unit: 4k +4.

Alternatively, hierarchical Gaussian networks have been proposed, joining several Gaussian networks with another gaussian pattern. Useful for CMPs sceneries, for example (different latency, link length, etc. in each hierarchical level): Lower level: interconnection between different cores inside a

chip. Fast and reliable, with low latency Higher level: interconnection between different chips. Slower

and with higher latency.

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 16Microgrid Workshop

Expansibility: Increasing # nodes

Gaussian Interconnections for On-chip Networks

Lower level (on-chip) with adense Gaussian pattern.Higher level, with the samepattern.

Central routers will have 8 links:4 internal links4 external links

Route from one node to another:1) Route to the central router of the same gaussian2) Route in the higher level to the desired gaussian.3) Route from the central router of the dest. Gaussian, to the destiny node.

R. Beivide, E. Vallejo 17Microgrid Workshop

Cache coherence in Gaussian networks Recent proposals based in broadcast, such as TokenB

(M. Hill) can beneficiate from a Gaussian interconnection:

Broadcast block requests (latency optimized with Gaussian interconection).

Unicast response with grants (Tokens) to use memory blocks, between different nodes and main memory.

There is no need to maintain a directory for coherence. Broadcast network can work as a bus with a snoopy-like

protocol.

Gaussian Interconnections for On-chip Networks

R. Beivide, E. Vallejo 18Microgrid Workshop

R. Beivide, E. Vallejo 19Microgrid Workshop

Additional comments (not presented) Dense Gaussian Networks are isomorphic to

Dense Midimew Networks. However, these two topologies are not isomorphic in the general case (not dense). In this work, related to Dense Gaussian networks, properties studied for both Gaussian and Midimew topologies are presented.

References in the next slide will be thus referred to both Midimew and Gaussian networks

R. Beivide, E. Vallejo 20Microgrid Workshop

Commented References (I) Midimew networks were first introduced in:

R. Beivide, E. Herrada, J.L. Balcázar, Agustín Arruabarrena, “Optimal Distance Networks of Low Degree for Parallel Computers”. IEEE Transactions on Computers, Vol. 40, No 10, Oct 1991, pp. 1109-1124. This paper introduces properties, analysis and some rectangular (mesh-like) layouts. Unicast routing is also proposed, but based on labeling nodes with integer labels (instead of Gaussian numbers).

Bounded link-length layouts were introduced in:E. Vallejo, R. Beivide y C. Martínez, “Practicable Layouts for Optimal Circulant Graphs”. Proceedings of the “13th Euromicro Conference on Parallel, Distributed and Network-based Processing”, Lugano, Switzerland, Feb. 2005. A previous work on Midimew folding, which obtained a worse result (maximum link length 4) is the following one:Francis C. M. Lau, Guihai Chen, “Optimal Layouts of Midimew Networks”. IEEE Transactions on Parallel and Distributed Systems, Vol 7, No 9, pp 954-961

R. Beivide, E. Vallejo 21Microgrid Workshop

Commented References (II) Gaussian Networks will be introduced in:

C. Martínez, R. Beivide, J. Gutierrez and E. Gabidulin. "On the Perfect t-Dominating Set Problem in Circulant Graphs and Codes over Gaussian Integers". Accepted for presentation at ISIT’05, September, Australia.This paper also deals with perfect resource placement.

Broadcasting in Dense Gaussian Networks will be introduced in:R. Beivide, C. Martínez, E. Vallejo, J. Gutierrez, C. Izu, “Gaussian Interconnection Networks”. Accepted for presentation at the Spanish Parallelism Conferences, Sept. 05, Granada, Spain.This paper also introduces unicast routing in terms of the Gaussian numbers (instead of integer labels)

Hierarchical Gaussian Networks will be introduced in:Miquel Moretó, Carmen Martínez, Enrique Vallejo, Ramón Beivide, Mateo Valero, “Hierarchical Topologies for Large-scale Two-level Networks”, Accepted for presentation at the Spanish Parallelism Conferences, Sept. 05, Granada, Spain.

R. Beivide, E. Vallejo 22Microgrid Workshop

Commented References (III) Bubble routing is described in

V. Puente, C. Izu, R. Beivide, J.A. Gregorio, F. Vallejo and J.M. Prellezo, “The Adaptative Bubble Router”, Journal of Parallel and Distributed Computing. Vol 61 - nº 9, September 2001

Inmunnet was introduced inV. Puente, J.A. Gregorio, F. Vallejo and R. Beivide. "Immunet: A Cheap and Robust Fault-Tolerant Packet Routing Mechanism". 31th Annual International Symposium on Computer Architecture (ISCA-31), pp. 198-209, 2004.

Token Coherence was presented in:M. M. K. Martin, M. D. Hill, and D. A. Wood. "Token Coherence: Decoupling Performance and Correctness". 30th Annual International Symposium on Computer Architecture (ISCA-30), pp. 182-193, 2003.