System Area Network

Abhiram Shandilya12/06/01

Overview

Introduction to System Area Networks SAN Design and Examples SAN Applications

Introduction

What is a SAN? Overview of Cluster Computing

Advent of Clusters Advantages of Clusters Communication in Clusters

SAN design goals

What is a SAN?

Primarily designed for communication between nodes in a cluster.

Other SANs: Storage, Server and Small

A Cluster

Cluster Computing: Advent of….

Early parallel computers: MPPs Custom-built Expensive

Intel Paragon, CM-1, nCube etc Changes in technology:

Faster processors Faster networking technologies Cheaper Commodity off-the-shelf (COTS) hardware

A cheaper option: clusters

Cluster Computing: Advantages

Better price/performance ratios Better availability

Survive node failures with appropriate software support Impressive scaling

100s to 1000s of processors Easy and economical upgrading and technology

migration.

Cluster Computing: Communication

Parallel computation is communication intensive Dominant contributors for message latency

Large messages: Bandwidth Small messages: Software Overhead

Cluster communication: small messages Software overhead on traditional networks due to:

Multiple data copies through protocol layers OS interrupts and context switching.

Software overhead due to multiple copies

SAN Design Goals

Minimize latency Avoid operating system by providing user application

direct access to the network interface Avoid multiple copies of data

High Bandwidth

SAN Design and Examples

Key Issues General Architecture Example 1: Virtual Interface Architecture (VIA) Example 2: Memory Channel

Key Issues

Design of direct user access to network interface Multiplexing between processes for access to

network interface Providing security: avoiding inter-process

interference Managing limited communication resources Efficient programming interface

General Architecture

Communication model Virtual Interface Architecture (VIA) User-Level Network Interface (U-Net) Virtual Memory Mapped Communication (VMMC)

Network Interconnect (NI) Memory Channel 1 & 2 (DIGITAL) ServerNet-II (Compaq)

Virtual Interface Architecture (VIA):An Overview

Promoted by Compaq, Intel and Microsoft Defines an industry-standard architecture for

communication within clusters Increase scalability

Allows a single parallel application to run efficiently over dozens of clustered nodes

Speeds up exchange of data between distributed application modules running on different servers

VIA: Goals Protected communication Latencies like memory operation

Eliminate intermediate data copies Avoid traps into the OS Eliminate driver running in protected kernel mode Minimize number of instructions for send/receive Remove the reliance on interrupts for communication Keep architecture simple to allow software and hardware

implementations Three types of services provided:

Unreliable datagram service Reliable delivery service Reliable reception service

A Virtual Interface (VI) Creates illusion that a process has its own network

interface.UserProcess

Send

Que

ue

Rec

eive

Que

ue

CN

TL

VI

Network Interface

UserProcess

Send

Que

ue

Rec

eive

Que

ue

CN

TL

VIKernelAgent

Controland interrupts

Per VI controland synchronization

VIA: Architecture

Virtual Interface Represents a connection Linked list of descriptors

Send, remote-DMA/write and remote-DMA/read on send queue Receive on receive queue

Maintains data ordering for a connection using VI Asynchronous I/O between different VIs

Kernel agent Unless required by user, not involved in communication Initializes tables and structures for communication Provides memory protection Pins physical memory for each connection.

Necessary to avoid delay on receiver’s side due to address translation

Memory Channel 1: An Overview

Developed at DIGITAL as a dedicated network interconnect

Designed to improve communication performance in a cluster of at most 8 nodes Can support clusters as large as 8 nodes by 12 processors

Implements Virtual Shared Memory Supports any computer using industry-standard

peripheral-component interconnect (PCI) bus

Memory Channel 1: Goals

Minimize cluster communication latencies Maintain the ability to repair hardware while cluster is

running Strict message ordering Support full detection and recovery from communication

errors Connect to PCI bus for compatibility with all PCI-based

Alphas and Intel computers. Support broadcast, multicast and point-to-point

communication

Memory Channel 1: Architecture (1)Address Space Mapping

Memory-mapped circuit Connection between

virtual-address space on sending node and a page of physical memory on receiver’s node.

Write-only connection

Memory Channel 1: Architecture (2)Hardware Implementation

MCNetwork

I/O Bridge

Transmitpath

ReceivePath

Processor

Memory I/ODevice

I/O Bridge

Transmitpath

ReceivePath

Processor

Memory I/ODevice

PCIBus

SystemBus

SystemBus

PCIBus

Node 2

Node 1

MC Adaptor

MC Adaptor

SAN Applications

File Server Clusters Database Server Clusters Storage Area Networks (SAN) Web Servers Clusters Scientific Applications

Beowulf clusters

Conclusions

Introduction to SAN SAN Design and examples

VIA Memory Channel 1

SAN Applications