SUNet the Stanford University Network

Presentation

for the Internet2

Campus Networking Workshop

A brief history of SUNet

• Originally a research project of EE, CS and Medical grad students

• Networking staff assembled in 1983

• Centrally funded since 1984

• Cable plant project in 1985

• Responsible for operations since 1986

Backbone technology evolution

• 3Mb ethernet from Xerox PARC ~ 1979

• Large bridged ethernet - DEC bridges 1986

• Campuswide FDDI ring 1990

• Multiple FDDI rings from Cisco 7513 1995

• Initial fiber install in 1985 - 12 MM

• Latest trunks: 96 MM/48 SM

In-building wiring

• Complete spectrum down to 3-pair RJ-11

• New buildings get at least 2 Cat-5 network jacks along with 2 Cat-3 phone wires

• Centrally funded wiring improvement project - recently approved - requires matching departmental investment in activation

SUNet design parameters

• TCP/IP only supported backbone protocol

• Appletalk is tunneled with Fastpaths

• Medical Center routes IPX/Appletalk

• DECNET still on in several routers

• Networking supports to building entrance

• Local Network Administrators in buildings

• Network consultants assist LNAs

Network components

• 49 routers from Cisco

• 3 7513, 2 7507, 6 7000, 4 7206, 5 4700

• 7 2501, 1 2514, 1 3000

• and still 9 AGS+, 10 AGS, 1 CGS

• FDDI concentrators from DEC

• FastEthernet switches: Catalyst 5000/5500

• ATM switches: LS1010

SUNet statistics 970914

• From NetDB, the network database

• Generates DNS and BootP tables

• 442 subnets defined, 342 active

• 327 physical subnets

• 240 main campus, 87 Medical Center

• 33866 hosts, 35445 addresses

• 24749 campus hosts, 9117 Medical Center

CoreCisco 7513

Student

DMZ

AFS

ACE

Engineering

GeneralAdmin servers

Med

MedNet

ACE/AFS servers

BBN Planet(and beyond….)SUNetUCnet

ACEworkstations

GSB GSB

SUNet Backboneas of early 1997

NCS7000

EncinaAGS+

Quad7206

Press7000

Pine7507

JenkinsAGS+

FDDI

FDDI

FDDI

ACE Servers

AFS/DCEServers

BBN Planet

CORE

7513

DMZ

CISAGS+

Gates 17000

Gates 27000

FDDI

Res4700

Res-West4700

Res-South4700

CERASAGS+

Res-CentralAGS+

Sweet7507

FDDI

DMZ4700

MedCenter

VarianAGS+

GSB-DMZ4700

GSB-DMZ4700

InfoSciAGS+

MedNetAGS+

Med7206

SUNet7000

Next generation requirements

• Migration from Mainframe to distributed computing model for Admin systems

• Choice of DCE as standard platform

• Higher bandwidth for image transfer, etc

• Increased reliability as well as capacity

Next Generation choices

• ATM seemed likely even 18 months ago

• New routers were ordered with ATM

• New Computer Science building provided early implementation experience for both LANE and FastEthernet VLAN trunking

• Sought advice from Cisco Consulting Engineers

Proposed ATM backbones

ATMswitch

ATMswitch

ATMswitch

ATMswitch

ATMswitch

ATMswitch

ATMswitch

ATMswitch

Concerns with ATM design

• Redundancy vs. Interface expense

• Total throughput

• Staff education on a new technology

N-dimensional mesh network

• Proposed by Cisco’s Roger Beeman

• Designed to have well defined expansion

• Multiple paths provide aggregate capacity

• Significant redundancy makes individual components less critical

• Networks on corners

• 2-way routers on edges or 4-way routers on faces

4-way connected routers on faces

• 3-D (cube) : 8 networks, 6 routers

• 4-D (hypercube): 16 networks, 24 routers

• 5-D: 32 networks, 80 routers

The SUNet choice - a cube

• Attracted by the combination of redundancy and expandable capacity

• Have purchased 6 7513s with 4-VIP2s with 100FX port adapter each

• Have purchased 4 Catalyst 5000s with 2 12x100FX boards, each will be two corners (using VLANs)

SUNet cube backbone layout and addressing

0

1

642

3 75

1

6

5

4

3

2

Sweet Press Forsythe Pine

.1

.17

.33

.33

.49

.49

.3

.3

.19

.67

.83

.2

.34 .66.2

.114

.114

.18

.18

.50

.98

.98

.83.82

.82

.35

.35.99

.51

.51 .115

.115

.113

.97

.97

.81

.81

.65

.65

Connections to the cube

• User routers connected to at least two corner networks

• Server nets directly connected to two backbone routers using HSRP <or>

• Server router directly connected to two opposite face backbone routers

• Three “hotspots” - ACE servers, Admin servers and Off-campus/Internet

User router choices

• Planned to serve multiple networks per router interface using VLANs and Catalysts

• Now planning to use router module (RSM) in Catalyst 5500 and feed Catalyst 5000s

• Have ordered several to replace remaining AGS+ routers

• Will use Catalyst 3200 replacing AGS “departmental routers”

SUNet Cube Backbone

R

R

RFE

FEFE

R

RUser routers

RFEFE

FEFER

FE

R Router to BBN

FDDI

CalREN-2Gigapop

User routersCommercial

InternetR

Legacy routers

R

R

R

Organization

• ITSS - Information Technology Systems and Services

• CCS - Computer and Communications Services

• Networking Systems - centrally funded

• Production Services - Operations/Systems

• SUNet Operations

Responsibilities

• Backbone routers

• End-to-end TCP/IP protocol

• Network to the building entrance

• Run network servers (DNS, DHCP, etc)

• Support network consultants/LNAs

Support profile

• Onsite staff - 7am - 11pm Monday-Friday

• Weekend - 8 hours/day

• On call support 7 by 24

• Off hour trouble calls taken by Computer Operations staff who page on call person

Staffing level

• Operations has 5 Network Specialists and 3 Systems Programmers

• Systems has 2 SysAdmins, 2 generalists

• One network specialist/one sysadmin work the late shift (3pm-11pm)

Multicast

• Currently using PIM dense mode

• Works in 11.0 AGS routers (if no loops)

• Plan to deploy in new backbone

• SCPD (SITN) Engineering classes

• Library - supplement class videotapes

• Conferencing with remote sites

Network monitoring

• DEC MSU / Remedy Health Profiler

• IBM Netview 6000

• Perl scripts

• Ethermeters / RMON

Traffic/usage data

• Have been using Cisco’s Netflow Export - looks very promising

• Other products being examined

Challenge of Internet2

• More of a co-operative than NSFNET

• Trouble ticket generation/responsibility

• Service level commitment between members?

• Chance to use the technology!

Gates Building Network

Campus Backbone FDDI 100

Cisco 7513 RouterCisco 5000100MB Vlan BBone

Cisco 5000

Cisco LS1010155MB ATM BBone

To 21 Catalyst 5000

To 7 Catalyst 5000

... ... ... ... ... ...

10ea.10MB ether ports connected to C5000

4ea. 100MB

connected to C5000

OC3 ATM

Cisco 5000 Cisco 5000 …. ….

21ea. Cisco Catalyst 50001000 10BaseT ports, 80 100BaseT ports, 20VLans

. . . . . . . .. .

9/8/97 CSDCF

Gates VLAN ConnectionsG Cat-0-

4

G Cat-0-2

G Cat-0-1

G Cat-0-3

G Cat-1-1

G Cat-1-4

G Cat-1-2

G Cat-1-3

G Cat-2-1

G Cat-2-4

G Cat-2-2

G Cat-2-3

G Cat-3-1

G Cat-3-4

G Cat-3-2

G Cat-3-3

G Cat-4-1

G Cat-4-4

G Cat-4-2

G Cat-4-3

G Cat-5-1

Cisco 7513 Router

VLAN 1: 171.64.70.10 - All GCATsVLAN 41: 171.64.64.10VLAN 3: 171.64.65.10VLAN 4: 171.64.67.10VLAN 5/45: 171.64.68.10VLAN 6: 171.64.71.10

Cisco LS1010

9/8/97 CSDCF

G Cat-0-4

G Cat-0-2

G Cat-0-1

G Cat-0-3

G Cat-1-1

G Cat-1-4

G Cat-1-2

G Cat-1-3

G Cat-2-1

G Cat-2-4

G Cat-2-2

G Cat-2-3

G Cat-3-1

G Cat-3-4

G Cat-3-2

G Cat-3-3

G Cat-4-1

G Cat-4-4

G Cat-4-2

G Cat-4-3

G Cat-5-1

Cisco 7513 Router

VLAN 7: 171.64.72.10 171.64.73.10

VLAN 8: 171.64.75.10VLAN 9: 171.64.76.10VLAN 11: 171.64.66.10 100MBVLAN 12: 171.64.74.10 100MBVLAN 13 171.64.77.10 100MB

Cisco LS1010

Gates VLAN Connections9/8/97

G Cat-0-4

G Cat-0-2

G Cat-0-1

G Cat-0-3

G Cat-1-1

G Cat-1-4

G Cat-1-2

G Cat-1-3

G Cat-2-1

G Cat-2-4

G Cat-2-2

G Cat-2-3

G Cat-3-1

G Cat-3-4

G Cat-3-2

G Cat-3-3

G Cat-4-1

G Cat-4-4

G Cat-4-2

G Cat-4-3

G Cat-5-1

Cisco 7513 Router

VLAN 13: 171.64.77.10VLAN 14/16: 171.64.79.10 100MBVLAN 15: 171.65.XX.XXVLAN 17: 171.64.67.10VLAN 19: 171.64.70.128ELAN 30:: 171.64.70.128ELAN 45: 171.64.68.10

Cisco LS1010

9/8/97

Gates VLAN Connections

Repeater from Med Center VLAN 15

VLAN 17 Mosquito Net(Baker)

Vlan 16 Firewall Net(Cheriton)

DHCP Public VLAN VLAN 19

Experimental ATM ELAN 30