December 2007

Neil FrancisNetwork Administrator

SWERN

University of Bath

JANET Optical Event

SWERN rebuilt using Ethernet50 VLANS and 3 Telcos

2

SWERN2 - Outline

SWERN

One of the RNOsMain University connections

Bath, Bath Spa, Bristol, UWE, Exeter, Plymouth, Falmouth

Approx. 40 Further Education InstitutionsA dozen or so other institutionsAbout 450,000 users

3

Swern2 - Some Background

SWERN is in the South West of EnglandMerger of SWAN and BWEMAN

Essentially 2 networks connected togetherPiecemeal build – varied active equipmentSome resilience (always a worry)

Mixture of ATM, POS, Ethernet etcLimited HR

Two part time engineers, network managers, some Ops support etc

Geographical issues – long and thin, rural(ish)13 POPs – some remote

The usual stuff.....

5

SWERN2 - Opportunities

For SWERN 2 we had an opportunity to tidy things upThe usual design factors

Homogeneous, Resilient, Cost Effective?, Engineering, HR.

The South West is long and thinWe had POPs in the large conurbations

CheltenhamSwindonBathBristolTauntonExeterPlymouth

8

SWERN2 - Options

OptionsDark fibre, managed solutions etcAll considered but not only cost ruled these out.

Long distance ethernet circuits becoming more comfortable for usConcerns were things like

ReliabilityResilienceScalability (bandwidth)QOSSecurityUnderstanding exactly what we were getting

9

SWERN2 - Ethernet

Ethernet – but not quiteTypes of connection:

Point-point, point-multipoint, multipoint-multipoint, etc.Our equipment:

RoutersSwitches

Architecture: VPWS - Virtual Private WiresVPLS - Virtual Private LANsEthernet over Optical

etc...

Service Characteristics: VLAN MultiplexingVLAN TransparencyL2 PDU Transparency

10

SWERN2 - Ethernet

Telcos had solutions availableWe already had some metro ethernet cctsTail circuits only

Ethernet on the backbone meant...Bandwidth - fast, cheap, tunableStandard simple equipmentStandard circuit presentations (fibre and copper)Layer 2 links – easier to manageJust use point to point vlansWe do layer 3

It looked OK

11

SWERN2 – Ethernet Backbone

Connecting these all togetherConnect to enable loops

Two POPs in each cityConnect these together

Fairly close together so relatively easy

Connect each POP back to an RNEPUse Metro Ethernet circuits

Connect the RNEPs togetherFairly easy

End up with something like this....

12

SWERN2 – Ethernet Circuits

Essentially 6 loops here

Connect to standard ethernet equipment...

13

SWERN 2 – Standard Equipment

Cisco 7609 OSR RoutersSup. Engine 720/1Gbyte RAM Located at RNEPs

Cisco 7206VXR RoutersNPE-G1 Processors

Cisco Catalyst 494848 Port 10/100/1000

Ethernet Switch

14

SWERN 2 – Loops & VLANS

SJ5

SJ5

RNEPsPOPsEnd Sites

Allow only specific VLANS across the trunks

Trunks

15

SWERN2 - VLANS

16

SWERN2 - Tender

Tender issued late 2005Solutions based on mixture of

WDMSDH/SONET/EPLLES/WESMPLS

But all presented to us as ethernetCarrier EthernetMetro EthernetEvolved EthernetNational Ethernet

etc......

17

SWERN2 - Bandwidths

All bandwidths uncontended

18

SWERN2 - Engineering

With 6 loops 8 VLANS per loopPlus some test VLANS

Over 50 VLANS – all terminate on our RNEP routersAll traffic flows through these boxes

All point to point VLANS

Plus a management VLAN

We do the layer 3 stuff (OSPF, BGP)

We still have to manage, monitor, administer, troubleshoot etc.

19

SWERN 2 - Routing

20

SWERN 2 – Routing and Resilience

OSPFSingle small backbone (Area 0)Stub areas from the POP areasTimers tuned low

Also looked at BFD for fast link failure detectionBest case scenario – as good as POS!Worked between our POP routers and RNEPsNeeded 12.4.x IOS on our 7200s

Experimented with timers but were finding even a flash disk insertion was causing BFD to baulk

We currently rely on fast converging OSPF

21

SWERN2 - Observations

We know we have a mixture of the following:

Simple LES/WES circuits

DWDM/SDH/EPL circuits

EoMPLS circuits

We run point to point VLANS across all this technology and it works

Seemed too easy in many waysJust 2-3 people doing all this

22

SWERN2 - Observations

Management VLANSpanning tree should be turned onHave had more than one occurrence of loopingSomewhere - BPDUs are not being passedTaking longer to fix than it should

Witnessed CDP neighbor inconsistenciesTracked to provider switches being at different OS levels. A bug here but related to above.

Lost our .ad/.ah port configuration3 VLANS downProvider insistent all was workingFixed at 3rd level support – too long

23

SWERN2 – EPL Latency

24

SWERN2 – MPLS Latency

25

SWERN2 – Next stages

10 Gbps links across BristolConnecting RNEPS

Dark Fibre and 10GE over Optical

Still presented as EthernetTerminate on 10GE switchesProbably do some routing on same devicesDirect connection to RNEP routers10G POS to Janet Core

Everything else the same

26

SWERN2 - Possibilities

The concept of SWERN2 is all very easyIdentical POPs and equipmentIdentical circuit connections

Ideally we would want less POPs

Possibility to use Telco infrastructure to backhaul everything to our RNEPS

Tail circuits to institutions not an issue any moreResilienceQoS

28

SWERN3 - Maybe

Can't keep up with the service offeringsEthernet over SONETMPLS/VPLST-MPLS (ITU G.8110)802.1ad, 802.1ah, 812.1Qay802.3ah – Ethernet in the first mile

Etc...

Management802.1ag – Connectivity Fault Management802.1ag – End to End OAM/CFM

Management - is this the important bit?

29

SWERN2 - Observations

30

Any Comments

What we had

What we did

How we did it

Would anyone else do it this way?

What do other people do?

Observations