Operating Wide-Area Operating Wide-Area Ethernet NetworksEthernet Networks
Operating Wide-Area Operating Wide-Area Ethernet NetworksEthernet Networks
Matt DavyMatt DavyGlobal NOCGlobal NOCMatt DavyMatt DavyGlobal NOCGlobal NOC
OutlineOutlineOutlineOutline
Overview of Networks
Configuration
Troubleshooting/Monitoring
Overview of Networks
Configuration
Troubleshooting/Monitoring
Overview of NetworksOverview of NetworksOverview of NetworksOverview of Networks
NLR FrameNet
nationwide ethernet over dwdm
18 Cisco 6509 switches
10GbE backbone
p2p and multipoint vlans
dedicated and best effort
NLR FrameNet
nationwide ethernet over dwdm
18 Cisco 6509 switches
10GbE backbone
p2p and multipoint vlans
dedicated and best effort
Overview of NetworksOverview of NetworksOverview of NetworksOverview of Networks
I-Light
Indiana’s statewide higher ed network
statewide ethernet over dwdm
19 Cisco 6509 switches (layer2 & layer3)
10GbE backbone with p2p vlans
I-Light
Indiana’s statewide higher ed network
statewide ethernet over dwdm
19 Cisco 6509 switches (layer2 & layer3)
10GbE backbone with p2p vlans
Overview of NetworksOverview of NetworksOverview of NetworksOverview of Networks
MANLAN
ethernet exchange in new york city
Cisco 6513 switch
1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn
local and wide-area connections
MANLAN
ethernet exchange in new york city
Cisco 6513 switch
1GbE and 10GbE connections over dwdm, sonet, direct fiber - even one over mpls l2 vpn
local and wide-area connections
Overview of NetworksOverview of NetworksOverview of NetworksOverview of Networks
Indiana University campus network
large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses)
Cisco 6500’s and HP Procurve
very interesting stp design
~1,500 total switches
Indiana University campus network
large layer-2 infrastructure from edge into core (capable of plumbing vlans between buildings and even between campuses)
Cisco 6500’s and HP Procurve
very interesting stp design
~1,500 total switches
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
configuration of vlans
very manual and time intensive (manual = error prone)
need to automate this process
various control plane projects are one option, but could use something more lightweight
could use vtp ?
configuration of vlans
very manual and time intensive (manual = error prone)
need to automate this process
various control plane projects are one option, but could use something more lightweight
could use vtp ?
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
VLAN ID Assignment
big problem when interconnecting multiple layer2 domains
does Q-in-Q solve this ?
does vlan id translation solve this ?
VLAN ID Assignment
big problem when interconnecting multiple layer2 domains
does Q-in-Q solve this ?
does vlan id translation solve this ?
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
Q-in-Q
sounds good, but not flexible enough
want to map some .1q tags to outside vlan and want other .1q tags to get switched normally
customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in-Q.
also not implemented in all switches
Q-in-Q
sounds good, but not flexible enough
want to map some .1q tags to outside vlan and want other .1q tags to get switched normally
customer A wants to trunk vlans to customer B, but also wants vlans to customers C, D and E who don’t want Q-in-Q.
also not implemented in all switches
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
VLAN ID Translation
could help, but limitations in currently implementation
each port needs it’s own translation table
on 6500, translation table is shared across multiple ports
greatly confuses cross-domain troubleshooting
VLAN ID Translation
could help, but limitations in currently implementation
each port needs it’s own translation table
on 6500, translation table is shared across multiple ports
greatly confuses cross-domain troubleshooting
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
loops and spanning tree fun
spanning-tree is often not well understood
some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology
often does not help anyway when multiple layer2 domains are interconnected
loops and spanning tree fun
spanning-tree is often not well understood
some people opt to leave it disabled or leave the default config - since they don’t plan to build loops in their topology
often does not help anyway when multiple layer2 domains are interconnected
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
things that might help some:
enable spanning-tree within your domain
filter bpdus at the edge of your domain
limit total broadcast traffic on every port
make sure config has enough granularity for port speed (1% of 10G is still too much)
things that might help some:
enable spanning-tree within your domain
filter bpdus at the edge of your domain
limit total broadcast traffic on every port
make sure config has enough granularity for port speed (1% of 10G is still too much)
Loops Outside of Your Domain
#1 #2
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
why will a loop outside your domain hose your switch ?
not 100% clear
one possibility is mac address learning overload
switch flooded with packets for which it has to learn source mac addresses
mac addresses quickly flip-flop between ports
why will a loop outside your domain hose your switch ?
not 100% clear
one possibility is mac address learning overload
switch flooded with packets for which it has to learn source mac addresses
mac addresses quickly flip-flop between ports
Configuration IssuesConfiguration IssuesConfiguration IssuesConfiguration Issues
how could this be avoided ?
turn off mac address learning
for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway
could also have out-of-band mechanism to statically configure mac forwarding tables
will this entirely protect you ? don’t know
how could this be avoided ?
turn off mac address learning
for p2p vlans, could leave mac learning off and just flood all packets - they only have 1 direction to go anyway
could also have out-of-band mechanism to statically configure mac forwarding tables
will this entirely protect you ? don’t know
Troubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/Monitoring
how can you tell when a vlan is down ?
hint: think break in the middle of the topology
on vlan trunks, can’t see how much traffic is associated with each vlan
CoS hack on the 6500’s for this
lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking
how can you tell when a vlan is down ?
hint: think break in the middle of the topology
on vlan trunks, can’t see how much traffic is associated with each vlan
CoS hack on the 6500’s for this
lack of netflow data - can get sflow on some platforms, but analysis tools for sflow lacking
Troubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/MonitoringTroubleshooting/Monitoring
tools to trace current vlan path across the network
IU has developed a spanning-tree mapping tool that helps with this
“turn-around interfaces” useful for debugging performance problems
tools to trace current vlan path across the network
IU has developed a spanning-tree mapping tool that helps with this
“turn-around interfaces” useful for debugging performance problems
Thank You Thank You Thank You Thank You