Využití systémů řad Nexus 2000 a Nexus 5000 v návrhu přístupovév návrhu přístupové vrstvy sítě datového centra
Jaromír Pilař ([email protected])Consulting Systems Engineer, CCIE 2910
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID 1
Agenda
1. Nexus 5000 and Nexus 2000
2 Virtualized access switch2. Virtualized access switch
3. I/O consolidation in access layer
2
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
N 5000 d 2000Nexus 5000 and 2000
Platform descriptionat o desc pt o
3
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Nexus 5000 Systems Product PortfolioIndustry’s First I/O Consolidation Virtualization Fabric for
Enterprise Data CenterIndustry’s First I/O Consolidation Virtualization Fabric for
Enterprise Data Center
Nexus 5000SwitchFamily 56 Port L2 Switch28 Port L2 SwitchFamily 56-Port L2 Switch
• 40 Ports 10GE/FCoE/DCE, fixed• 2 Expansion Modules
28-Port L2 Switch• 20 Ports 10GE/FCoE/DCE, fixed• 1 Expansion Module
FC + Ethernet • 4 Ports 10GbE/FCoE/DCE
Fibre Channel • 8 Ports 1/2/4G FC
ExpansionModules Ethernet
• 6 Ports 10GE/FC E/DCE• 4 Ports 1/2/4G FC 10GE/FCoE/DCE
PartnersSW FCoE/DCE + 2x10GE2x10GE/DCE/FCoE 2x10GE
4
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
OS
Cisco Fabric Manager and Cisco Data Center Network Manager
Cisco NX-OS
Mgmt
Nexus 2000 Fabric Extender1GE Connectivity1GE Connectivity
5
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
N 5000 d 2000Nexus 5000 and 2000
Virtualized access switchtua ed access s tc
6
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center ArchitectureEvolution of the Hierarchical Design
The Data Center Architecture is based on a hierarchical design model
Layer 3
Core Aggregation block contains
the access and aggregation layersC id l 3
Layer 2
Layer 3
Services
Aggregation Core provides layer 3
boundary to the rest of the network
Dedicated service switches Servicesprovide application load balancing, firewall, etc.
Architecture is based on optimized design for control plane stability
Access
design for control plane stability and scalability
Need to understand how the design needs to evolve to accommodate server applications
7
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
accommodate server, applications and facilities requirements
Datacenter network fabricTop-of-rack Modular
8
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureVirtualized Access Switch
Nexus 5010/5020 Dual N5K supported as of NX-OS 4.1(3)N1
Nexus 5000/2148T Virtualized Access Switch provides a number of design options to address evolving g p gData Center requirements
Fabric Extender provides for flexibility in the design of the physical topologiestopologies
Aids in building larger layer 2 designs safely Support of latest spanning tree pp p g
enhancements Single virtual access switch
(Simplifies the layer 2 design)S t f 16 10GE Support of 16-way 10GE Etherchannel combined with vPC in NX-OS 4.1(3)N1 provides for increased network capacity Nexus 2148T Fabric
4 x 10GE Fabric Links per Fabric Extender
9
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Extender– 48 GE Portsp
(CX-1 Cu)
Data Center ArchitectureN5K/N2K - Logical Topology
1 Cisco Nexus 2148T Fabric1. Cisco Nexus 2148T Fabric Extender (N2K) and Nexus 5000 (N5K) Pod
2. N2K + N5K Pod t t ki
Each Virtualized Access Switch Pod configured to support
represents networking Access layer
3. Nexus 7000 at Distribution Layer
up to 576 1GE server ports at FCS
y
Nexus 5000/2000 Virtualized Access
Switch Pods . . .
10
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureDistributed Access Fabric
De-Coupling of the Layer 1 and Layer 2 Topologies
Optimization of both Layer 1 (Cabling) and Layer 2 p y ( g) y(Spanning Tree) Designs
Mixed cabling environment (optimized as required)
Flexible support for Future RequirementsFlexible support for Future Requirements
Nexus 5000/2000
. . .
11
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Combination of EoR and ToR cabling
Mixed ToR & EoR
Nexus 5000 and 2000
St ti i i P tStatic pinning, Port Channel and Virtual Port Channel on Nexus 5000C a e o e us 5000
12
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Fabric ExtenderF b i M d
Static PinningFabric Modes
Fabric Extender associates (pins) a server side (1GE) port with ana server side (1GE) port with an uplink (10GE) port
Server ports are either individually pinned to specific uplinks (static p p p (pinning) or all interfaces pinned to a single logical port channel
Behavior on FEX uplink failure depends on the configuration P t Ch l
Server Interface goes down
depends on the configuration Static Pinning – Server ports
pinned to the specific uplink are brought down with the failure of
Port Channel
brought down with the failure of the pinned uplink
Port Channel – Server traffic is shifted to remaining uplinks based
t h l h hServer Interface
stays active
13
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
on port channel hash stays active
Static Pinning
A B C D
N5k01
Fabric Ports1,2,3,4
N5k01
Fabric Ports1,2,3,4
N5k01
Fabric Ports1,2,3,4
N5k01
Fabric Ports1,2,3,4
1-48
N2k01Host Ports N2k01
1-24 25-48
Host Ports N2k01
1-16 17-32
Host Ports
33-48
N2k01
1-12 13-24
Host Ports
25-36 37-481-48
pinning max-links 1 pinning max-links 2
1-24 25-48
pinning max-links 3
1-16 17-32 33-48
pinning max-links 4
1-12 13-24 25-36 37-48
14
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Port-Channeling
1. With Static Pinning if a fabric uplink port fails, the associated HIFs are shut
Aassociated HIFs are shut down
2. With Port-Channeling if a f b i li k f il HIF
N5k01fabric uplink fails HIFs use the remaining fabric uplinks.
Fabric Ports
H t P t
1,2,3,4
1-48
N2k01
i i li k 1
Host Ports
pinning max-links 1
15
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureDual Supervisor – NXOS 4.0(1a)N2p ( )
Prior to the 4.1(3) release (Q3CY09) a Fabric Extender can ( )only be actively connected to a single N5K
During the link-up SDP exchange Fabric Extender identifies if it
N5K-A N5K-B
Fabric Extender identifies if it actively connected to an N5K
If the SDP identifies a different N5K as already being actively connected
SDP exchange as already being actively connected to the Fabric Extender the newly connected N5K port will err-disable
No prioritization between N5KErr-disable
Potential Race Condition if FEX connected to two N5K
Manual process can be used to
16
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
pprovide a backup configuration
Data Center Access ArchitectureDual Supervisor – NX-OS 4.1(3)xxp ( )
NX-OS 4.1(3) provides vPC support on the N5K either for use ppto provide server NIC teaming or to provide dual supervisor configuration for the virtualized switch
In the redundant supervisor mode the Etherchannel fabric uplink is split across two N5Ks
Static pinning is not supported in a redundant supervisor mode
Server ports appear on both N5K Currently configuration for all ports
must be kept in sync manually on both N5Ks (N2K has in fact 2 configuration points in this setup)
17
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
configuration points in this setup)
Fabric ExtenderPort Channel Configurationg
interface port-channel1switchport mode fex-fabricdescription Fabric Extender 100fex associate 100
Configure the Physical Ports as
interface Ethernet1/1switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100
Physical Ports as Members of the
Fabric EtherChannel
interface Ethernet1/2switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100
Configure the Port interface Ethernet1/3switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100
interface Ethernet1/4
Configure the Port Channel and Its Members to be
Associated with a Specific Fabric
Extenderinterface Ethernet1/4switchport mode fex-fabricdescription Member of Fabric Extender 100 Etherchannel Linkchannel-group 1fex associate 100
fex 100
Extender
18
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
pinning max-links 1description Fabric Extender 100 – Using Etherchannel 1
Fabric ExtenderVirtual Port Channel Configurationg
interface port-channel1switchport mode trunkswitchport trunk allowed vlan 1,10vpc peer-link
Configure the vPC Peer Link (Full vPC Configuration not included in this
example)interface port-channel100
switchport mode fex-fabricvpc 100fex associate 100
interface Ethernet1/17
example)
interface Ethernet1/17switchport mode trunkswitchport trunk allowed vlan 1,10channel-group 1 mode active
interface Ethernet1/18switchport mode trunk
Configure the Physical Ports as Members of the
Fabric EtherChannelswitchport trunk allowed vlan 1,10channel-group 1 mode active
interface Ethernet1/37switchport mode fex-fabricchannel-group 100fex associate 100fex associate 100
interface Ethernet1/38switchport mode fex-fabricchannel-group 100fex associate 100
Configure the Port Channel and Its Members to be
Associated with a Specific Fabric
19
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
fex 100pinning max-links 1
Extender
Data Center Access ArchitecturevPC Redundancy Models Dual ChassisvPC Redundancy Models – Dual Chassis vPC provides two redundancy designs for the virtualized access switch Option 1 - MCEC connectivity from the serverp y
Two virtualized access switches bundled into a vPC pair Full redundancy for supervisor, line card, cable or NIC failure Logically a similar HA model to that currently provided by VSS Logically a similar HA model to that currently provided by VSS
vPC peersTwo Virtualized access switches Each with a Single Supervisor
20
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
MCEC from server to the access switch
Data Center Access ArchitecturevPC Redundancy Models Dual SupervisorvPC Redundancy Models – Dual Supervisor vPC Option 2 – Fabric Extender connected to two Nexus 5000
From the server perspective a single access switch with each line card p p gsupported by redundant supervisors
Full redundancy for supervisor, fabric via vPC and cable or NIC failure via active/standby NIC redundancyL i ll i il HA d l t th t tl id d b d l Logically a similar HA model to that currently provided by dual supervisor based modular switch
Fabric Extender dual homed to redundant Nexus 5000
21
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureVirtual Port Channel Design ConsiderationsVirtual Port Channel – Design Considerations
Dual tiered vPC is not supported with the NX-OS 4.1(3) release
4. TCP Flow also from SRC A appears
on fabric interface Po 3. TCP Flow from SRC A
appears on fabric vPC provides a logical port
channel interface on the N5K In the unsupported configuration
shown each N2K is attached with a
100ppinterface Po 200
shown each N2K is attached with a port channel fabric interface and then carried over that interface is a second server port channel interface
5. MAC flap between two
different fabric ports
Two dependent layers of Etherchannel hashing (Server and N2K) for the same flowsT i f PC h ill
2. N2K hashes both flows to the same
ports
Two tiers of vPC as shown will result in MAC addresses flapping between two fabric ports
Timeframes to support this
1. Server hashes two flows up
alternate uplinks
N5K
22
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Dual Tier vPC shown above is notsupported with the 4.1(3) N5K release
ppconfiguration is still TBD
Data Center Access ArchitectureVirtual Port Channel Design ConsiderationsVirtual Port Channel – Design Considerations
Two independent vPC configurations are supported onconfigurations are supported on the same N5K
vPC upstream to the N7K is independent of the downstreamindependent of the downstream vPC connecting the N2K
Independent hashing decisions for the upstream and d t t ffi fldownstream traffic flows
23
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureVirtual Port Channel – Design Considerationsg
Multiple Interfaces on the Single Interface
VMVMVM
Interfaces on the same N5K
requires HW resource
VMVMVM
gon each N5K
VM #4
VM #3
VM #2
VM #4
VM #3
VM #2
Nexus 5000 is limited to 12 physical (HW) Ethernet port channels (increasing to 16 in NX-OS 4 1(3))(increasing to 16 in NX OS 4.1(3))
A MCEC vPC configuration with only two links (one per N5K) does ‘not’ consume a HW port channel resource
A HW port channel will be required in a vPC environment only if there is
24
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
A HW port channel will be required in a vPC environment only if there is more than one interface on the same N5K (e.g. 4 x 10GE links)
Data Center Access ArchitectureVirtual Port Channel – FCoE Considerations
SAN ‘B’SAN ‘A’vPC - MCEC
Two logical views of the
VM VM VM VMVMVM
same physical topology
#4#3#2VM #4
VM #3
VM #2
In a Unified I/O configuration (FCoE) we have two distinct topologies
Isolated access switches - SAN ‘A’ and SAN ‘B’
Combined access switches – vPC supporting MCEC
To ensure correct forwarding behaviour ‘vfc’ interface can only be
25
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
g yassociated with a vPC etherchannel (only one physical interface per switch)
Data Center Access ArchitectureNexus 5000 & 2000 Deployment Optionsp y p
Not SupportedNot SupportedNot SupportedNot Supported
With PC
26
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
With vPCWithout vPC
FCS OptionsFCS OptionsFCS OptionsFCS Options vPC vPC -- Q3CY09Q3CY09vPC vPC -- Q3CY09Q3CY09 TBDTBDTBDTBD
Nexus 5000 and 2000
Fabric Extender Layer 2 Design Considerationses g Co s de at o s
27
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureN2K/N5K Spanning Tree Design Considerations
Root BridgeHSRP A ti
Secondary Root B idHSRP Active Bridge
HSRP Standby
BPDU Guard Global BPDU Filter reduces the spanning
tree load (BPDUs generated on a Host
Bridge Assurance
UDLD
gPort)
VMWare Server Trunk Needs to Carry
VSwitch
yMultiple VLANs which can increase the STP
load
28
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
VM #1
VM #4
VM #3
VM #2
Data Center Access ArchitectureN2K/N5K Spanning Tree Design Considerations
Nexus-5000# show spanning-tree interface ethernet 100/1/48 detail
Port 560 (Ethernet100/1/48) of VLAN0100 is designated forwarding Port path cost 4 Port priority 128 Port Identifier 128 560Port path cost 4, Port priority 128, Port Identifier 128.560Designated root has priority 24776, address 0023.ac64.73c3Designated bridge has priority 32968, address 000d.eca4.533cDesignated port id is 128.560, designated path cost 2Timers: message age 0, forward delay 0, hold 0Number of transitions to forwarding state: 1The port type is edgeLink type is point-to-point by default BPDU Guard Is Enabled by Default and Link type is point to point by defaultBpdu guard is enabledBPDU: sent 215784, received 0
BPDU Guard Is Enabled by Default and Can Not be Disabled on FEX Server Ports
interface port-channel200switchport mode trunkswitchport mode trunkswitchport trunk allowed vlan 200-202spanning-tree port type network
interface Ethernet1/33switchport mode trunkswitchport trunk allowed vlan 200-202udld enable
Bridge Assurance Requires the Port Type to be
Configured as ‘network’
udld enablechannel-group 200 mode active
interface Ethernet1/37switchport mode trunkswitchport trunk allowed vlan 200-202udld enablechannel-group 200 mode active
29
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
channel group 200 mode active
Nexus5000(config)# spanning-tree port type edge bpdufilter default Global BPDU Filter
Data Center Access ArchitecturevPC - Spanning Tree Design Considerations
Both vPC PeersAct as the
Enabling vPC on the access to aggregation links improves
Single Logical Link to STP
Act as the default GW
aggregation links improves layer 2 scalability Removing physical loops
out of the layer 2 topologyvPC
Link to STPy p gy Reducing the STP state on
the access and aggregation layer
Fabric Links(No
Spanning Tree)
The use of vPC does result in a reduction of logical port count on the aggregation but does involve CFS synchronization of
Server PortsBPDU Guard
involve CFS synchronization of state between the two aggregation nodes
30
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureSupported Topologies and Design Considerations
Fabric Extender design needs to take into account topology
Spanning Tree - Rapid
PVST+ or to take into account topology and hardware capabilities of the current generation FEX
Currently all Fabric Extender
PVST+ or MST
Switch server ports are hard coded as STP edge ports BPDU Guard is enabled and
can not be disabled
Switch Fabric No STP
Spanning Tree Edge Ports
can not be disabled
Current generation HW does not support port channel between any two server ports
th N 5000on the same Nexus 5000 Currently all Fabric Extender
server ports support 1000BaseT only
VSwitch
VM #1
VM #4
VM #3
VM #2
31
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
1000BaseT only #1 #4#3#2
Data Center Access ArchitecturePort Channel Design Considerations – 4.0(1a)N2
With NX-OS 4.0(1a)N2 the Nexus Fabric Extender does not support
S tpp
Port Channel (Ether Channel) between any two server ports (Logical Interfaces) on the same Nexus 5000 N5K
Server ports
Not possible to bundle two or more ports on the same Fabric Extender
Not possible to bundle ports spreadNot possible to bundle ports spread across two or more Fabric Extenders that are associated with the same N5K (line cards in the same virtual switch)same virtual switch)
Servers are unable to use a port channel configuration unless vPC is enabled (two port vPC only)
32
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitecturePort Channel Design Considerations – 4.1(3)N1
With the Cronulla release (Q3CY09) Virtual Port Channel (vPC) can be N5KA N5KB( )used to create a server side port channel between two ports on different N5K’s
Not possible to bundle two or more
N5KA N5KB
Not possible to bundle two or more physical links to two Logical Interface Ports even in a vPC bundle
Not possible to bundle ports spreadNot possible to bundle ports spread across two or more Fabric Extenders that are associated with the same N5K (line cards in the same virtual switch)switch)
Servers are unable to use a port channel configuration unless vPC is enabled
Maximum of two ports supported in a vPC bundle
33
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Data Center Access ArchitectureDownstream Device Design Considerations
Fabric Extender server ports are configured as either STP edge
Attached devices canutilize trunks and
source multiple MAC addressesg g
access or edge trunk ports BPDU Guard is enabled by default
and can not be disabled Fabric Extender ports can be
configured as trunks and can receive multiple MAC addressesI ll F b i E t d d i th In all Fabric Extender designs the layer 2 topology must not require STP to be configured on the attached device
Fabric Extender ports are STP edge ports, BPDU Guard is
enabled. Downstream devices
VSwitch
VM #1
VM #4
VM #3
VM #2
STP Disabled
34
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
enabled. Downstream devices can not generate BPDU’s.
STP Disabled
Data Center Access ArchitectureFlexlink and Fabric Extender
Design Topologies that do not require STP for downstream link qrecovery can be configured
Flexlink is one option for connecting a downstream switch that is STP independentindependent
Flexlink disables Spanning Tree on the active and backup interfacesBPDU d d i HW
STP edge BPDUs are dropped in HW on
ingress and not transmitted upstream on the ‘flexlink’ ports
Note: Extending layer 2 downstream
gports
Note: Extending layer 2 downstream of the N2K is not a recommended best practices solution Flexlink
configured on the stub switch
li k
35
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
dc11-2960-1#sh spanning-tree interface gig 0/25 detail no spanning tree info available for GigabitEthernet0/25
uplinks
Data Center Access ArchitectureFlexlink and Fabric Extender
Spanning Tree should ‘not’ be disabled on either the Nexus 5000 nor the downstream switches (e.g. Cisco 2960)
External devices can still createExternal devices can still create loops in the topology
DO NOT extend the downstream device topologydevice topology
This design suitable only for migration environments
2960 for 10/100 iLO
1GE Blade Switches
36
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
BPDU Guard, port security, storm control, 802.1x
Nexus 5000 and Nexus 2148 Fabric Extender
Fabric Extender Configurations
Nexus 5000
DC Access Layer consolidation
37
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Datacenter network fabricTop-of-rack Modular
38
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Datacenter storage fabricgTop-of-rack Modular
Smaller switch per bi t
Modular switch for group of cabinets
cabinet
Pair of switches for dual-homing
Uplinks to redundant core layer - 2/4/10G FC
g p
Pair of switches for dual-homing
Uplinks to redundant core layer
2/4/10G FCcore layer 2/4/10G FC
Use of portchanneling
Larger port count in core is required
- 2/4/10G FC
Use of portchanneling
Lower port count in core is required
C
Pro: Efficient CablingPro: Improved CoolingCon: Number of Devices/MgmtCon: Overhead protocol load
Core
Pro: Number of Devices/MgmtPro: Overhead protocol loadCon: Cabling ChallengesCon: Cooling Challenges
Core
Edge
Core
Edge
39
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Where is the main consolidation potential ?Where is the main consolidation potential ?1. Majority of ports in fabric is in access layer regardless
of fabric type => access layer has the highest potential for consolidation
2. Question is which technology can serve different d f diff t f b i d h it t d ?needs of different fabrics, do we have it today?
3. What are the requirements and what are other driving factors?factors?
40
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Consolidation - historyConsolidation history1. Previous attempts
Fibre ChannelNever credible as data network infrastructre
InfiniBandInfiniBandNot Ethernet
iSCSINot Fibre Channel
2. Key enabling technologies: PCI-Express and 10GEBefore PCI-Express there was not enough I/O bandwidth in the
servers
It needs to be Ethernet but
41
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
It needs to be Ethernet, but…1 GE didn’t have enough bandwidth
Drivers for 10GE to the Servers
Multicore CPU Architectures Allowing Bigger and M lti l W kl d th S M hi
Drivers for 10GE to the Servers
Multiple Workloads on the Same Machine
S Vi t li ti D i i th N d f MServer Virtualization Driving the Need for More Bandwidth per Server Due to Server Consolidation
Growing Need for Network Storage Driving the Demand for Higher Network Bandwidth to the Server
Multicore CPUs and Server Virtualization Driving the Demand for Higher Bandwidth Network Connections
42
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Demand for Higher Bandwidth Network Connections
I/O Consolidation in the Network
Processor
Memory
Processor
MemoryMemory Memory
I/O SubsystemI/O I/O I/OLA
N
orag
e
IPC
y
AN
orag
ePCL
Sto I LA
Sto IP
43
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
IPC: Inter-Process Communication
I/O Consolidation in the Host
1. Fewer CNAs (Converged Network Adapters) instead of NICs, HBAs, and HCAs
2. Limited number of interfaces for Blade Servers
FC TrafficFC HBA
FC TrafficFC HBAAll Traffic Goes over
10 GE
CNA
CNA
NIC Enet Traffic
NIC Enet Traffic
NIC Enet Traffic
HCA IPC Traffic
44
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
IPC TrafficHCA
Cabling and I/O Consolidation
45
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Unified I/O Use Case
Today:
FC HBA
FC HBA
FC Traffic
FC Traffic
FC HBA
FC HBA
FC HBA
FC HBA
1. Parallel LAN/SAN Infrastructure
2. Inefficient use of Network Infrastructure
3 5 ti hi hFC HBA
NIC
FC Traffic
Enet Traffic
FC HBA
NIC
FC HBA
NIC
3. 5+ connections per server – higher adapter and cabling costs
Adds downstream port costs; cap-ex and op-ex
NIC Enet TrafficNICNIC
p p
Each connection adds additional points of failure in the fabric
Power and cooling
4. Longer lead time for server provisioning
5. Multiple fault domains – complex di ti
46
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
diagnostics
6. Management complexity –firmware, driver-patching, versioning
What is Data Center Bridging?
Provides class of service flow control. Ability to support Priority-based Flow
BenefitFeature
g g
Grouping classes of traffic into “Service Lanes”IEEE 802.1Qaz, CoS based Enhanced Transmission
CoS Based BW Management
o des c ass o se ce o co o b y o suppostorage traffic (IEEE 802.1Qbb)
o ty based oControl (PFC)
End to End Congestion Management for L2 network (IEEE 802.1Qau)
Congestion Notification (BCN/QCN)
Eliminate Spanning Tree for L2 topologies L2 Multi-path for Unicast &
Auto-negotiation for Enhanced Ethernet capabilities DCBX (Switch to NIC)
Data Center Bridging Exchange
Provides ability to transport various traffic types (e.g. Storage, RDMA)
Lossless Service
p g p g
Utilize full Bi-Sectional bandwidth with ECMPL2 Multi path for Unicast & Multicast
47
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Merging the requirements
1. LAN/IP1. LAN/IPMust be Ethernet
Losing some frames is good for TCP
2. StorageMust follow the Fibre Channel model
Losing frames is not an option
3. IPCDon’t care of the underlying network, provided that
It is cheap
48
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
It is low latency
It supports APIs like OFED, RDS, MPI, etc. . .
What is Fibre Channel over Ethernet?
1. From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… an Ethernet
cloud
2. From an Ethernet standpoints it’s2. From an Ethernet standpoints it sYet another ULP (Upper Layer Protocol) to be transported,
but… a challenging one!
3. And technically…
FCoE is an extension of Fibre Channelf
49
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
onto a Lossless Ethernet fabric
FCoE Enablers and Encapsulation
1. 10Gbps Ethernet
2. Lossless EthernetMatches the lossless behavior guaranteed in FC by B2B credits
3. Ethernet jumbo framesMax FC frame payload = 2112 bytes
T t l f i 2180 b t
Same as a physical FC frame
Normal ethernet frame, ethertype = FCoE
Total max frame size = 2180 bytes
Ethe
rnet
Hea
der
FCoE
Hea
der
FCH
eade
r
FC Payload CR
C
EOF
FCS
Same as a physical FC frame
50
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
E
Control information: version, ordered sets (SOF, EOF)
Unified I/O Use Case
Today:
Management
SAN BSAN ALAN
1. Aggregation/Core it h
FC HBA
FC HBA switchesFC HBA
NIC
2. Access – Top of the Rack switches
NIC
Rack switches
3. Servers
51
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
EthernetFC
FCoE I/O Consolidation BenefitFewer HBA/NIC’s per Server
FC E CNA
FC HBA
FC HBA
FC Traffic
FC Traffic FC EFCoE CNA
FCoE CNA
FC HBA
NIC
FC Traffic
Enet TrafficFCoE
&Enet@
NIC Enet Traffic
@10GE
FCoE Converged Network Adaptor
52
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Customers purchase fewer NIC’s and HBA’s
Unified I/O Use Case
TodayUnified I/O Phase 1Unified I/O Phase 11. Reduction of server
d t
Management
SAN BSAN ALAN adapters2. Fewer Cables3. Simplification of access
l & blilayer & cabling4. Gateway free
implementation - fits in installed base of existing
FCoE Switch
installed base of existing LAN and SAN
5. L2 Multipathing Access –Distribution
6. Lower TCO7. Investment Protection
(LANs and SANs)
53
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
FCoEEthernetFC
8. Consistent Operational Model
9. One set of ToR Switches
Unified I/O Use Case
Unified I/O Phase 21 Elimination of parallel network
Unified I/O Phase 2
Enh E Fabric w/ FCoE
1. Elimination of parallel network infrastructure
2. L2/L3 Multipathing end to endManagement
StorageArrays
3. Faster infrastructure provisioning
4. Lower TCO5. Disk array access via
Enhanced Ethernet or Native FC
FCoE Switch
54
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
Enh E and FCoEFC
Unified I/O Use Case
Unified I/O Phase 31 Datacenter wide Unified
Today:Unified I/O Phase 1 (Mid 2008)Unified I/O Phase 31. Datacenter wide Unified
Fabric for LAN and SAN2. L2/L3 Multipathing end to
endManagement
SAN BSAN ALANManagement
SAN BSAN ALANEnh E Fabric w/ FCoE
StorageArrays
end3. Consistent network policies
across datacenter4 Lo er TCO
Management
4. Lower TCO
55
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID
DCE and FCoEEthernetFC
DCE and FCoEEthernetFCEnh E and FCoE
56
© 2009 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialPresentation_ID