BRKSAN-2047
FCoE - Design, Operations and Management Best Practices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 2
Before We Get Started
Intermediate level session focused on Unified Data Centre Design using Fibre Channel over Ethernet
Prerequisites:basic understanding of Fibre Channel and Storage Designbasic understanding of Ethernet and LAN Designbasic understanding of the FCoE protocol and terminology
Other recommended sessions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 3
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks and Terminology
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 4
The Evolving Data Centre AccessThe Consolidated Nexus Edge Layer
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
Cor
e/A
ggre
gatio
n La
yer
Virt
ualiz
ed E
dge/
Acc
ess
Laye
r
The Access Layer is becoming more than just a port aggregatorEdge of the growing Layer 2 topology
Scaling of STP Edge PortsVirtual embedded switchesvPC and loop free designsLayer 2 Multi-Pathing (future)
Foundational element for Unified I/O ‘and’Unified Wire
DCB and Multi-Hop FCoE SupportEnhanced Multi-hop FCoE with E-NPV
Single Point for Access Management VN-Tag and Port Extension – Nexus 2000 (current) VSM and VN-Link (future)
FC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 6
Understand the design requirements of a Unified Network
Be able to design single-hop Unified Networks available today which meet the demands of both SAN and LAN networks
Start the conversation between Network and Storage teams regarding consolidation and FCoE beyond the access layer
Understand the Operations and Management aspects of a Unified Network
Why are we here?Session Objectives
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 7
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks and Terminology
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 8
CEE (Converged Enhanced Ethernet) is an informal group of companies that submitted initial inputs to the DCB WGs.
Data Centre EvolutionIEEE DCB (Data Centre Bridging)
Feature / Standard Standards Status
Priority Flow Control IEEE 802.1Qbb (PFC)
PAR approved, Editor Claudio DeSanti (Cisco), draft 1.0 published
Bandwidth Management IEEE 802.1Qaz (ETS)
PAR approved, Editor Craig Carlson (Qlogic), draft 0.2 published
Data Center Bridging Exchange Protocol (DCBX)
This is part of:Bandwidth Management IEEE 802.1Qaz
** Nexus 5000 supports CEE-DCBX as well as previous generations (CIN-DCBX)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 9
Packet
R_R
DY
Fibre ChannelFibre Channel
Transmit Queues Ethernet Link Receive Buffers
EightVirtualLanes
OneOne OneOne
TwoTwo TwoTwo
ThreeThree ThreeThree
FourFour FourFour
FiveFive FiveFive
SevenSeven SevenSeven
EightEight EightEight
SixSix SixSix
STOP PAUSE
B2B CreditsB2B Credits
Enables lossless Ethernet using PAUSE based on a COS as defined in 802.1pWhen link is congested, CoS assigned to “no-drop” will be PAUSEDOther traffic assigned to other CoS values will continue to transmit and rely on upper layer protocols for retransmissionNot only for FCoE traffic
Priority Flow ControlFCoE Flow Control Mechanism
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 10
Once feature fcoe is configured, 2 classes are made by default
Priority Flow ControlOperations Configuration – Switch Level
DCB Switch
DCB CNA Adapter
class-fcoe is configured to be no-drop with an MTU of 2158
Best Practice - use the default COS value of 3 for FCoE/no-drop trafficCan be changed through QOS class-map configuration
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 11
Checking the PFC settings on an interface
VL bmap = COS set for PFC
Priority Flow ControlVerifying Configurations
VL bmap Binary COS
1 00000001 02 00000010 14 00000100 28 00001000 316 00010000 432 00100000 564 01000000 6128 10000000 7
show interface priority-flow-control
Shows ports where PFC is configured, the COS value associated with PFC as well as the PAUSE packets received and sent on that port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 12
Offered Traffic
t1 t2 t3
10 GE Link Realized Traffic Utilization
3G/s HPC Traffic3G/s
2G/s
3G/sStorage Traffic3G/s
3G/s
LAN Traffic4G/s
5G/s3G/s
t1 t2 t3
3G/s 3G/s
3G/s 3G/s 3G/s
2G/s
3G/s 4G/s 6G/s
Prevents a single traffic class of “hogging” all the bandwidth and starving other classesWhen a given load doesn’t fully utilize its allocated bandwidth, it is available to other classes Helps accommodate for classes of a “bursty” nature
Enhanced Transmission SelectionBandwidth Management
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 13
Enhanced Transmission SelectionBandwidth Management
Once feature fcoe is configured, 2 classes are made by defaultBy default, each class is given 50% of the available bandwidth
1Gig FC HBAs
1Gig Ethernet NICs
Traditional Server
A typical server has equal BW per traffic type
Best Practice : FCoE and Ethernet each receive 50%Can be changed through QoS settings when higher demands for certain traffic exist (i.e. HPC traffic, more Ethernet NICs)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 14
Discovers DCB capabilities of peer devicesNegotiates Ethernet capability’s – PFC, ETS, CoS valuesSimplifies management of DCB nodes
Allows for configuration and distribution of parameters from one node to another
Responsible for Logical Link Up/Down signaling of Ethernet and Fibre ChannelUses Link Layer Discovery Protocol (LLDP) defined by 802.1AB to exchange and discover DCB capabilitiesDCBX negotiation failures result in:
per-priority-pause not enabled on CoS valuesvfc not coming up – when DCBX is being used in FCoE environment
Data Center Bridging eXchangeControl Protocol
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 15
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks and Terminology
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 16
FCoE Benefits
Mapping of FC frames over Ethernet
Enables FC to run on a lossless Data Center Ethernet network
Wire Server Once
Fewer cables and adapters and switches
Software Provisioning of I/O
Interoperates with existing SANs
No gateway—stateless
Standard – June 3, 2009FibreChannel
Ethernet
Fibre Channel over Ethernet
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 17
From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… Ethernet
From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported
FC-0 Physical Interface
FC-1 Encoding
FC-2 Framing & Flow Control
FC-3 Generic Services
FC-4 ULP Mapping
Ethernet Media Access Control
Ethernet Physical Layer
FC-2 Framing & Flow Control
FC-3 Generic Services
FC-4 ULP Mapping
FCoE Logical End Point
Fiber Channel over Ethernet Protocol Mapping
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 18
Unified FabricFibre Channel over Ethernet (FCoE)
FCoE is Fibre Channel at the host and switch level
Same Operational Model Same Operational Model
Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management
Same Managementand Security ModelsSame Managementand Security Models
Easy to UnderstandEasy to Understand
Completely based on the FC model
Same host-to-switch and switch-to-switch behavior of FC
E.g., in order delivery or FSPF load balancing
WWNs, FC-IDs, hard/soft zoning, DNS, RSCN
Aligned with the FC-BB-4 Model,
Standardizedin FC-BB-5
Aligned with the FC-BB-4 Model,
Standardizedin FC-BB-5
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 19
Both Protocols Have…• Two different Ethertypes• Two different frame formats• Both are defined in FC-BB-5
FCoE itself Is the data plane protocol
It is used to carry most of the FC frames and all the SCSI traffic
Uses Fabric Assigned MAC address (dynamic)
FCoE itself Is the data plane protocol
It is used to carry most of the FC frames and all the SCSI traffic
Uses Fabric Assigned MAC address (dynamic)
FIP (FCoE Initialization Protocol)It is the control plane protocol
It is used to discover the FC entities connected to an Ethernet cloud
It is also used to login to and logout from the FC fabric
FIP (FCoE Initialization Protocol)It is the control plane protocol
It is used to discover the FC entities connected to an Ethernet cloud
It is also used to login to and logout from the FC fabric
http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html http://www.cisco.biz/en/US/prod/collateral/switches/ps9441/ps9670/white_paper_c11-560403.html
Fiber Channel over EthernetData and Control plane
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 20
Protocol used by FCoE capable devices to discover other FCoE capable devices within the Ethernet Cloud
Enables FCoE adapters (CNAs) to discover FCoE switches (FCFs) on a VLAN (the FCoE VLAN)Establishes a virtual link with between the adapter and FCF or between two FCFs (VE_ports) – accomplished with a FLOGI
FIP frames use a different Ethertype from FCoE frames making FIP-Snooping by DCB capable Ethernet bridgesBuilding foundation for future multi-hop FCoE topologies
Multi-hop refers to FCoE extending beyond a single “hop” or “access” switchToday, “multi-hop” is achievable with a Nexus 4000 (FIP Snooping Bridge) connected to Nexus 5000 (FCF)
Fibre Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 21
Step 1: FCoE VLAN DiscoveryFIP sends out a multicast to
ALL_FCF_MAC address looking for the FCoE VLAN
FIP frames use the native VLAN
Step 2: FCF DiscoveryFIP sends out a multicast to the
ALL_FCF_MAC address on theFCoE VLAN to find the FCFs answering for that FCoE VLAN
FCF’s responds back with their MAC address
Step 3: Fabric LoginFIP sends a FLOGI request to the
FCF_MAC found in Step 2Establishes a virtual link between
host and FCF
Enode Initiator
FCoE SwitchFCF
VLANDiscover
y
FLOGI/FDISC
FLOGI/FDISC Accept
FC Command
FC Command Responses
FCoEInitialization
Protocol (FIP)
FCoEProtocol
VLANDiscovery
FCFDiscovery
SolicitationFCF
DiscoveryAdvertisement
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
** FIP does not carry any Fibre Channel frames
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 22
The FCoE VLAN is manually configured on the Nexus 5000
The FCF-MAC address is configured on the Nexus 5000 by default once feature fcoe has been configured
This is the MAC address returned in step 2 of the FIP exchangeThis MAC is used by the host to login to the FCoE fabric
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
** FIP does not carry any Fibre Channel frames
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 23
Fiber Channel over Ethernet ProtocolFCoE Initialization Protocol (FIP)
Step 3 - login process: show flogi database and show fcoe database show the logins and associated FCiDs, xWWNs and FCoE MAC addresses
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 24
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks and Terminology
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 25
FCF : Fibre Channel Forwarder (Nexus 5000, Nexus 7000, MDS 9000)FPMA : A unique MAC address that is assigned by an FCF to a single EnodeEnode : a Fiber Channel node that is able to transmit FCoE frames using one or more ENode MACs.FCoE Pass-Through : a DCB device capable of passing FCoE frames to an FCF (i.e. FIP-Snooping)
FIP Snooping BridgeEthernet N-Port Virtualizer
Single hop FCoE : running FCoE between the host and the first hop access level switchMulti-hop FCoE : the extension of FCoE beyond a single hop into the Aggregation and Core layers of the Data Centre Network
FCF
E_Node
FCoE Building BlocksThe Acronyms Defined
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26Session_IDPresentation_ID
Enode MAC AddressFibre Channel over Ethernet Addressing Scheme
Enode MAC assigned for each FCIDEnode MAC composed of a FC-MAP and FCID
FC-MAP is the upper 24 bits of the Enode’s MACFCID is the lower 24 bits of the Enode’s MAC
FCoE forwarding decisions still made based on FSPF and the FCID within the Enode MAC
FC FabricFC Fabric
Domain ID
FC-MAP(0E-FC-xx)
FC-ID7.8.9
FC-MACAddress
FC-MAP(0E-FC-xx)
FC-ID10.00.01
Fibre Channel FCID Addressing
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 27
Fibre Channel Drivers
Ethernet Drivers
Operating System
PCIe
Ethernet
Fibre Channel
10GbE
10GbE
Link
Ethernet Driver bound to Ethernet NIC PCI address
FC Driver bound to FC
HBA PCI address
Replaces multiple adapters per server, consolidating both Ethernet and FC on a single interfaceAppears to the operation system as individual interfaces (NICs and HBAs)First Generation CNAs from support PFC and CIN-DCBXSecond Generation CNAs support PFC, CEE-DCBX as well as FIP
Single chip implementation
FCoE Building BlocksConverged Network Adapter
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 28
FCF (Fibre Channel Forwarder) is the Fibre Channel forwarding element inside an FCoE switch
Fibre Channel logins (FLOGIs) happens at the FCFConsumes a Domain ID
FCoE encap/decap happens within the FCFForwarding based on FC information
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethport
Ethernet Bridge
FCport
FCport
FCport
FCport
FCF
FCoE SwitchFC Domain ID : 15FC Domain ID : 15
FCoE Building BlocksFibre Channel Forwarder
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 29
VE_Port
VF_Port
VF_Port
VE_Port
VN_Port
VN_Port
Fibre Channel over Ethernet Switch
E_NPV SwitchVF_Port VNP_PortFCF
Switch
End Node
End Node
FCoE Switch : FCF
**Available NOW
**EagleHawk + Timeframe**EagleHawk Timeframe
FCoE Building BlocksFCoE Port Types
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 30
Unified I/O – using Ethernet as the transport medium in all network environments -- no long needing separate cabling options for LAN and SAN networks
Unified Wire – a single DCB Ethernet link actively carrying both LAN and Storage (FC/FCoE/NAS/iSCSI) traffic simultaneouslyUnified Dedicate Wire -- a single DCB Ethernet link capable of carrying all traffic types but actively dedicated to a single traffic type for traffic engineering purposes
Unified Fabric – An Ethernet Network made up of “Unified Wires”everywhere: all protocols – network and storage –transverse all links simultaneously
FCoE Building BlocksThe New Buzzzword…”Unified”
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 31
CNA
Unified Dedicated Wire
Unified Dedicated Wire
L2
L3
Core
Aggregation
Shared Access
Fabric ‘A’ Fabric ‘B’
Unified WireUnified Wire
Unified Wire to the access switchcost savings in the reduction of required equipment“cable once” for all servers to have access to both LAN and SAN networks
Unified Dedicated Wire from access to aggregation
separate links for SAN and LAN traffic - both links are same I/O (10GE)advanced Ethernet features can be applied to the LAN linksmaintains fabric isolation
FCoE Building BlocksUnfied Wire vs Unified Dedicated Wire
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 32
L2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Ethernet an Storage traffic EVERYWHERE
Ethernet an Storage traffic EVERYWHERE
A single networkAll links carry all types of traffic simultaneously
all/any Storage and Network protocols
Possible reduction of equipment leading to cost savingsAbolition of Fabric A and Fabric B
Single SAN fabric with redundant fabric services
FCoE Building BlocksThe Unified Fabric - Definition
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 33
Unified Technology
L2
L3
Core
Aggregation
Access
Core
Edge
Fabric ‘A’ Fabric ‘B’
Ether-channel Multi-pathing
Virtual Port-Channel (VPC)
NIC /
CNA
CNA
Fibre Channel over Ethernet SAN
Fibre Channel over Ethernet SAN
Native Ethernet LANNative Ethernet LAN
LAN and SAN networks share the same Unified I/O building blocks: switches and cablingmaintains operations, management and troubleshooting
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 34
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 35
Ethernet is non-deterministic.Flow control is destination-basedRelies on TCP drop-retransmission / sliding window
Fibre-Channel is deterministic.Flow control is source-based (B2B credits)Services are fabric integrated (no loop concept)
The Design RequirementsEthernet vs Fibre Channel
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 36
Ethernet/IPGoal is to provide any-to-any connectivity
Unaware of packet loss – relies on ULPs for retransmission and windowingProvides the transport without worrying about the servicesEast-west vs north-south traffic ratios are undefinedServices provided by upper layers
Network design has been optimized for:Control protocol interaction (STP, OSPF, EIGRP, L2/L3 boundary, …)High Availability from a transport perspective by connecting nodes in mesh architecturesHigh Availability for Services is implemented seperately
??
?
?
????
?
?
??
Switch Switch
Switch
?
Client/Server Relationships are not pre-defined
? ?
?
Fabric topology and traffic flows are highly flexible
The Design RequirementsClassical Ethernet
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 37
Servers typically dual homed to two or more access switches
LAN switches have redundant connections to the next layer
Distribution and Core can be collapsed into a single box
L2/L3 boundary typically deployed in the aggregation layer
Spanning tree or advanced L2 technologies (vPC) used to prevent loops within the L2 boundaryL3 routes are summarized to the core
Services deployed in the L2/L3 boundary of the network (load-balancing, firewall, NAM, etc)
L2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Virtual Port-Channel (VPC)
Outside Data Center “cloud”
STP
STP
The Design RequirementsLAN Design – Access/Aggregation/Core
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 38
Fibre Channel SANTransport and Services are on the same layer in the same devicesWell defined end device relationships (initiators and targets)Does not tolerate packet drop – requires lossless transportOnly north-south traffic, east-west traffic mostly irrelevant
Network designs optimized for Scale and Availability
High availability of network services provided through dual fabric architecture
SAN ‘A’ and SAN ‘B’ : physically separate and redundant fabrics
Strict change isolation - end to end driver certification
Client/Server Relationships are
pre-defined
I(c)
I(c)T(s)
T2
I5
I4I3I2I1
I0
T1T0
Switch Switch
Switch
DNS FSPF
ZoneRSCN DNS
FSPF Zone
RSCN
DNS
Zone
FSPF
RSCN
Fabric topology, services and traffic flows are structured
The Design RequirementsClassical Fibre Channel
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 39
“Edge-Core” Topology
Servers connect to the edge switches
Storage devices connect to one or more core switches
Core switches provide stroage services to one or more edge switches, thus serviceing more servers in the fabric
ISLs have to be designed so that overall fan-in ratio of servers to storage and overall end-to-end oversubscription are maintained
HA achieved in two physically separate, but identical, redundant SAN fabrics
FC
CoreCore CoreCore
The Design RequirementsSAN Design – Two Tier Topology
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 40
“Edge-Core-Edge” Topology
For environments where future growth of the network has the number of storage devices exceeding the number of ports available at the core switch
A set of edge switches dedicated to server connectivity and another set of dedicated for storage devices
Extra edge can also be “services edge”for advanced network services
Core is for transport only, rarely accommodates end nodes
HA achieved with dual fabrics
FC
CoreCore CoreCore
The Design RequirementsSAN Design – Three Tier Topology
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 41
??
?
?
????
?
?
??
Switch Switch
Switch
?
T2
I5
I4I3I2I1
I0
T1T0
Switch Switch
Switch
DNS FSPF
ZoneRSCN DNS
FSPF Zone
RSCNDNS
Zone
FSPFRSCN
Question Do we build a FC network on top of an Ethernet Cloud? Or and Ethernet Network on top of a Fibre Channel Fabric?
Unified Fabric design has to incorporate the super-set of requirements
Network -- Lossless ‘and’ Lossfull Topologies Transport – undefined (any-to-any) ‘and’defined (one-to-one)High Availability – redundant network topology (mesh/full mesh) ‘and’ physically separate redundant fabricsBandwidth – FC fan-in and oversubscription ratios ‘and’ Ethernet oversubscriptionSecurity – FC controls (zoning, port security, …) ‘and’ IP controls (CISF, ACL, …) Manageability and Visibility – Hop by hop visibility for FC ‘and’ the cloud for Ethernet
The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 42
Can’t we just fold down the dotted line??FC
Core
Core
Core
Core
L2
L3
Core
Aggregation
Access
Virtual Port-Channel (VPC)
Virtual Port-Channel (VPC)
Outside Data Center
“cloud”
STP
STP
The Design RequirementsClassical Ethernet + Classical Fibre Channel == ??
Fold
Her
eFo
ld H
ere
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 43
To expand the reach of Unified I/OBlade Server environments (e.g. Nexus 4000 / UCS)Core and backbone links and devices
SAN scalabilityBuild-up the edge, from 20% attach-rate up to 100%Allow LAN and SAN to scale independently
To introduce the support for native FCoE Storage arrays
Preserve SAN design best practicesOversubscription, Fan-in ratios, hop count practices honored
Preserve SAN and LAN management modelsDeterministic management of FC flows through all devices - No opaque LAN “clouds” transporting SAN traffic
The Design RequirementsFCoE Design Objectives
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 44
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsidersationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 45
Host connected over unified wire to first hop access switch
Access switch (Nexus 5000) is the FCFFibre Channel ports on the access switch can be in NPV or Switch mode for native FC traffic
DCBX is used to negotiate the enhanced Ethernet capabilitiesFIP is use to negotiate the FCoE capabilities as well as the host login processFCoE runs from host to access switch FCF – native Ethernet and native FC break off at the access layer
FC
CNACNA
FC FabricFC Fabric
ENodeENode
Target
Target
Ethernet FabricEthernet Fabric
DCB capable Switchacting as an FCF
DCB capable Switchacting as an FCF
Unified WireUnified Wire
Single Hop DesignToday’s Solution
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 46
The first phase of the Unified Fabric evolution design focused on the fabric edge
Unified the LAN Access and the SAN Edge by using FCoE
Consolidated Adapters, Cabling and Switching at the first hop in the fabrics
The Unified Edge supports multiple LAN and SAN topology options
Virtualized Data Center LAN designs
Fibre Channel edge with direct attached initiators and targets
Fibre Channel edge-core and edge-core-edge designs
Fibre Channel NPV edge designs The Unified Edge
Fabric A Fabric BLAN Fabric
FC
FCoEFC
LAN Access/SAN
Edge
Single Hop DesignUnified Wire at the Access
Nexus 5000 FCF-A
Nexus 5000 FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 47
Fibre Channel Drivers
Ethernet Drivers
Operating System
PCIe
Ethernet
Fibre Channel
10GbE
10GbE
Link
Converged Network Adapter (CNA) presents two PCI address to the Operating System (OS)
OS loads two unique sets of drivers and manages two unique application topologies
Server participates in both topologies seperately
Two stacks and thus two views of the same ‘unified wire’
SAN Multi-Pathing provides failover between two fabrics (SAN ‘A’ and SAN ‘B’)
NIC Teaming provides failover within the same fabric (VLAN)
Ethernet Driver bound to
Ethernet NIC PCI address
FC Driver bound to FC
HBA PCI address
Unified Wire shared by both
FC and IP topologies
Nexus Unified Edge supports both FC and IP
topologies
Nexus Edge participates in both distinct FC and IP Core
topologies
Nexus 5000 FCF-A
Nexus 5000 FCF-B
Single Hop DesignThe CNA Point of View
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 48
Fabric A
Direct Attach Topologies
CEE-DCBX
Generation 1 CNA
CIN-DCBX
Generation 2 CNA
Fabric BLAN Fabric
VN
VF Direct attach VN_Port to
VF_Port
In this first phase we were limited to direct attached CNAs at the access
Generation 1 CNA
Utilized Cisco, Intel, Nuova Data Center Bridging Exchange protocol (CIN-DCBX)
Only supports direct attachment of an ‘VN_Port’ to an ‘VF_Port’ over the ‘unified wire’
Generation 2 CNA
Utilizes Converged Enhanced Ethernet Data Center Bridging Exchange protocol (CEE-DCBX)
Utilizes FCoE Initialization Protocol (FIP) as defined by the T.11 FC-BB-5 specification
Supports both direct and multi-hop attachment
Single Hop DesignThe CNA Point of View
Nexus 5000 FCF-A
Nexus 5000 FCF-A
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 49
Fabric A
Direct Attach Topologies
Fabric BLAN Fabric
VN
VF Direct attach VN_Port to
VF_Port
Physical link is brought up (today requires 10GE)
DCBX negotiation – discovers DCB capable devices and negotiates lossless Ethernet capabilities/configs
FIP Process – discovery and negotiation of FCoE devices and characteristics
FCoE VLAN Discovery
FCF Discovery on the specific FCoE VLAN
Fabric Login - builds the logical wire from the end node to the FCF (VN_port to VF_port)
FCoE traffic flows from host to target; LAN traffic flows
Single Hop DesignAttaching an Initiator
Nexus 5000 FCF-A
Nexus 5000 FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 50
VLAN 10,30
VLAN 10,20
Maintaining the two distinctedge/access topologies
Isolated SAN edge switches: SAN ‘A’ and SAN ‘B’
LAN Access switches connected to the same LAN fabric and carrying the same VLANs
Server participates in both topologies but may have different High Availability approaches
SAN Multi-Pathing provides failover between two fabrics
NIC Teaming provides failover to the same fabric (VLAN)
VSAN 2
VLAN 10 VSAN 3
Three concurrent topologies:LAN, SAN ‘A’ and SAN ‘B’
Fabric A Fabric BLAN Fabric
FC
FCoE
Single Hop DesignTwo Distinct Topologies
Nexus 5000 FCF-A
Nexus 5000 FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 51
VLAN 10,30
VLAN 10,20
A VLAN is dedicated for every VSAN in the fabric
FIP is used to discover the FCoE VLAN and signal it to the hosts
Trunking not required on the host driver – all FCoE frames are tagged by the CNA
FCoE VLANs must not be configured on Ethernet links that are not designate for FCoE
Maintains isolated edge switches for SAN ‘A’ and ‘B’ and separate LAN switches for NIC 1 and NIC 2 (standard NIC teaming)
! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20 (config-vlan)# fcoe vsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric BLAN Fabric
Nexus 5000 FCF-A
Nexus 5000 FCF-B
VSAN 3
Single Hop DesignThe FCoE VLAN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 52
VLAN 10,30
VLAN 10,20
In order to maintain the integrity of FC forwarding over FCoE, FCoE VLANs are treated differently than LAN VLANs
No flooding, MAC learning, broadcasts, etc.
The FCoE VLAN must not be configured as a native VLAN
FIP uses native VLAN
Separate FCoE VLANs must be used for FCoE in SAN-A and SAN-B
Unified Wires must be configured as trunk ports and STP edge ports
! VLAN 20 is dedicated for VSAN 2 FCoE traffic(config)# vlan 20 (config-vlan)# fcoe vsan 2
VSAN 2
STP Edge Trunk
Fabric A Fabric BLAN Fabric
Nexus 5000 FCF
Nexus 5000 FCF
VSAN 3
Single Hop DesignThe FCoE VLAN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 53
VLAN 10,30
VLAN 10,20
FCoE Fabric ‘A’ will have a different VLAN topology than FCoE Fabric ‘B’ which are different from the LAN Fabric
PVRST+ allows unique topology per VLAN
MST requires that all switches in the same Region have the same mapping of VLANs to instances
MST does not require that all VLANs be defined in all switches
A separate instance must be used for FCoE VLANs
Recommended: three seperate instances – native Ethernet VLANs, SAN ‘A’ VLANs and SAN ‘B’ VLANs
spanning-tree mst configurationname FCoE-Fabricrevision 5instance 5 vlan 1-19,40-3967,4048-4093instance 10 vlan 20-29instance 15 vlan 30-39
Fabric A Fabric BLAN Fabric
VSAN 3VSAN 2
VLAN 10
Nexus 5000 FCF-A
Nexus 5000 FCF-B
Single Hop DesignThe FCoE VLAN and STP
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 54
Optimal layer 2 LAN design often leverages Multi-Chassis Etherchannel (MCEC)
Nexus utilizes Virtual Port Channel (vPC) to enable MCEC either between switches or to 802.3ad attached servers
MCEC provides network based load sharing and redundancy without introducing layer 2 loops in the topology
MCEC results in diverging LAN and SAN high availability topologies
FC maintains separate SAN ‘A’ and SAN ‘B’ topologiesLAN utilizes a single logical topology Direct Attach vPC Topology
MCEC
vPC Peers
vPC Peer Link
Fabric A Fabric BLAN Fabric
Nexus 5000 FCF-A
Nexus 5000 FCF-B
Single Hop DesignUnified Wires and MCEC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 55
In vPC enabled topologies in order to ensure correct forwarding behavior for SAN traffic specific design and forwarding rules must be followed
With the NX-OS 4.1(3) releases a ‘vfc’ interface can only be associated with a vPC etherchannel with one (1)CNA port attached to each edge switchWhile the port-channel is the same on N5K-1 and N5K-2, the FCoE VLANs are differentvPC configuration works with Gen-2 FIP enabled CNAs ONLYFCoE VLANs are ‘not’ carried on thevPC peer-linkFCoE and FIP ethertypes are ‘not’forwarded over the vPC peer link
Direct Attach vPC Topology
VLAN 10,30
VLAN 10,20STP Edge Trunk
VLAN 10 ONLY HERE!
Fabric A Fabric BLAN Fabric
Nexus 5000 FCF-A
Nexus 5000 FCF-B
Single Hop DesignUnified Wires and MCEC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 56
Nexus 5000 FCF-B
Nexus 5000 FCF-A
VLAN 10,30VLAN 10,20
VLAN 10,20,30
Dual CNA (FC initiator) connected via an Etherchannel to a single edge switch is unsupported
A ‘vfc’ interface can only be bound to a port channel with one local interface
Not consistent with Fibre Channel High Availability design requirements (No isolation of SAN ‘A’ and SAN ‘B’)
If SAN design evolves to a shared physical with only VSAN isolation for SAN ‘A’ and ‘B’ this ‘could’ change (currently this appears to be a big ‘if’)
ISLs between the Nexus 5000 access switches breaks SAN HA requirements
Single homed dual CNA Direct Attach
Topology
Fabric A Fabric BLAN Fabric
Single Hop DesignUnsupported Topologies
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 57© 2009 Cisco Systems, Inc. All rights reserved.Presentation_ID © 2009 Cisco Systems, Inc. All rights reserved.Presentation_ID 5757
32 server facing 10Gig/FCoE ports
T11 standard based FIP/FCoE support on all ports
8 10Gig/FCoE uplink ports for connections to the Nexus 5000
Management and configuration handled by the Nexus 5000
Support for Converged Enhanced Ethernet including PFC
Part of the Cisco Nexus 2000 Fabric Extender family
FEX-2232Remote Line Card of
the Nexus 5000
Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender
Nexus 2232
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 58
Parent Switch: Acts as the combined Supervisor and Switching Fabric for the virtual switch
Fabric Links: Network Interface Forts (NIFs) Extends the Switching Fabric to the remote line card (Connects Nexus 5000 to Fabric Extender)
Host Interfaces (HIF)
Fabric connectivity between Nexus 5000 and Nexus 2000 (FEX) can leverage either pinning or port-channels
Nexus 5000
FEX100 FEX101
dc11-5020-1# show interface fex-fabric Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial ---------------------------------------------------------------100 Eth1/17 Active 1 N2K-C2148T-1GE JAF1311AFLL100 Eth1/18 Active 2 N2K-C2148T-1GE JAF1311AFLL100 Eth1/19 Active 3 N2K-C2148T-1GE JAF1311AFLL100 Eth1/20 Active 4 N2K-C2148T-1GE JAF1311AFLL101 Eth1/21 Active 1 N2K-C2148T-1GE JAF1311AFMT101 Eth1/22 Active 2 N2K-C2148T-1GE JAF1311AFMT
Single Hop DesignIntroduction of 10Gig/FCoE Fabric Extender
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 59
Nexus 223210GE FEX
Nexus 223210GE FEX
SAN BSAN AFEX-2232 extends the reach of 10Gig Ethernet/FCoE to distributed line card (ToR)
• Support for up to 384 10Gig/FCoE attached hosts managed by a single Nexus 5000 (FCS number may vary)
• Nexus 5000 is the FCF or can be in a FCoE pass-through mode (when supported)
• In the presence of FCoE -- Nexus 2232 needs to be single homed to upstream Nexus 5000 (straight through N2K) to ensure SAN ‘A’and SAN ‘B’ isolation
Nexus 223210GE FEX
Nexus 223210GE FEX
Nexus 5000
Nexus 5000
Requires FIP enabled CNAs
TE
TE
Single Hop DesignExtending the Unified Access
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 60
Nexus 223210GE FEX
Nexus 223210GE FEX
SAN BSAN A
Fabric Links Option 1: Single
Homed Port Channel
Fabric Links Option 2: Static
Pinned
Server Option 1: FCoE on
individual links. Ethernet traffic is Active/Standby
• Server Ethernet driver connected to the FEX in NIC Teaming (AFT, TLB) or with vPC (802.3ad)
• FCoE runs over vPC member port with a single link from server to FEX
• FEX single homed to upstream Nexus 5000
• FEX fabric links can be connected to Nexus 5000 with individual links (static pinning) or a port channel
• oversubscribed 4:1
• Consistent with separate LAN Access and SAN Edge Topologies
Server Option 2: FCoE on a vPC
member PC with a single link
Nexus 223210GE FEX
Nexus 223210GE FEX
Requires FIP enabled CNAs
Single Hop DesignExtending the FCoE Edge – Nexus 2232
Nexus 5000 FCF-A Nexus 5000
FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 61
Nexus 223210GE FEX
Nexus 223210GE FEX
SAN BSAN A
Fabric Links: vPC Port Channel
Nexus 2232 can not be configured in a dual homed configuration (vPC between two N5K) when configured to support FCoE attached servers
MCEC Port Channel will not keep SAN ‘A’ and San ‘B’ traffic isolated
Nexus 2000 not supported with dedicated FCoE and dedicated IP upstream fabric links
Nexus 2232 can only currently be connected to the Nexus 5000 when configured to support FCoE attached servers
Nexus 7000 will support Nexus 2000 in Ethernet only mode in CY2010 (support for FCoE on FEX targeted for CY2011 on next generation N7K line cards)
Nexus 223210GE FEX
Nexus 223210GE FEX
Nexus 5000Nexus 5000 Nexus 7000Nexus 7000
Single Hop DesignExtending the FCoE Edge – Nexus 2232
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 62
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 63
What is NPIV?N-Port ID Virtualization (NPIV) provides a means to assign multiple FCIDs to a single N_Port
allows multiple applications to share the same Fiber Channel adapter port
different pWWN allows access control, zoning, and port security to be implemented at the application level
usage applies to applications such as VMWare, MS Virtual Server and Citrix
Application Server FC NPIV Core Switch
Web
File Services
Email I/ON_Port_ID 1
Web I/ON_Port_ID 2
File Services I/ON_Port_ID 3
F_Port
F_PortF_Port
N_PortN_Port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 64
What is NPV?N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a “switch” to act like aserver performing multiple logins through a single physical link
Physical servers connected to the NPV switch login to the upstream NPIV core switch
Physical uplink from NPV switch to FC NPIV core switch does actual “FLOGI”Subsequent logins are converted (proxy) to “FDISC” to login to upstream FC switch
No local switching is done on an FC switch in NPV mode
FC edge switch in NPV mode Does not take up a domain ID
Scalability will be dependent on FC “login” limitation (MDS is ~10K per fabric)
Nexus 5000, MDS 91xx, MDS blade switches, UCS Fabric Interconnect FC NPIV Core Switch
Eth1/1
Eth1/2
Eth1/3
Server1N_Port_ID 1
Server2N_Port_ID 2
Server3N_Port_ID 3
F_Port
N-Port
F-Port
F-PortNP-Port
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 65
FCFFCF
DCB + FIP Snooping
Bridge
DCB + FIP Snooping
Bridge
What design considerations do we have when extending FCoE beyond the Unified Edge?
High Availability for both LAN and SAN
Oversubscription for SAN and LAN
Ethernet Layer 2 and STP design
Where does Unified Wire make sense over Unified Dedicated Wire?
Unified Wire provides for sharing of a single link for both FC and Ethernet traffic
Fabric A Fabric BLAN Fabric
Multi - Hop DesignConsiderations for FCoE Multi-hop
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 66
FCFFCF
DCB + FIP Snooping
Bridge
DCB + FIP Snooping
Bridge
Growing FCoE fabrics is achived by connecting multiple FCoE capable devices together
An FCF contains a domain ID
Fabric A Fabric BLAN Fabric
Multi - Hop DesignThe Need for FCoE Pass Through
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 67
Multi-hop FCoE networks allow for FCoE traffic to extend past the access layer (first hop)
In Multi-hop FCoE the role of a transit Ethernet bridge needs to be evaluated
Avoid Domain ID exhaustionEase management
FIP Snooping is a minimum requirement suggested in FC-BB-5
Ethernet NPV (E-NPV) is a new capability intended to solve a number of design and management challenges
FCFFCF
FCFFCF
SAN BSAN A
DCB Capable Ethernet Switch
DCB Capable Ethernet Switch
DCB Capable Ethernet Switch
DCB Capable Ethernet Switch
VN
VN
VF
VF
Multi - Hop DesignFCoE Pass-through options
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 6868
What is FIP-Snooping?Efficient, automatic configuration of ACLs used to lock down the forwarding path accomplished by snooping FIP packets going from CNA to FCF
Why FIP-Snooping?Security - Protection from MAC Address spoofing of FCoE end devices (ENode)Fibre Channel links are Point-to-PointEthernet bridges can utilize ACLs to provide the equivalent path control (equivalent of point-to-point)
Support for FIP-Snooping?Nexus 4000 (Blade switch for IBM BC H)
Multi - Hop DesignFIP-Snooping
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 69
FCFFCF
FCF MAC 0E.FC.00.DD.EE.FF
FIP Capable Multi-Hop Topology
FIP Snooping
FIP Snooping
ENodeENode
Spoofed MAC 0E.FC.00.DD.EE.FF
ENode MAC 0E.FC.00.07.08.09
FIP Snooping – Nexus 4000Security (Protection from MAC Address spoofing of FCoE end devices “ENode”)Fibre Channel links are Point-to-PointEthernet bridges can utilize ACLs to provide the equivalent path control (equivalent of point-to-point)FIP protocol allows for efficient automatic configuration of the ACLs necessary to lock down the forwarding path (FIP Snooping)
Ethernet-NPV (E-NPV) – FutureIntelligent proxying FIP functions between a CNA and an FCFAdded control to FCF logins/mappings and load-balancing
SANMulti - Hop DesignExtending FCoE with FIP Snooping
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 70
On the control plane (FIP ethertype), an Ethernet NPV bridge improves over a "FIP snooping bridge" by intelligently proxying FIP functions between a CNA and an FCF
- takes control of how a live network will build FCoE connectivity- makes the connectivity very predictable, without the need for an FCF at the next hop from the CNA
On the data plane (FCoE ethertype), an “Ethernet NPV bridge” offers more ways to engineer traffic between CNA-facing ports and FCF-facing ports
An “Ethernet NPV bridge” knows nothing about Fibre Channel, and can’t parse packets with FCoE ethertype
Multi - Hop DesignEthernet NPV Bridge
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 7171
Proxys FIP functions between a CNA and an FCFFCoE VLAN configuration and assignment
FCF Assignment
E-NPV load balance logins from the CNAs evenly across the available FCF uplink ports
E-NPV will take VSAN into account when mapping or ‘pinning’logins from a CNA to an FCF uplink
Operations and management process are in line with today’s SAN-Admin practicesSimilar to NPV in a native Fibre Channel network
**Name subject to change
Multi - Hop DesignEthernet NPV Bridge
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 72
FC
Target
TargetFABRIC A
E_NPV bridgeE_NPV bridge
FCFFCFDomain ID and FC-MAP come from the FCF
FCoE Pass – through deviceAll FCoE Switching is performed at the upstream FCFAddressing is pass out by the upstream FCF
more FCoE connectivity to hosts without
Running into the domain ID issueLess-expensiveConsistent management
E_NPV is the “FIP-Snooping Plus”
FLO
GI
FLO
GI
VN
E_NPV does not consume a domain ID
E_Node MAC Address
VF
VF
VNP
Multi - Hop DesignEthernet NPV - Enode Login Process
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 73
SAN BSAN ANexus 4000 is a Unified Fabric capable Blade Switch
DCB enabled
FIP Snooping Bridge
Dual Topology requirements for FCoE multi-hop
Servers IP connection to the Nexus 4000 is Active/Standby
MCEC is not currently supported from blade server to Nexus 4000
Options 1: Unified Dedicated Wires from Nexus 4000 to Nexus 5000
Options 2: Single Unified Wire Port Channel from Nexus 4000 to Nexus 5000
Option 2: Single Homed Unified
Wire
Mezzanine Converged Network
Adapter
Option 1:Unified
Dedicated Wire
PCIe
Ethernet
Fibre Channel
10GbE
10GbE
Link
Multi - Hop DesignExtending FCoE with FIP Snooping
Nexus 5000 FCF-A
Nexus 5000 FCF-B
Nexus 4000 FIP Snooping
Bridge-B
Nexus 4000 FIP Snooping
Bridge-A
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 74
DCB + FIP Snooping
Bridge
DCB + FIP Snooping
Bridge
Extending FCoE Fibre Channel fabrics beyond direct attach initiators can be achieved in two basic ways
Extend the Unified Edge (Stage 1)
Add DCB enabled Ethernet switches between the VN and VF ports (stretch the ‘link’between the VN_Port and the VF_Port)
Extend Unified Fabric capabilities into the SAN Core
Leverage FCoE wires between Fibre Channel of Ethernet switches (VE_Ports)
Fabric ALAN Fabric
Using FCoE for ISL
between FC Switches
Extending FCoE into a multi-hop Ethernet
‘Access’ Fabric
VN
VF
VE
VE
Fabric B
VE
VE
Multi - Hop DesignExtending FCoE with VE_Ports
Nexus 5000 FCF-A
Nexus 5000 FCF-A
MDS 9000FCF-A
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 75
E-NPVE-NPV
SAN BSAN A
E-NPVE-NPV
Two basic design options are possible when we deploy any FCoE multi-hop configuration
Option 1 – Unified Dedicated Wire
Allows MCEC for IP/Ethernet
Dedicated FCoE links for Storage
Option 2 – Unified Wire
Leverage Server side failover mechanisms for both SAN and LAN
Allows for Unified Wire beyond the Server to first device
Option 2: Single Homed Unified
Wire
Option 1: Dedicated Links and Topologies
Multi - Hop DesignExtending FCoE with Ethernet NPV
Nexus 5000 FCF-A
Nexus 5000 FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 76
FCFFCF FIP and FcoE frames loadshared over
MCEC on a per flow basis
NO SAN ‘A’ and SAN ‘B’ isolation
FIP and FcoE frames loadshared over
MCEC on a per flow basis
NO SAN ‘A’ and SAN ‘B’ isolation
SAN BSAN ASAN and LAN high availability design requirements are not always identical
Optimal layer 2 LAN design may not meet FC high availability and operational design requirements
Features such as vPC & MCEC are not viable and not supported beyond the direct attached server
Server has two stacks and manages two topologies
Layer 2 network has a single topology
L2MP and TRILL provide options to change the design paradigm and come up with potential solutions
DCB Enabled
DCB Enabled
Multi - Hop DesignUnsupported Topologies
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 77
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 78
Where is it efficient to leverage ‘unified wire’, shared links for both SAN and LAN traffic?
At the edge of the fabric the volume of end nodes allows for a greater degree of sharing for LAN and SANIn the core we will not reduce the number of links and will either maintain separate FC or FCoE links to the SAN core and Ethernet links to the LAN core
LAN and SAN HA models are very different (and not fully compatible)
FC and FCoE are prone to HOLB in the network and therefore we are limited in the physical topologies we can build
e.g. 10 x 10G uplinks to LAN aggregation will require 10 x 10G links to a next hop SAN core (with targets attached) – No savings, actually spending more to achieve this direct uplinks to SAN core
Targets are attached to the SAN core (the LAN aggregation and SAN core have different topology functions)
Where is it more beneficial to deploy two cores – SAN and LAN over a “unified core” topology
FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 79
Migration to 10G FCoE in place of 4/8G FC links (Ethernet price per bit economics)
Edge switch running as FCF with VE_ports connecting to FCF on Core switch
Must be careful of Domain ID creeping
FSPF forwarding for FCoE traffic is end-to-end
Hosts will log into the FCF which they are attached to (access FCF)
Storage devices will log into the FCF at the core/storage edge
Maintains HA requirements from both LAN and SAN perspective
VE PortsVE Ports
SAN BSAN A
FCoE Deployment ConsiderationsMigration Strategy for FCoE
Nexus 5000 FCF-B
MDS 9000FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 80
Migration to 10G FCoE in place of 4/8G FC links (Ethernet price per bit economics)
Edge switch running either as FCF in NPV mode or in E-NPV mode with FCF migrating to the SAN Core
Ethernet NPV (E-NPV) is a new construct intended to solve a number of system management problems
Using E_NPV alleviates domain ID issue
HA planning for the SAN side required
Does loosing a core switch mean the loss of a whole fabric?
FCFFCF
FCFFCF
E-NPVE-NPV
SAN BSAN A
FCoE Deployment ConsiderationsMigration Strategy for FCoE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 81
SAN BSAN ADoes passing FCoE traffic through a
larger aggregation point make sense?
Multiple links required to support the HA models
1:1 ratio between access to aggregation and aggregation to SAN core is required
SAN is more vulnerable to HOLB so need to plan for appropriate capacity in any core ISL
When is a direct Edge to Core links for FCoE are more cost effective than adding another hop?
Smaller Edge device more likely to be able to use under-provisioned uplinks
1:1 Ratio of links required unless E-NPV FCoE uplink is over-provisioned
CORECongestion on
Agg-Core links will HOLB all
attached edge devices
FCoE Deployment ConsiderationsShared Aggregation/Core Devices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 82
vv
Different requirements for LAN and SAN network designs
Factors that will influence this use case
Port density
Operational roles and change management
Storage device types
Potentially viable for smaller environments
Larger environments will need dedicated FCoE ‘SAN’ devices providing target ports
Use connections to a SAN
Use a “storage” edge of other FCoE/DCB capable devices
Direct Attach FCoE Targets
CORE
Multiple VDCsFCoE SANLAN AggLAN Core
FCoE Deployment ConsiderationsShared Aggregation/Core Devices
Nexus 5000 FCF-A
Nexus 5000 FCF-B
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 83
FCFFCFFCFFCF
SAN BSAN A
Topology will vary based on scale (single vs multiple tiers) Architecture as defined for product development has a dual core Question - where is the demarc between Unified Wire and Unified FabricAs the topology grows less Unified WireIn all practical terms the ‘edge’ is the unified point for LAN and SAN (not the core/agg)
In smaller topologies where core and edge merge then everything collapses but the essential design elements remain
Dedicated SAN and LAN Core
VLAN 10,30VLAN 10,20
CORE
FCoE Deployment ConsiderationsDedicated Aggregation/Core Devices
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 84
MCEC results in diverging LAN and SAN high availability topologies
FC maintains separate SAN ‘A’ and SAN ‘B’ topologiesLAN utilizes a single logical topology
In vPC enabled topologies in order to ensure correct forwarding behavior for SAN traffic specific design and forwarding rules must be followedWhile the port-channel is the same on N5K-1 and N5K-2, the FCoE VLANs are differentvPC configuration works with Gen-2 FIP enabled CNAs ONLYFCoE VLANs are ‘not’ carried on the vPC peer-linkFCoE and FIP ethertypes are ‘not’forwarded over the vPC peer link
N5K2N5K2
SAN BSAN A
Direct Attach vPC Topology
N5K1N5K1
Virtualized Access Switch - FCoEFCoE - Unified Wires at the Edge
MCEC for IP Only – VLAN 10
vPC Peers
VLAN 10,30
VLAN 10,20
STP Edge Trunk
vPC Peer Link VLAN 10 ONLY
HERE!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 85
Nexus 223210GE FEX
Nexus 223210GE FEX
Virtualized Access Switch - FCoEExtending FCoE – Nexus 2232
SAN BSAN A
Fabric Links Option 1: Single
Homed Port Channel
Fabric Links Option 2: Static
Pinned
Nexus 5000 as FCF or as E-NPV
device
Server Option 1: FCoE on
individual links. Ethernet traffic is Active/Standby
FEX-2232 extends the reach of 10Gig Ethernet/FCoE to distributed line card (ToR)
• Support for up to 384 10Gig/FCoE attached hosts managed by a single Nexus 5000
• Nexus 5000 is the FCF or can be in FIP Snooping + mode (when supported)
• Currently Nexus 2232 needs to be single homed to upstream Nexus 5000 (straight through N2K) to ensure SAN ‘A’ and SAN ‘B’ isolation
• Server Ethernet driver connected to the FEX in NIC Teaming (AFT, TLB) or with vPC (802.3ad)
Server Option 2: FCoE on a vPC
member PC with a single link
Nexus 223210GE FEX
Nexus 223210GE FEX
Nexus 5000Nexus 5000 Nexus 5000Nexus 5000
Requires FIP enabled CNAs
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 86
FCoE Multi-Tier Fabric Design Extending FCoE past the Unified Edge
DCB + FIP Snooping
Bridge
DCB + FIP Snooping
Bridge
Extending FCoE Fibre Channel fabrics beyond direct attach initiators can be achieved in two basic ways
Extend the Unified Edge
Add DCB enabled Ethernet switches between the VN and VF ports (stretch the ‘link’ between the VN_Port and the VF_Port)
Extend Unified Fabric capabilities into the SAN Core
Leverage FCoE wires between Fibre Channel switches (VE_Ports)
What design considerations do we have when extending FCoE beyond the edge?
High Availability
Oversubscription for SAN and LAN
Ethernet layer 2 and STP design
Fabric ALAN Fabric
Using FCoE for ISL
between FC Switches
Extending FCoE into a multi-hop Ethernet
‘Access’ Fabric
VN
VF
VE
VE
FCFFCF
Fabric B
FCFSwitch Mode
FCFSwitch Mode
Please see session “BRKSAN-2047 - Storage and the Unified Fabric” for more information on FCoE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 87
FCFFCFFCFFCF
Unified Fabric and HA DesignExtending FCoE past the Edge – Current State
SAN BSAN ANexus 4000 is a Unified Fabric capable Blade Switch
DCB enabled
FIP Snooping Bridge
Dual Topology requirements for FCoE multi-hop
Servers IP connection to the Nexus 4000 is Active/Standby
vPC is not currently supported from blade server to Nexus 4000
Separate dedicated FCoE links run from Nexus 4000 to Nexus 5000
Single homed Port Channel supported for N4K to N5K FCoE uplinks
Option 2: Single Homed Unified
Wire
Mezzanine Converged Network
Adapter
Nexus 4000DCB Blade
Switch
Nexus 4000DCB Blade
SwitchNexus 4000DCB Blade
Switch
Nexus 4000DCB Blade
Switch
FIP SnoopingFIP SnoopingFIP SnoopingFIP Snooping
Option 1: Dedicated Links and Topologies
PCIe
Ethernet
Fibre Channel
10GbE
10GbE
Link
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 88
Servers, FCoE attached StorageServers, FCoE
attached Storage
Virtualized Access Switch - FCoELarger Fabric Multi-Hop Topologies
Multi-hop edge/core/edge topologyCore SAN switches supporting FCoE
N7K with DCB/FCoE line cardsMDS with FCoE line cards (Sup2A)
Edge FC switches supporting either N5K - E-NPV with FCoE uplinks to the FCoE enabled core (VNP to VF)N5K or N7K - FC Switch with FCoE ISL uplinks (VE to VE)
Scaling of the fabric (FLOGI, …) will most likely drive the selection of which mode to deploy
N7K or MDS FCoE enabled Fabric
Switches
FC Attached Storage
FC Attached Storage
ServersServers
VE
Edge FCFSwitch ModeEdge FCF
Switch Mode
VE
Edge Switch in E-NPV
Mode
Edge Switch in E-NPV
Mode
VF
VNP VE
VE
Please see session “BRKSAN-2047 - Storage and the Unified Fabric” for more information on FCoE
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 89© 2006 Cisco Systems, Inc. All rights reserved. Cisco Confidential14497_04_2008_c1
Evolution of the Data Centre ArchitectureVirtualized Access Layer
The Evolving Data Centre AccessWhere is the edge?
Phase 1 – Physical Virtualization (Nexus2000)
Decoupling Layer 1 and Layer 2vPC – Redundancy in the AccessDesign Considerations
Phase 2 – Hypervisor Virtualized Switching (Nexus 1000v)
Components of Nexus 1000vDesign Considerations
Phase 3 – Unifying the Fabric (Nexus & FCoE)Integrating the Unified Compute FabricThe Virtualized Access Layer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 90
Evolution of the DC Access ArchitectureUCS 6100 – End Host Mode
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
UCS Fabric Interconnect supports two modes of operation
Switch ModeEnd Host Mode <- Recommended
In End Host Mode the Fabric Interconnects don’t function like regular LAN switches
They don’t forward frames based on destination MAC addresses They don’t run spanning-tree!They don’t learn MAC addresses from external LAN switchesForwarding is based on server-to-uplink pinningActs a true Layer 2 stub device and never reflects traffic back upstreamLoop-free topology without STP
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
Spanning Tree - Rapid
PVST+ or MST
Spanning Tree Edge Ports
Access/Edge Layer
Aggregation/Core
6100 - EHM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 91
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
FCFFCFFCFFCF
FCoE Multi-TierPotential Migration for UCS
SAN BSAN A
FCoE design will follow the same evolution Should 6100 become a fabric switch
Technically viable but does it add too much operational complexity (e.g. SW certification cycles, operational change management, …)
Migration from NPV to E-NPV mode Again key question is where is the demarcation between ‘Unified Wire’and Unified FabricNote: 6100 Fabric HA required due to lack of vPC uplinks when running Unified Wire from 6100 to next hop
Dedicated SAN and LAN Core
VLAN 10,30VLAN 10,20
CORE
E-NPVE-NPV
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 92
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
Switch ModeSwitch Mode
FCoE Multi-TierPotential Migration for UCS
SAN B
Option 1 –FC/FCoE targets
Smaller scaleWhat do we collapse to?What is the migration to FCoE targetsAgain key question is where is the demarc between ‘Unified Wire’and Unified FabricIf we direct attach targets do we need to support 6100 as a switch in a multi-switch fabric?Is Option 1 viable only for standalone implementations?
CORE
E-NPVE-NPV
FC Switch ModeFC Switch Mode
Option 2 – E-NPV
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 93
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsFuture??Questions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 94
E-NPVE-NPVFCFFCF
L2MP provides the potential to change the design of the Data Center FabricL2MP is based on an edge/spine topologyMultiple forwarding topologies can be supported
Edge/Spine architecturesUnique topologies for different forwarding groups (VLANs) are possible
High Availability design rules change, ECMP and ‘routed’ designTraffic capacity planning can change as well due to new load sharing capabilitiesMay still find design value in multiple cores and dedicated links
Multi Topology
L2MP Supporting SAN A, SAN
B and Scalable
LAN L2MP Edge
L2MP Spine
L2MP Edge
FCoE in the FutureWhat about L2MP?
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 95
FCFFCFFCFFCF
E-NPVE-NPV
SAN BSAN ADoes L2MP with multi-topology support change this?
Still have two oversubscription models and traffic patterns to analyze
Multi-topology L2MP fixes the MCEC problem by allowing per VLAN topologies (unique SAN ‘A’ & ‘B’ and LAN)
Traffic Capacity Planning becomes far more complex as measured load varies amongst ECMP links
Unified Dedicated Wires are still the recommendation: Provides better traffic isolation than VSANs/VLANs: different ports, different protocols!
E-NPVE-NPV
These 2 links are more prone to congestion
VLAN 10,30VLAN 10,20
VLAN 10 ONLY HERE!
FCoE in the FutureL2MP -- Unified Wire vs Unified Dedicated Wire
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 96
NAS and FCoE Attached Storage
NAS and FCoE Attached Storage
FCoE Multi-Tier Fabric DesignHow will Future LAN capabilities change the HA design options
L2MP is based on an edge/spine/edge topologyLarger SAN topologies utilize an edge/core/edge designsHigh Availability design rules change, ECMP and ‘routed’ designWith L2MP multiple forwarding topologies can be supported
Traffic capacity planning can change as well due to new load sharing capabilitiesDedicated or shared links for Storage, IP, vMotion, backup, ….
Common edge/core/edge for both NAS and FC/FCoE Storage
Consistent low latency from any server to any storage (cut-thru)L2MP provides potential for very large capacity designs
Multi Topology L2MP Supporting SAN A, SAN
B and LAN
Edge
FC Attached Storage
FC Attached Storage
Edge
Servers Leveraging Bock and File Based
Storage
Servers Leveraging Bock and File Based
Storage
Core
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 97
Redefinition of the Fabric Services and associated design requirements
We discussed it in FC-BB-5, it was too much work and we delayed, it is a good idea (it may happen in FC-BB-6 - that is probably 2 years away)
Prior to that operational changes may allow a different design approach that will meet both HA requirements (VSAN/VLAN isolation, L2MP and LAN design evolution, changing SAN support matrix, …)
Initiators and targets are connected directly to the Ethernet cloud
Inside the cloud the data plane is pure Ethernet
Zoning is enforced at the edge of the cloud
EthernetCloud
FC services can run on any server
FCoE in the FutureWhat about FC-BB-6?
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 98
Why Nexus Edge LayerConsistent Architecture for Heterogeneous Environments
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
slot 1slot 2slot 3slot 4slot 5slot 6slot 7slot 8
blade1blade2blade3blade4blade5blade6blade7blade8
Spanning Tree Edge PortsMulti-Hop FCoE
VMVMVM
VMVMVM
NEXUS 1000v VM
VMVM
VMVMVM
NEXUS 1000v
VMVMVM
VMVMVM
NEXUS 1000v
VMVMVM
VMVMVM
NEXUS 1000v
VMVMVM
VMVMVM
NEXUS 1000v
VMVMVM
VMVMVM
NEXUS 1000v
VMVMVM
VMVMVM
NEXUS 1000v
Cor
e/A
ggre
gatio
n La
yer
Virt
ualiz
ed E
dge/
Acc
ess
Laye
r
1G and 10GE Blade Servers Pass-Thru
HP/IBM/Dell
N4K - DCB Blade Switch
IBM/Dell
1G and 10GE Rack Mount
ServersUCS Compute
Pod10GE Blade
(HP) UCS Compute Pod
Cor
e
10G DCB FCoE
SAN ‘A’ SAN ‘B’
Uniform Network Fabric supporting
Heterogeneous Compute Environments
Unified Access Layer
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 99
AgendaWhy are we here?Background Information
DCB StandardFCoE Protocol InformationFCoE Building Blocks
Design RequirementsClassical Ethernet + Classical Fibre Channel = ??
Single Hop DesignsMulti-Hop DesignsFCoE Deployment ConsiderationsQuestions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 100
Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily.
Receive 20 Cisco Preferred Access points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live and Networkers Virtual account for access to all session materials, communities, and on-demand and live activities throughout the year. Activate your account at any internet station or visit www.ciscolivevirtual.com.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 101
Check the Recommended Reading brochure for suggested products available at the Cisco Store
Enter to Win a 12-Book Libraryof Your Choice from Cisco Press
Visit the Cisco Store in the World of Solutions, where you will be asked to enter this Session ID code
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicPresentation_ID 102