Unified Fabric aka FCOE

© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential (please do not distribute) 1

Unified Fabricaka FCOE

Dave Gibson Senior Systems Engineer Cisco Systems

2© 2008 Cisco Systems, Inc. All rights reserved. Cisco ConfidentialC97-485980-00

Legal Disclaimer

Many of the products and features described herein remain in varying stages of development and will be offered on a when-and-if-available basis. This roadmap is subject to change at the sole discretion of Cisco, and Cisco will have no liability for delay in the delivery or failure to deliver any of the products or features set forth in this document.


Agenda

The Evolution of the Data Center

Introduction to FCoE

Standards Defined

Nexus and the Unified Fabric

Nexus 5000


The Evolution of theData Center


Data Center Access Layer Trends

Multi-Core CPU architectures allowing bigger and multiple workloads on the same machine

Server virtualization driving the need for more I/O bandwidth per server

Growing need for network storage driving the demand for higher network bandwidth to the server

Increasing adoption of Blades in data centers.

10G LOM on server Motherboard


Next-Gen Switch Design Goals

•Enable Virtualization•Address increase in server processing power

•End-to-end data center architecture•Operational consistency across platforms

•Consolidate LAN & SAN infrasctucture•Standards based solution•Reduce total cost of ownership

•Build with superior performance in mind•Support low latency applications (e.g. HPC, clustered app’s)

•Scale to 40G and 100G in future•Increase feature velocity


Cisco Nexus Family

Complete data center class switching portfolio

Consistent data center operating system across all platforms

Infrastructure scalability, transport flexibility and operational manageability

1K1K1K1KCisco Nexus 1000V

Cisco Nexus 1000V

2008

2008

x86x86

NX-OS Data Center Operating System

Data Center Network Manager

Nexus 2000

(Fabric Extender)

Nexus 2000

(Fabric Extender)

Nexus 5000

(Fixed Config Switch)

Nexus 5000

(Fixed Config Switch)

Nexus 7000

(Modular Switch Platform)

Nexus 7000

(Modular Switch Platform)

Nexus 1000V

(Virtual Switch)

Nexus 1000V

(Virtual Switch) Nexus 4000

(Blade Switch)

Nexus 4000

(Blade Switch)


Parallel LAN/SAN Infrastructure

Inefficient use of Network Infrastructure

5+ connections per server – higher adapter and cabling costs

• Adds downstream port costs; cap-ex and op-ex

• Each connection adds additional points of failure in the fabric

Multiple switching modules in Blade Chassis

Longer lead time for server provisioning

Multiple fault domains – complex diagnostics

Management complexity

Before I/O Consolidation

Ethernet FC

LAN SAN BSAN A

Blade Chassis with I/O ModulesBlade Chassis

with I/O ModulesServer with

NICs and HBAsServer with

NICs and HBAs


I/O Consolidation

Data Center Bridging and FCoE

Ethernet Fibre Channel (FC)

Nexus 5000Nexus 5000

LAN SAN BSAN A

Reduction of server adapters

Simplification of access layer and cabling

Gateway free implementation – fits in installed base of existing LAN and SAN

Lower Total Cost of Ownership

Fewer Cables

Investment Protection (LANs and SANs)

Consistent Operational ModelBlade Chassis with Nexus 4000

Blade Chassis with Nexus 4000

Server with CNAsServer with CNAs

Nexus 5000Nexus 5000


Adapter Evolution:Consolidation Network Adapter


Operating System View


Evolution of 10G Ethernet Physical MediaRole of Transport in Enabling these Technologies!

100Mb 1Gb 10Gb

UTP Cat 5 UTP Cat 5SFP Fiber

10Mb

UTP Cat 3

Mid 1980’s Mid 1990’s Early 2000’s Late 2000’s

X2SFP+ Cu (BER better than 10 )

SFP+ FiberCat 6/7

-18

Technology Cable Distance Power(each side)

Transceiver Latency (link)

SFP+ CUCopper

Twinax 7m ~0.1W ~0.1μs

SFP+ USRUltra short reach

MM OM2MM OM3

10m100m

1W ~0.1μs

SFP+ SRShort reach

MM 62.5 μmMM 50 μm

26-33m66-300m

1W ~0.1μs

SFP+ LRLong range

SMF G.652 10km 0.5W

10GBASE-T Cat6Cat6a/7Cat6a/7

55m100m30m

~8W~8W~4W

2.5μs2.5μs1.5μs


Introduction to FCoE


What is Fibre Channel over Ethernet?

From a Fibre Channel standpoint it’sFC connectivity over a new type of cable called… an Ethernet cloud

From an Ethernet standpoints it’sYet another ULP (Upper Layer Protocol) to be transported

FCoE is an extension of Fibre Channelonto a Lossless Ethernet fabric


Unified Fabric OverviewFibre Channel over Ethernet (FCoE)

04/22/23 16

• Mapping of FC Frames over Ethernet

• Enables FC to Run on a Lossless Ethernet Network

• Mapping of FC Frames over Ethernet

• Enables FC to Run on a Lossless Ethernet Network

• Fewer Cables• Both block I/O & Ethernet

traffic co-exist on same cable

• Fewer adapters needed• Overall less power• Interoperates with

existing SAN’s• Management SAN’s

remains constant

• No Gateway

• Fewer Cables• Both block I/O & Ethernet

traffic co-exist on same cable

• Fewer adapters needed• Overall less power• Interoperates with

existing SAN’s• Management SAN’s

remains constant

• No Gateway

FCoEFCoE BenefitsBenefits

Fibre Channel Traffic

Ethernet


FCoE EnablersE

ther

net

Hea

der

FC

oEH

eade

r

FC

Hea

der

FC Payload

CR

CE

OF

FC

S

Same as a physical FC frame

Control information: version, ordered sets (SOF, EOF)

Normal ethernet frame, ethertype = FCoE

10Gbps Ethernet

Lossless EthernetMatches the lossless behavior guaranteed in FC by B2B credits

Ethernet jumbo frames


Unified I/OFibre Channel over Ethernet (FCoE)

FCoE is managed like FC at initiator, target, and switch level

Same Operational Model Same Operational Model

Same Techniques ofTraffic ManagementSame Techniques ofTraffic Management

Same Managementand Security ModelsSame Managementand Security Models

Easy to UnderstandEasy to Understand

Completely based on the FC model

Same host-to-switch and switch-to-switch behavior as FC

e.g. in order delivery, FSPF load balancing

WWNs, FC-IDs, hard/soft zoning, DNS, RSCN

FCoE isFibre ChannelFCoE isFibre Channel


Ethernet

IP

Network Stack Comparison

TCP

iSCSI

FCIP

FCoE

FCP

FC

IP

TCP

FCPFCP

Ethernet Ethernet

SCSI

FC FC

SCSI SCSI SCSI SCSI

SCSI iSCSI FCIP FCoE FC

PHYSICAL WIRE

Less Overheadthan FCIP, iSCSI


Destination MAC Address

Source MAC Address

(IEEE 802.1Q Tag)

ET = FCoE Ver Reserved

Reserved

Reserved

Reserved SOF

Encapsulated FC Frame (with CRC)

EOF Reserved

FCS

FCoE Frame Format


FCoE StandardsDefined


A larger picture

IEEE 802•Evolution of Ethernet (10 GE, 40 GE, 100 GE, copper and fiber)

•Evolution of switching (Priority Flow Control, Enhanced Transmission, Congestion Management, Data Center Bridging eXchange)

INCITS/T11•Evolution of Fibre Channel (FC-BB-5)

•FCoE (Fibre Channel over Ethernet)

IETF•Layer 2 Multi-Path

•TRILL (Transparent Interconnection of Lots of Links)


DCE versus DCB

DCE is an old Cisco marketing term

Cisco is now using the term DCBThe term IEEE uses

Cisco supports the DCB standard activityBy implementing products that are DCB compliant

CIN-DCBX – Cisco, Intel, Nuova Data Center Bridging Exchange protocol, pre-standard

CEE-DCBX – Converged Enhanced Ethernet Data Center Bridging Exchange protocol, which is standards base


What’s FC-BB-5

FC-BB-5 covers the majority of the FC features, using Ethernet

From an Ethernet perspective, FC-BB-5 isEthernet control plane referred to as FIP (Fibre Channel over Ethernet Initiation Protocol)

discover and build virtual paths between end points

Ethernet data plane providing FCoE forwarding

including both FC control plane and FC data plane (FCF)


FC-BB-6

It is an active working group of T11 that will discuss the future of FCoE or FCoE v2.0

It is just started, 18 months to have a standardApproximate target spring 2011

You can track it on http://www.fcoe.com

http://www.fcoe.com/


The two protocols have:• Two different Ethertypes• Two different frame formats

Protocol Organization

FCoE itself … Is the data plane protocol

It is used to carry most of the FC frames and all the SCSI traffic

FIP (FCoE initiation protocol)

It is the control plane protocol

It is used to discover the FC entities connected to an Ethernet cloud

It is used to login to and logout from the FC fabric


What’s NOT FC-BB-5

FC-BB-5 doesn’t deal with how lossless is realized in Ethernet

no Priority Flow Control, Bandwidth Management, etc.

FC-BB-5 doesn’t deal with management functions


Feature / Standard Standards Status

IEEE 802.1Qbb

Priority Flow Control (PFC)

Enable multiple traffic types to share a common Ethernet link without interfering with each other

PAR approved

1.0 published

IEEE 802.1Qaz

Bandwidth Management (ETS)

Enable consistent management of QoS at the network level by providing consistent scheduling

PAR approved

1.0 published

Data Center Bridging Exchange Protocol (DCBX)

Management protocol for enhanced Ethernet capabilities

This is part of IEEE 802.1Qaz

IEEE DCB standards statusDCB technologies allow Ethernet to be lossless and to manage bandwidth allocation of SAN and LAN flows


04/22/23 Nuova Systems Inc. 30

Data Center Ethernet: PFC & Bandwidth Management

CoS based Bandwidth ManagementCoS based Bandwidth Management

• Enables Intelligent sharing of bandwidth between traffic classes control of bandwidth• 802.1Qaz Enhanced Transmission

10 GE Realized Traffic Utilization

3G/s HPC Traffic3G/s

2G/s

3G/sStorage Traffic3G/s

3G/s

LAN Traffic4G/s

5G/s3G/s

t1 t2 t3

Offered Traffic

t1 t2 t3

3G/s 3G/s

3G/s 3G/s 3G/s

2G/s

3G/s 4G/s 6G/s

Priority Flow ControlPriority Flow Control

• Enables lossless behavior for each class of service• PAUSE sent per virtual lane when buffers limit exceeded

Transmit QueuesEthernet Link

Receive Buffers

ZeroZero ZeroZero

OneOne OneOne

TwoTwo TwoTwo

FiveFive FiveFive

FourFour FourFour

SixSix SixSix

SevenSeven SevenSeven

ThreeThree ThreeThreeSTOP PAUSE EightVirtualLanes


DCBX Overview

Auto-negotiation of capability and configuration

Priority Flow Control capability and associated CoS values

Allows one link peer to push config to other link peer

Link partners can choose supported features and willingness to accept

Discovers FCoE Capabilities

Responsible for Logical Link Up/Down signaling of Ethernet and FC

DCBX negotiation failures will result in: vfc not coming up

Per-priority-pause not enabled on CoS values with PFC configuration

http://download.intel.com/technology/eedc/dcb_cep_spec.pdfhttp://www.ieee802.org/1/files/public/docs2008/

http://download.intel.com/technology/eedc/dcb_cep_spec.pdf

http://www.ieee802.org/1/files/public/docs2008/

http://www.ieee802.org/1/files/public/docs2008/


FCoE control plane


FIP: FCoE Initialization Protocol

FCoE VLAN discovery

Automatic discovery of FCoE VLANs

Device discovery

ENodes discover VF_Port capable FCF-MACs for VN_Port to VF_Port Virtual Links

VE_Port capable FCF-MACs discover other VE_Port capable FCF-MACs for VE_Port to VE_Port Virtual Links

The protocol verifies the Lossless Ethernet network supports the required Max FCoE Size

Virtual Link instantiation

Builds on the existing Fibre Channel Login process, adding the Negotiation of the MAC address to use

Fabric Provided MAC Address (FPMA), or

Server Provided MAC Address (SPMA)

Virtual Links maintenance

Timer based


Fabric Provided MAC Addresses

MAC address assigned for each FC_ID:

Consistent with the Fibre Channel model

Multiple FC-MAPs may be supported

One per SAN

No table needed for Encapsulation

Multiple MACs may be needed for NPIV

48 bits

FC-MAP(0E-FC-00)

FC-ID7.8.9

24 bits

24 bits

FC-MAP(0E-FC-00)

FC-ID7.8.9

Cisco Nexus 5000 uses FPMA

Server Provided MAC Addresses

Adapter uses burned-in or configured MAC address:

Consistent with the Ethernet model

FCF needs a table to map between MAC addresses and FC_IDs

48 bits

Burned in or Configured

MACAddress


Initial Login Flow ladder

ENode FCoE Switch

VLANDiscovery

FLOGI/FDISC FLOGI/FDISC Accept

FC CommandFC Commandresponses

FIP:FCoEInitialization Protocol

FCOEProtocol

VLANDiscovery

FCFDiscovery

Solicitation FCFDiscoveryAdvertisement


FCoE data plane


ENode: Simplified Model

ENode (FCoE Node): a Fibre Channel HBA implemented within an Ethernet NIC aka CNA (Converged Network Adapter)

FCoE LEP : The data forwarding component that handles FC frame encapsulation/decapsulation

FCoE Controller is the functional entity that performs the FIP and instantiates VN_Port/FCoE_LEP pairs.

Enetport

Enetport

FC Node

FCoE_LEP FCoE_LEP

FCoE_Controller FCoE_Controller


FCoE Switch: Simplified Model

FCF (Fibre Channel Forwarder), the forwarding entity inside an FCoE switch

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethport

Ethernet Bridge

FCport

FCport

FCport

FCport

FCF

FCoE_LEP

FCoE Switch

FCoE_Controller


FCoE Network Topology


FCoE: Initial Deployment

SAN A SAN B10GE

Backbone

10GE

4/8 Gbps FC

VF_Ports

VN_Ports

Nexus 5000 (FCF)


FCoE: Adding Blade Servers

SAN A SAN B

10GEBackbone

10GE

4/8 Gbps FC

VF_Ports

VN_Ports


FCoE: Adding Native FCoE Storage

SAN B

10GE

4/8 Gbps FC

VF_Ports

VN_Ports

SAN A

VN_Ports

10GEBackbone


FCoE: Adding VE_ports

10GEBackbone

10GE

4/8 Gbps FC

VF_Ports

VE_Ports

SAN ASAN B

VN_Ports


Nexus Topologies


The Unified Data Center Architecture

L2

L3

VM

VM

VM

VM

VMVM

VM VM

VM VM

A

Aggregation: Typical L3/L2 boundary. DC aggregation point for uplink and DC services offering key features: VPC, VDC, 10GE density and 1st point of migration to 40GE and 100GE

Access: Classic network layer providing non-blocking paths to servers & IP storage devices through VPC. It leverages Distributed Access Fabric Model (DAF) to centralize config & mgmt and ease horizontal cabling demands related to 1G and 10GE server environments

Virtual Access: A virtual layer of network intelligence offering access layer-like controls to extend traditional visibility, flexibility and mgmt into virtual server environments. Virtual network switches bring access layer switching capabilities to virtual servers without burden of topology control plane protocols. Virtual Adapters provide granular control over virtual and physical server IO resources

L3

PODPOD

Core: L3 boundary to the DC network. Functional point for route summarization, the injection of default routes and termination of segmented virtual transport networks

L2

vL2

Rack 1 Rack 2 Rack 3

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VMVMVM

VM VM

VMVM

VMVM

VM VM

VM VM

VMVM

VMVM

VM VM

VM VM

VMVM

B

Rack 1 Rack x

NEXUS 2000 NEXUS 1000v

NEXUS 7000 - VPC

NEXUS 5000

NEXUS 7000 - VPC

NEXUS 7000

Service Appliances

Catalyst 6500

Service Modules

Unified Compute System


CBS 3100| MDS 9100 Blade

Catalyst 49xxRack

Nexus 7000 End-of-Row

Nexus 5K|2KTop of Rack

1GbE,10GbE Server Access

MDS 9500Storage

Catalyst 6500End-of-Row

Storage

IP+MPLS WAN Agg Router

WAN

MDS 9500StorageServices

Gigabit Ethernet

10 Gigabit Ethernet

4, 8Gb Fibre Channel

UCS bladeorNexus 4K

10 Gigabit FCoE/DCE

Catalyst 650010GbE VSS AggDC Services

Nexus 700010GbE Core

Fitting the pieces together…

Nexus 700010GbE AggCatalyst 6500DC Services

DC AggregationSAN A/B

1GbE Server Access

Nexus 1000V VN-Link

DC Access

DC Core


Cisco Nexus 5000 Architecture


Hardware Architecture


Virtual Output Queues

CrossbarFabric

Ingress Port

Packet Buffer

Egress Port

Packet BufferPacket Buffer

SchedulerEgress Queue

Q1

Q8

Q1

Q8

Egress Queue

VOQ 1

VOQ N

Egress Port

Packet BufferPacket Buffer

Egress Queue

Q1

Q8

VOQ 1

VOQ N

Q1

Q8

Q1

Q8


Policy Enforcement

Frames evaluated by multistage engine searches occur in parallel results, and are evaluated in pipeline diagnostics, and control plane tap pipelines.

Multipath Expansion

VLAN MembershipCheck

Interface, VLAN, and MAC Binding

MAC and L3 Binding(IP & Fibre Channel)

Fibre Channel Zone Membership Check

Port ACLs

VLAN ACLs (ingress)

QoS ACLs (ingress)

Role Based ACLs (egress)

Control P

lane Redirect/S

nooping

Sw

itch Port A

nalyzer (SP

AN

) and Diagnostic S

ampling

failpass

pass fail

pass

pass

permit

permit

permit

permit

fail

fail

deny

deny

deny

policerdrop

To SupTo SPANsession

Parsed Packet

Collect Interface Configuration and

State

Virtual Interface Table (512)

Vlan Translation Table (4K)

Vlan Translation Table (4K)

Vlan State Table (1K)

Determine Destination

(ingress only)

Fibre Channel Switch Table (4K)

EthernetLearning

Policy EnforcementACL Search Engine

(2K)

MultipathExpansion

(ingress only)

Zoning Table(2K)

Zoning Table(2K)

RBACL Label Table(2K)

RBACL Label Table(2K)

Binding Table(2K)

Binding Table(2K)

Fibre Channel Multipath Table (1K)

PortChannelTable(16)

Multicast Vector Table (4K)Station Table

(16K)

Station Table(16K)

Editing Instructions &Virtual Output Queue List


Default QoS Configuration

switch1# sh policy-map

Type qos policy-maps ==================== policy-map type qos default-in-policy class type qos class-fcoe set qos-group 1 class type qos class-default set qos-group 0

Type queuing policy-maps ======================== policy-map type queuing default-in-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50 policy-map type queuing default-out-policy class type queuing class-fcoe bandwidth percent 50 class type queuing class-default bandwidth percent 50

Type network-qos policy-maps =============================== policy-map type network-qos default-uf-policy class type network-qos class-fcoe pause no-drop mtu 2240 class type network-qos class-default mtu 1538

switch2# show class-map

Type qos class-maps ===================

class-map type qos class-fcoe match cos 3

class-map type qos class-default match any

Type queuing class-maps =======================

class-map type queuing class-fcoe match qos-group 1

class-map type queuing class-default match qos-group 0

Type network-qos class-maps ==============================

class-map type network-qos class-fcoe match qos-group 1

class-map type network-qos class-default match qos-group 0

• Qos is always on.

• Four default class of services defined when system boots up

• Two for control traffic. One for FCoE traffic and another one for Ethernet traffic

• Match CoS 3 for class-fcoe.

• Class-fcoe is no-drop with MTU 2240.

• Match any for class-default

• Class-fcoe and class-default get 50% of guaranteed bandwidth by default


Nexus 5000 Software Features Set

Layer 2

802.1w (Rapid Spanning Tree), 802.1s (Multiple Spanning Tree), RPVST+, Root Guard, Uplink Guard, Bridge Assurance, PortFast, CDP, PVLANs, UDLD, LACP, IGMP Snooping, 802.1Q trunks, Port-Channel, SVI, SPAN, Jumbo Frames, NTP, Link State Tracking (LST)

Management/Security

Radius, Tacacs+, AAA, CallHome, SSHv1/V2, telnet, IPv4 & IPv6 mgmt, SNMP MiBs, Traps, EthAnalyzer (wireshark), RBAC, DCNM, RME support via Cisco Works, syslog, coredump, RMON, first-setup script, accounting log, checkpoint and configuration rollback

ACL/QOS

PACLs, VACLs, Session based ACLs, ACL based QOS, egress Bandwidth Limiting, 802.1p priority, strict priority scheduling, WRED, Tail Drop, Storm Control (broadcast, multicast), Egress Shaper

FCOE FIP Snooping Bridge, DCBXP, PFC (Priority Flow Control), 8 Virtual Lanes, ETS (Enhance Transmission Selection)


Nexus 5000 and FC Connectivity


Switch Mode

Nexus 5000 FC module can be ISL’ed to another FC switch (E_port) Zoning, DPVM, etc. are enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name

server run on Nexus 5000 Require a domain ID for every VSAN Interop mode considerations when connecting to non-Cisco FC

switches

Note: Nexus 5000 supports direct connectivity to FC initiator (server HBAs) and targets (storage arrays)


N-Port Virtualization (NPV) mode

Nexus 5000 FC module can work in NPV modeServer-facing ports are regular F portsUplinks toward SAN core fabric are NP ports

Nexus 5000 switches assign FCIDs to attached devices First byte in FCID received from core SAN switch

One VSAN per uplink on Nexus 5000 (will change in future)No trunking or channelling of NP ports

Zoning, DPVM, etc. are not enforced on the Nexus 5000 Domain manager, FSPF, zone server, fabric login server, name server

They do not run on Nexus 5000

No local switchingAll traffic routed via the core SAN switches


N-Port Virtualization (NPV): An Overview

Nexus 5000 to SAN Fabric A & BAssign FCIDs to servers – no domain to configure!

F-port

NPV-Core Switch (MDS or 3rd party switch with NPIV support)

NP-port

FC

VSAN 5

Can have multipleuplinks – one VSAN per uplinkTwo uplinks can be in the same VSANNo port channel or trunking

HostHost

Host

F-ports

N-ports

VSAN 10

Servers log in (FLOGI) locally


Working withNexus 2148(Optional)


Nexus 2000 Fabric ExtenderVirtual Chassis


Nexus 2000 Fabric Extender1GE Connectivity


Fabric ExtenderUplink Modes

Fabric Extender associates (pins) a server side (1GE) port with an uplink (10GE) port

Server ports are either individually pinned to specific uplinks (static pinning) or all interfaces pinned to a single logical port channel

Behaviour on FEX uplink failure depends on the configuration

Static Pinning – Server ports pinned to the specific uplink are brought down with the failure of the pinned uplink

Port Channel – Server traffic is shifted to remaining uplinks based on port channel hash

Static Pinning

Port Channel

Server Interface goes down

Server Interface stays active

Date post:	01-Feb-2016
Category:	Documents
Upload:	tovah
View:	36 times
Download:	2 times

Unified Fabric aka FCOE

Documents