FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

1 © Copyright 2012 EMC Corporation. All rights reserved.

FC/FCoE Topologies, Protocols and Limitations

Erik Smith Consulting Technologist – Connectrix BU


Abstract

An in-depth discussion of FC and FCoE protocols focusing on

– topologies currently supported

– topologies under development

– known issues

Review of current EMC SAN best practices and reasons behind them


Goals

Describe technical benefits and limitations of both FC and FCoE

Describe currently supported FC and FCoE topologies and EMC-recommended best practices

Discuss known limitations with FC and FCoE


Agenda

FC or Ethernet

FC vs. FCoE vs. iSCSI

Supported Topologies

Best Practices

Futures


FC or Ethernet

Today


FC or Ethernet

Fibre Channel

Today

16G

32G ??


FC or Ethernet

Fibre Channel

Ethernet

Today

16G

32G ??

10G 40/100G

??


FC or Ethernet

If you’re asking yourself this question, you’re not alone

Fibre Channel

Ethernet

Today

16G

32G ??

10G 40/100G

??


FC or Ethernet You don’t really need to decide right now

Physical Fibre Channel

Physical Ethernet

Today Time Line




Physical Ethernet

Host Storage Network

Today Time Line




Physical Ethernet


Today




Physical Ethernet


Today




Physical Ethernet


FC

Today

FC FC




Physical Ethernet


FC

Today

FCoE




Physical Ethernet


Today




Physical Ethernet

Today


FC or Ethernet Summary

Ethernet wins eventually – There’s still plenty of time to decide what this

means to you

Migrating to Ethernet does not equal rip and replace all FC

– Evolutionary versus Revolutionary


Agenda

FC or Ethernet

FC vs. FCoE vs. iSCSI – FC/FCoE vs. iSCSI

– FC vs. FCoE


Best Practices

Futures


FC/FCoE vs. iSCSI

It’s more about what’s right for your environment and less about which protocol is better


FC/FCoE vs. iSCSI

0 5 10 15 20 25

Cut through

Store and Forward

Latency (usec)


FC/FCoE vs. iSCSI

0 1000 2000 3000 4000 5000 6000 7000

Cut through

Store and Forward

SSD

FC

SAS

Latency (usec)

>=10x


FC/FCoE vs. iSCSI

With block I/O, uncongested network latency is practically a rounding error

0 1000 2000 3000 4000 5000 6000 7000

Cut through

Store and Forward

SSD

FC

SAS

Latency (usec)

>=325x


FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)


FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Zone 1

Zone 2

Fabric (FC or DCB)

Zones are created by grouping the WWPNs of the host interface and storage interface into a “zone”. The set of zones created are put into a “zone set”

and activated on the fabric.


FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Zone 1

Zone 2

Zone 3

Zone 4

Zone 5

Zone 6Zone 7

Zone 8

Zone 9

Zone 10

Zone 11

Zone 12

Fabric (FC or DCB)

The number of zones in the fabric should always be greater than, or equal to, the number of initiators in

the fabric.


FC/FCoE vs. iSCSI

Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

LAN

With iSCSI, each host needs to be individually and manually pointed at a storage port by specifying

either an IP Address, IQN, or both.


FC/FCoE vs. iSCSI

FC vs. FCoE vs. iSCSI provisioning steps…


FC/FCoE vs. iSCSI


...on

host …on

network

...on

storage Total


FC/FCoE vs. iSCSI


OS ...on

host …on

network

...on

storage Total

Windows

Linux

VMware


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC

FCoE

iSCSI

Linux

FC

FCoE

iSCSI

VMware

FC

FCoE

iSCSI


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 7 FCoE 7 iSCSI 7

Linux

FC 7 FCoE 7 iSCSI 7

VMware

FC 7 FCoE 7 iSCSI 7


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 7 FCoE 7 iSCSI 7

Linux


VMware



FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 FCoE 7 iSCSI 7

Linux


VMware



FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 7

Linux


VMware



FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7

Linux

FC 1 5 7 13 FCoE 7 iSCSI 14 7

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7 7

Linux

FC 1 5 7 13 FCoE 7 iSCSI 14 7 7

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7 7


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 7 iSCSI 19 7 7 33

Linux

FC 1 5 7 13 FCoE 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 7 iSCSI 23 7 7 37


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 2 7 iSCSI 19 7 7 33

Linux

FC 1 5 7 13 FCoE 1 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 7 iSCSI 23 7 7 37


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 2 37 7 iSCSI 19 7 7 33

Linux

FC 1 5 7 13 FCoE 1 37 7 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 37 7 iSCSI 23 7 7 37


FC/FCoE vs. iSCSI


OS Protocol ...on

host …on

network

...on

storage Total

Windows

FC 2 5 7 14 FCoE 2 37 7 46 iSCSI 19 7 7 33

Linux

FC 1 5 7 13 FCoE 1 37 7 45 iSCSI 14 7 7 28

VMware

FC 1 5 7 13 FCoE 1 37 7 45 iSCSI 23 7 7 37


FC/FCoE vs. iSCSI

Management during runtime – FC/FCoE perform some amount of self

documentation due to FC Login and Name Server registrations


FC/FCoE vs. iSCSI Summary

It’s more about what’s right for your environment and less about which protocol is better

– Network/Network stack latency are not currently the best place to focus on to reduce response time

– EMC believes ▪ Network-centric is better suited for larger

environments

▪ End-node-centric is better suited for smaller environments


FC vs. FCoE

Essentially the same in terms of – Network-centric

– Similar management tools

– Same multipathing software (for iSCSI as well)

– Similar basic port types ▪ N_Ports / F_Ports vs. VN_Ports and VF_Ports

▪ E_Ports vs. VE_Ports

– Same scalability limits ▪ Number of domains

▪ Number of N_Ports / VN_Ports

▪ Number of hops


FC vs. FCoE

Completely different transports – Physical FC versus Physical Ethernet

– Point-to-point links cannot be assumed with FCoE

– FCoE uses PFC instead of BB_Credit ▪ Distance implications


FC vs. FCoE

Completely different transports – Point-to-point links cannot be assumed


FC vs. FCoE


Must support FIP Snooping


FC vs. FCoE



FC vs. FCoE


VLAN 1

VLAN 2


FC vs. FCoE

Completely different transports – Point to point links cannot be assumed

DO NOT DO THIS! Only one Fabric per VLAN


FC vs. FCoE

Completely different transports – FCoE uses PFC instead of BB_Credit

▪ Distance implications


FC vs. FCoE

+16

+16

15

14

0 Waiting for

R_RDY


FC vs. FCoE


FC vs. FCoE

Buffer reaches High water mark


FC vs. FCoE

Effective bandwidth

Distance

10G

1k 5k Max Max+x


FC vs. FCoE Summary

FCoE is FC – Management tools

– Basic concepts

– Multipathing

– Scalability limits

Transport use has implications – Just because you can, doesn’t mean you should

▪ One fabric per VLAN

▪ Pay attention to topologies that are being created

– Do not use FCoE for distances that exceed the maximum supported by the hardware vendor


Agenda

FC or Ethernet


Supported Topologies – General Guidelines

– Vendor-Specific

Best Practices

Futures


Supported Topologies General Guidelines

FC – Maximum 5 hops

– Maximum 55 domains

– Maximum 6000 N_Ports

– Maximum 10,000 N_Ports per L3 SAN

– Avoid heterogeneous FC-SW interop (please) ▪ Use NPIV if possible


Supported Topologies General Guidelines

FCoE – All FC guidelines apply (including multi-hop)

– Heterogeneous FC-SW interop is not supported

– When possible, connect directly to an FCF

– When not possible, use a FIP Snooping Bridge


Supported Topologies Vendor-Specific

There are at least 100 possible topologies – Refer to either of the EMC FCoE TechBooks


Supported Topologies Vendor-Specific – Cisco – End Device Connectivity

N2k FC

FCoE Host

EMC

N7k MDS

N5k



N2k FC

FCoE Host

EMC

N7k MDS

N5k



N2k FC

FCoE Host

EMC

N7k MDS

N5k



N2k FC

FCoE Host

EMC

N7k MDS

N5k



N2k FC

FCoE Host

EMC

N7k MDS

N5k


Supported Topologies Vendor-Specific – Cisco – Network Connectivity

FCoE

N2k

N7k

FC

N5k

MDS

FC Fabric



FCoE

N2k

N7k

FC

N5k

MDS

FC Fabric



FCoE

FC

MDS

FC Fabric

N2k

N7k

N5k



Vendor-Specific – Cisco – FIP Snooping Bridge Support

FCoE

N5k

MDS

FSB

N7k


Supported Topologies Cisco Summary

Over 100 supported Cisco topologies are described in the FCoE TechBook

Extensive support for multi-hop FCoE

Support for FIP Snooping Bridges

Connectivity to existing FC SAN does not require the use of an FC router


Supported Topologies Vendor-Specific – Brocade – End Device Connectivity

FC

FCoE Host

EMC

DCX

8000

VDX



FC

FCoE Host

EMC

DCX

8000

VDX



FC

FCoE Host

DCX

8000

EMC

VDX



FC

FCoE Host

EMC

DCX

8000

VDX



FC

FCoE Host

EMC

DCX

8000

VDX


VCS

Supported Topologies Vendor-Specific – Brocade – Network Connectivity

FC FCoE

VDX

DCX

IR

VDX

E\Ex

E\Ex

8000 FC

Fabric


VCS


FC FCoE

VDX

DCX

IR

VDX

Ex

Ex

E\Ex

E\Ex

8000 FC

Fabric


VCS


FC FCoE

VDX

DCX

IR

VDX

Ex

Ex

E\Ex

E\Ex

8000 FC

Fabric


Supported Topologies Vendor-Specific – Brocade – FIP Snooping Bridge

FCoE

FSB

Brocade


Supported Topologies Brocade Summary

A number of supported Brocade topologies are described in the FCoE TechBook

Support for multi-hop FCoE in VCS only

No support for FIP Snooping Bridges

Connectivity to existing FC SAN requires the use of an FC router


Agenda

FC or Ethernet



Best Practices

Futures


Best Practices

Maximum hops

Maximum N_Ports

Single Initiator Zoning

Monitor for bit errors


Best Practice – Max Hops

Best practice – Do not exceed 5 hops

Reason – Concern about data corruption

– Concern about fabric segmentation in certain circumstances



4 - Layer 2 Ethernet Hops Example:

4 Ethernet switches



3 FC Hops Example:

4 FC switches



Queue hold time – Length of time a frame is held before it’s discarded

– Typically between 500ms and 2 seconds

Switches contain queues – Used for buffering

– These queues have a “hold time” value associated with them



R_A_TOV >= (max hops * Hold time) * 2

R_A_TOV – Maximum life span of a frame x 2 (FC-FS)

– Typically 10 seconds



Rack

Sw mod

Host

Ethernet (Row) FC SAN

Blade Server

Rack



Rack

1

Sw mod

ToR

Host


Blade Server

Rack



Rack

EoR

1

Sw mod

2ToR

Host


Blade Server

Rack



Rack

EoR

1

Sw mod

2ToR

Host

SAN

Core

3


Blade Server

Rack



Rack

EoR

1

Sw mod

2ToR

Host

SAN

Core

3

SAN

Core

4


Blade Server

Rack



Rack

EoR

1

Sw mod

2ToR

Host

SAN

Core

3

SAN

Core

4

5


StorageBlade Server

Rack


Best Practice – Max Hops Summary

Do not exceed 5 hops – If you must, ensure that the configuration will

never create a condition where frames older than ½ of R_A_TOV will exist ▪ IOW, you need to know the hold time for every element

▪ Consider error conditions and failures

Links between any network elements that contain buffers must be counted as a FC Hop

– This includes FSBs and NPIV Gateways


Best Practice – Max N_Ports

Best practice – The number of N_Ports should not exceed 6000

Reason – Originally

▪ Testing revealed fabric segmentation due to HLO timeout between switches with around 1000 N_Ports






Best Practice – Max N_Ports Summary

Do not exceed the maximum number of N_Ports supported

– Results will be unpredictable

– In extreme cases, fabric segmentation can result


Best Practice – Single Initiator Zoning

Best practice – Each zone should only contain one initiator and

the targets it will access

Reason – Originally

▪ Concern about initiators logging in to other initiators

– Today ▪ Limits the number of name server queries that are sent

to the switch

▪ Related to Max N_Ports

▪ Recommended still due to its use in test configurations



Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)



Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)



Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)

Without zoning, the response would include information about 15 other N_Ports.



Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)

For each N_Port returned in the NS Response, host will query the Name Server for additional information. If a host only

needs to access 1 Target this means 14 extra NS queries per N_Port.



Initiator login with Single Initiator Zoning



Initiator login without Zoning



Storage 1A

Storage 2A

Storage 1B

Storage 2B

Host 1

A

B

Host 2

A

B

Host 3

A

B

Host 4

A

B

Host 5

A

B

Host 6

A

B

Fabric (FC or DCB)

Host will attempt PLOGI/PRLI with all N_Ports and perform report LUNs with all Targets.


Single Initiator Zoning Summary

Use single Initiator / single target Zoning if at all possible

If not possible or practical – e.g., some VMware and RecoverPoint

environments

– Learn to watch for the signs ▪ Randomly missing N_Port logins during power on or

after link events

▪ Elongated boot times

– Target Driven Zoning will help when available!


Monitor for Bit Errors

Best practice – Monitor for bit errors

Reason – VERY bad for performance

– FC – Buffer loss

– FCoE – Bit errors can cause unintentional data distribution ▪ An I/O convergence issue not an FCoE one


Agenda

FC or Ethernet



Best Practices

Futures – TDZ

– Network Virtualization


Futures — Target Driven Zoning (TDZ)

• TDZ is an EMC initiative aimed at automating the provisioning of networked storage

• TDZ allows for the SAN to automatically configure zoning based on information provided to it by a storage port

• TDZ makes use of Peer Zoning

– Approved in FC-GS-6 and FC-SW-6


Futures – Network Virtualization

• Network Virtualization

– Generic term being used to describe “Overlay Networks”

– Encapsulations being discussed to support this functionality are

• NVGRE

• VXLAN

• STT

– Could be very disruptive to today’s SANs


Futures

• Join us for more information!

• Birds-of-a-Feather: Storage Networking for the Future

– Time: Tuesday 1:30p

– Room: Marcello 4401 A


In Summary

Physical FC will be around for a while

Migrating to Ethernet – Can be done at any time

– Does not require rip and replace

Follow Best practices – Unless you have a compelling reason not to

– Understand the risks

The EMC FCoE Tech books – contain hundreds of supported topologies; and

– detailed installation instructions


Additional information

Please check out my blog www.brasstacksblog.typepad.com


Additional information

Or follow me on twitter!

@ErikSmith7


Q & A


Provide Feedback & Win!

125 attendees will receive $100 iTunes gift cards. To enter the raffle, simply complete:

– 5 sessions surveys

– The conference survey

Download the EMC World Conference App to learn more: emcworld.com/app


Date post:	20-Jan-2015
Category:	Technology
Upload:	emc-academic-alliance
View:	4,046 times
Download:	2 times

FC/FCoE - Topologies, Protocols, and Limitations ( EMC World 2012 )

Technology