PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE

HUAWEI TECHNOLOGIES CO., LTD.

www.huawei.com

HUAWEI ENTERPRISE A BETTER WAY

Storage w sieciach Ethernet, czyli coś o iSCSI i FCoE

2014/09/30

Artur Pająk Senior IT Solution Architect



Background of Storage



History of Storage Technologies

3

Embedded Storage (ES)

Deployment Characteristics

Storage devices are

deployed in servers Storage medium cannot be

shared, and scalability is low

Servers are directly

connected to a storage

array

Storage medium cannot be

shared, and scalability is

limited by access capabilities

of storage devices

Direct Storage (DS)

Servers are connected to

a storage array through

an IP network

Shared storage medium,

high scalability, and file

storage

Network Attached Storage (NAS)

Servers are connected to

a storage array through

an IP or FC network

Shared storage medium,

high scalability, and highly

efficient block storage

Storage Area Network (SAN)



FC SAN Networking

4

SAN Characteristics

Dual planes ensure reliability: A SAN has two

independent planes (fabric networks). Each server and

storage device connects to the fabric networks through

two independent links. When any node or link fails, the

server software switches to traffic to the other link.

Auto terminal registration: Switches provide services,

as well as registration and state maintenance functions

for server and storage devices.

FC switch

Server

F_PORT

Fabric B

N_PORT

Network-side port on user device

User-side port on FC switch

Interconnection port

of FC switch

E_PORT

Fabric A

Storage

array

RSCN NS

FSPF Zone



FC Addressing

5

WorldWide Name (WWN)

WWN: (similar to MAC address) is a globally unique

64-bit address defined by a device vendor to identify all

devices (FC NICs and switch ports) on an FC network

FCID Domain Area Port

WWN

address

Fiber Channel Identifier (FCID)

FCID: Similar to an IP address. A 24-bit address used

for data forwarding on an FC network. Each FCID

identifies as a user node and is dynamically allocated

by an FC switch when a user registers with the FC

switch.

FCID Fields: The FCID consists of domain, area, and

port fields. The domain field identifies an FC switch and

accommodates a maximum of 236 domain IDs (similar

to masks used on an IP network).

FCID: used for forwarding on a fabric network.



FC Terminal Registration

6

Fabric B

Storage array

FC switch

Server (ENode)

Fabric

Login

Port Login

FC data stream

Fabric Login (FLOGI)

FLOGI: Used to request service from an FC switch and obtain

an FCID when the user connects to a fabric network.

Port Login (PLOGI)

PLOGI: Used to send a session request to the destination

node. The user can access the destination node after a

session is set up.

FC communication

After a server and a storage array set up a session through

PLOGI, they can exchange FC data. FC switches on the

fabric network search for transmission paths for FC data

based on FCIDs of the source and destination nodes.

Fabric A



FC Distributed Network Services (Fabric Services)

7

NS

Name Service: A service that an FC switch provides to

enable users to query information such as FCID-WWN

mapping.

FSPF

Fabric Shortest Path First: Similar to OSPF, this

protocol forwards FC data based on the domain ID in

the destination FCID.

RSCN

Registered State Change Notification: When a

user’s state changes, the FC switch connected to the

user sends an RSCN message to notify the FC

network of the change.

Fabric B Fabric A

Fabric Login NS:

FCID1<->WWN1

WWN1: 24.00.XX

WWN2: 36.00.XX

FSPF:

Domain in

D-FCID=switch 1

FC data

D_FCID：disk1

Switch1

Switch2

HBA

HBA



Storage and iSCSI



Why iSCSI?

FC SAN IP SAN

Converged with data

network N Y

Network Interface Card FC HBA NIC Ethernet NIC

Flow control FC TCP/IP

Transmit efficiency high low

Transmit delay low high

Impact on CPU less more

Deploy and maintain

costs

high low

Commercial maturity mature mature

Comparison of FCSAN, and IPSAN



iSCSI Addressing

10

Name (three types)

iQN: (similar to WWN address)

example

iqn.2006-07.pl.iscg:storage.disks.rack5.shelf2.vdisk2

EUI: (similar to WWN address) exemple: eui.1234567890ABCDEF

NAA: (similar to WWN address) example naa.FEDCBA0987654321

iSNS

iSNS: Similar to DNS Server.



iSCSI target and initiator

11

Target (two types)

Hardware (hidden software): disk

arrays

Software: example StarWind, MS WSS

Initiator

Mostly software



iSCSI frame size

12

Jumbo Frame

Jumbo Frame: extended size of

frame abow 1500 bytes (max size.

9k)

Function Jumbo frames must be

set on each devices in path



iSCSI Security

13

CHAP

CHAP: password on target and initiator , default not used

IPSec

IPSec: encryption transmition, default

not used.

Data Encryption

Data encryption: data encryption

example: BitLocker.

Recomendation: use dedicated network.



Converged Network



Converged Data Center Network

15

Traditional DC Network Architecture

Front-end

Network

Server

Cluster

Back-end

Network

LAN

SAN

Converged DC Network Architecture

Converged

Network

Server Cluster

& Disk Array

Converged



Network Convergence Trend

Future

Now

Past

Multiple networks

converge into one

Multiple network

adapters converge into

one, and multiple

network converge at

access layer

Multiple network adapters of

various types connect to

multiple physical networks

Server

FC IB

GE

IPC

SAN

LAN

PCIe

IPC

IPC

Server IPC/SAN/LAN PCIe Converged

Server

IPC

SAN

LAN

PCIe CNA

CNA

High bandwidth: 10GE/40GE/100GE

converged Ethernet network

Low latency: CutThrough forwarding

No packet loss: lossless congestion

control mechanism

Three elements for

network convergence



FCoE Technology



Why FCoE?

FC SAN IP SAN

Converged with data

network N Y

Network Interface

Card

FC HBA NIC Ethernet NIC

Flow control FC TCP/IP

Transmit efficiency high low

Transmit delay low high

Impact on CPU less more

Deploy and

maintain costs

high low

Commercial

maturity

mature mature

FCoE inherits the advantages of FC SAN network, such as , high transmission efficiency, low propagation delay and packet loss, etc. ；

FCoE use ethernet to bear the storage service, which reduce the cost of deployment and maintain;

FCoE is compatible with FC SAN, which can take full use of existing storage network;

FCoE

Y

CNA NIC

DCB

high

low

less

medium

medium



Technology and Standard Evolution

19

10 GE and Lossless Enhanced Ethernet Technologies have been standardized, while

FCoE is still being optimized

Large Scale 10 GE Access

10 GE CNA: Well-developed technology with many products

on the market

40 GE/100 GE network products have emerged

Lossless Enhanced Ethernet

DCB: A published standard that defines service based back-

pressure congestion control mechanisms based on FC BB-

Credit and EE-Credit

Fiber Channel over Ethernet

FC-BB-5: A published FCoE standard that defines the

encapsulation format, signaling control, and forwarding model

of FC packets over Ethernet

FC-BB-6: A draft standard that optimizes the local forwarding

mechanism and defines FDF local forwarding entities.



FCoE Encapsulation

20

FCoE Packet Format

DA: Destination MAC address, indicating the MAC address of

the next FCF hop in unicast forwarding, or reserved MAC

address in multicast forwarding.

SA: Source MAC address, indicating the MAC address of

each FCF or terminal hop.

VLAN: FIP-specified VLAN or data VLAN for FCoE packets.

Type: Ethernet type. The value 8906h indicates FCoE packets.

FCS: Ethernet frame check sequence.

VER: Version number

SOF: Start of frame

EOF: End of frame

D_ID: Destination FCID

S_ID: Source FCID

SEQ_ID: Sequence ID

Ethernet DA SA Ethernet Payload FCS

D_ID S_ID SCSI Command and data CRC

VLAN

Type=FCoE_Type (8906h)

FC SEQ_ID

Type VER FC Frame EOF FCoE SOF

Ethernet frame header: 12-byte MAC address + 4-byte VLAN tag

FCoE header: 16-byte (16-bit Ether-Type, 4-bit VER, 8-bit SOF)

EOF: 1 byte + 3-byte reserved field

FC header: 24 bytes

FC payload: more than 2112 bytes



FCoE Networking and FCoE Switches

21

FCoE Networking

Servers use Converged Network Adapters (CNAs) to connect

to FCoE switches.

FCoE switches differentiate and distribute Ethernet and FC

service flows.

The switches use the FIP protocol to complete FCoE

initialization and allocate FCIDs and MAC addresses to users.

FIP Protocol

FCoE Initialization Protocol: Allocates MAC addresses and

VLAN IDs to online users.

Types of FCoE Switches

Classification: FCoE switches are classified into FCF, NPV,

FDF, and FSB switches, based on the functions they provide.

FCFs are core components on an FCoE network and provide

key functions such as FIP and fabric services.

Ethernet

network

FC Fabric network

FCoE switch

Link type: Converged link (FCoE & Ethernet)

FC FCoE

Server

Ethernet

FCoE switch

FC switch

CNA CNA



FCoE Switch: FSB (FIP Snooping Bridge)

22

FSB functions

FSB: FIP Snooping Bridge

Control plane: Does not participate in FIP protocol control,

but listens to FIP messages and controls link access based on

FIP messages to enhance security.

Forwarding plane: Forwards packets based on MAC

addresses.

Function: Provides 10 GE FCoE access and transparently

transmits FC data over Ethernet.

FSB advantages and disadvantages

Advantages: Easy to deploy. Highly efficiency MAC-based

forwarding.

Disadvantages: Cannot be deployed independently. An

FSB supports FCoE ports only and is not compatible with

FC networks. FC traffic must be transmitted by FCFs

(circuitous forwarding paths).



FCoE Switch: N_Port Virtualizer (NPV)

23

NPV functions

NPV: N_Port Virtualizer

Control plane: Participates in FIP protocol control. Functions

as a proxy agent for servers to send FIP requests.

Forwarding plane: Forwards packets based on FCIDs.

Function: Provides FCoE access and supports connection of

FC ports to traditional FC networks.

NPV advantages and disadvantages

Advantage: Compatible with FC networks.

Disadvantages: Cannot be deployed independently. FC

traffic must be transmitted by FCFs (circuitous forwarding

paths).



FCoE Switch: FC Forwarder (FCF)

24

FCF functions

FCF: FC Forwarder.

Control plane: Provides FIP registration service, and

provides fabric networks with the Naming Service (NS),

Registered State Change Notification (RSCN), and FC

Shortest Path First (FSPF) services.

Forwarding plane: Forwards packets based on FCIDs and

supports local forwarding.

Function: Provides FCoE initialization function (FIP) and

fabric service.

FCF advantages and disadvantages

Advantage: Compatible with FC networks and can be

deployed independently.

Disadvantage: Domain ID specification limits network

scale.



FCoE Switch: FCoE Data Forwarder (FDF)

25

FDF functions

FDF: FCoE Data Forwarder.

Control plane: Participates in FIP protocol control but does

not provide fabric service.

Forwarding plane: Forwards packets based on FCIDs and

supports local forwarding.

Function: Provides FCoE access and FC ports to connect to

traditional FC networks, and supports local forwarding.

FDF advantages and disadvantages

Advantage: Compatible with traditional FC networks and

supports local forwarding to avoid circuitous forwarding paths.

Disadvantage: Cannot be deployed independently and must

work with FCFs.

FDF is defined in FC-BB-6 but has not

been standardized



Changes brought by FCoE

Server do not need both Ethernet NIC and FC HBA but one CNA converged card, which can reduce the CAPEX of NIC;

Data network and Storage network begin to converge, which reduce the CAPEX of network;

Take full use of existing FC SAN network, which protect the CAPEX of network;

Switches support FSB/NPV function, use single or multiple hop mechanism to access storage, the scale of storage become larger;

Switches support DCB function, which give a lossless Ethernet to bear storage service;

FCF FCF

FSB/NPV

FSB/NPV

Converged Network

FC SAN Storage

Data Network

Storage Network

26



FCoE Forwarding Path Optimization

27

FSBs and NPVs forward traffic through

circuitous path

FC service flows are forwarded by FCFs or FC switches based

on FCIDs.

Servers and storage devices connected to the same FSB or

NPV cannot communicate directly, and data exchanged

between them must be forwarded by the upstream FCF or FC

switch.

FDFs support local forwarding

A fabric network can be expanded by adding FDFs, which do not

occupy domain IDs.

FDFs support local forwarding. Data exchanged between servers and

storage devices under the same FDF is forwarded by the FDF, and

data exchanged between those under different FDFs are forwarded

by the upstream FCF.

FSB NPV

FCF

FDF

FDF

FCF



FCoE Switches Comparisons

28

Support

Traditional FC

Forwarding

Mechanism

Shortest Path

First

Distributed

Fabric service

Use Domain ID Scalability Complexity

FSB No MAC No No No Good Low

NPV Yes FCID No No No Good Medium

FCF Yes FCID Yes Yes Use

exclusively

Poor High

FDF Yes FCID Yes No Share Good Medium

FCF

FSB

FCF

NPV FCF

FCF

FCF FDF

FCF

FDF

?

FSB NPV

FCF + FSB networking

No forwarding path

optimization

FCF + NPV networking

No forwarding path

optimization

FCF networking

Optimized forwarding paths

FCF + FDF networking

Optimized forwarding paths



Enhanced Ethernet Technology



Ethernet Technology Improvements

31

Traditional Ethernet

Congestion Control: Uses TCP

retransmission and sliding window

mechanisms to perform flow control

based on destinations.

Pause Mechanism: Traditional

Ethernet switches send Pause frames

to request downstream devices to stop

sending frames.

Traditional FC

Congestion Control: Uses buffer-to-

buffer credit (BB-Credit) and End-to-

End credit (EE_Credit) mechanisms.

(BB_Credit and EE_Credit are

similar to the token bucket

mechanism and perform flow control

based on bandwidth capabilities

between nodes.)

Enhanced Ethernet

Congestion Control: Combines the

Ethernet Pause mechanism and FC

flow control to define the multi-level

congestion control mechanism based

on PFC, ETS, and the CN based end-

to-end congestion control mechanism.

Storage services require lossless transmission.

Traditional FC technology has Credit mechanism to ensure lossless transmission, but traditional Ethernet only

implements “best effort” transmission.

DCB technology is introduced in Ethernet ensures FCoE (QoS).



Priority Flow Control (PFC)

Page 32

Server FCOE switch FCOE switch Storage array

E Ethernet service packet

F

FC storage service packet F IPC service packet I

Ethernet 0/1 Ethernet 0/2

Ethernet 0/1 Buffering queue




Ethernet 0/1 Ethernet 0/2

pause pause pause F F F F F F F F F

PFC service identification

Different priorities (0 – 7) and queue thresholds are configured

for various services in network planning.

Switches perform congestion control based on service priorities.

When a service traffic priority exceeds the threshold, excess

packets are dropped. Packets of other services are forwarded

normally.

Multi-level back pressure

Ethernet Pause mechanism: When congestion occurs, switches send

back-pressure signals to downstream devices, requesting downstream

devices to stop sending service packets.



Enhanced Transmission Selection (ETS)

33

ETS service identification

Different priorities, scheduling modes, and bandwidth thresholds

are configured for various services in network planning.

Priority queuing is performed for latency sensitive IPC service,

and polling scheduling is performed for SAN and LAN services.

Cooperation with CN and PFC

ETS uses PFC or CN back-pressure for services requiring low latency

and high reliability, such as IPC and SAN. When the rate of such a

service exceeds the bandwidth threshold, ETS sends a back-pressure

signal to the downstream node to reduce loads on the local device.

For latency insensitive LAN service, retransmission mechanism of

TCP/IP is used, instead of the back pressure mechanism.

The TCP/IP retransmission is used for latency-insensitive LAN services.



CN(Congestion Notification)

Page 34

Server A

FCOE switch FCOE switch Storage Array

Ethernet 0/2 Ethernet 0/1 Ethernet 0/2

Source node address

…

Congested Queue

Server B

Server C

Server A

Server C

CN implementation

Implementation: An FCoE switch searches for the source node

that is causing the congestion and sends CN messages to that

node until the congestion is removed.

CN characteristics

Finds the source node that is causing the congestion and prevents the

node from sending packets to the network.

FCoE switch forward the FCoE frame hop by hop, the source MAC address may be not the MAC address of initial sender, so in this case,

there is few difference between CN and PFC.



Huawei Converged Network Solution



Huawei FCoE Solution

37

Fabric B

Ethernet link FCoE link Ethernet & FCoE converged link

40 GE trunk

Aggregation

Switch

CE12800

FC Storage

Access Switch CE6800

Ethernet FCF

FSB

Server

CNA CNA CNA CNA CNA CNA

10 GE

Fabric A

Illegal user

MAC spoofing

MAC:0066-6666-666

eNode

VN_PORT1：

MAC:0066-6666-666

High reliability

Double Fabric Planes: Each server connects to two

fabric planes through double CNAs to ensure reliability of

storage services.

Multiple Ethernet links: Each fabric network has multiple

Ethernet links bundled into a trunk link to improve link

reliability.

No packet loss

DCB: Deployed between servers, FSBs, and FCFs to

prevent packet loss for FC storage services on the

Ethernet network.

10 GE access: Next generation data center 10 GE

switches provide sufficient bandwidth for FCoE access.

High security

FSB: Listens to FIP messages to control user access.

Links are open only to authorized users.

FC SAN



Huawei Converged Network Evolution Roadmap

38

Servers

Core switch CE12800

Access Switch CE6800

FC/FCoE switch(FCF)

FC Storage

Ethernet port

FCoE/FC port

SAN

Access and storage network convergence

TOR switches support FCoE (V1R1:FSB) / FC (V1R2: FCF/NPV) ports

and DCB.

TOR switches distribute storage traffic to the SAN.

Core and storage network convergence

TOR and core switches support FCoE/FC ports and DCB.

Core switches distribute storage traffic to the SAN.



Huawei FCoE Configuration Example



Huawei FCoE Solution Topology Example

Page 40

Topology statement

― CE6850 connects to Server A and Server B

through port 10GE1/0/1 and 10GE1/0/2

respectively；

― CE6850 connects to FCF through port 10GE1/0/3

to access SAN storage;

― CE6850 connects to data network through port

40GE

Port Statement

― VNP-port：the port of CE6850 connecting to FCF

― Enode-facing： the port of CE6850 connecting to

Server



Huawei FCoE Solution Example Configuration step ——FSB configuration

Configure an FC instance: Configure an FC instance FSB on SwitchA and specify FCoE VLAN 2094 in this instance;

Configure port roles: Configure 10GE1/0/3 on SwitchA to allow VLAN 2094. The configurations of 10GE1/0/1 and 10GE1/0/2

are similar to the configuration of 0GE1/0/3, and are not mentioned here. An interface is an ENode-facing port by default, so the

configurations of 10GE1/0/1 and 10GE1/0/2 are not mentioned.

Page 41



Huawei FCoE Solution Example Configuration step ——Priority configuration

Configure priority mapping:

― create a DiffServ domain ds1and map 802.1p priorities of received server packets to CoS values.

― Bind the DiffServ domain to 10GE1/0/1, 10GE1/0/2, and 10GE1/0/3.

Page 42



Huawei FCoE Solution Example Configuration step ——DCBX and PFC Configuration

Configure DCBX

Configure PFC:

― Enable PFC for queues with priority 3. By default, PFC has been enabled for queues with priority 3.

― Enable PFC on an interface.

Page 43



Huawei FCoE Solution Example Configuration step ——ETS Configuration

Configure ETS:

― Create an ETS profile.

― Map queue 3 to PG1, queue 7 to PG15, and other queues to PG0. Queue 3 maps PG1, and queues 6 and 7 map PG15 by

default, so you only need to add queue 6 to PG0.

― Configure flow control based on the priority group and set DRR weights of PG0 and PG1 to 60% and 40% respectively.

― Apply the ETS profile to 10GE1/0/1 and 10GE1/0/2.

Page 44



Huawei FCoE Solution Example Configuration step ——FCF Configuration

Configure CEE profile, map profile, and so on.

FCoE profile must be configured on the port connecting to CE6800

FCoE profile must be configured on the port connecting to SAN storage.

Zone must be created on FCF. Host and storage must be added in zone. The default rule of

zone must be set as “permit”

LUN, host group, host must be created on SAN storage, FC initiator must be created and

indicated as WWN server.

FC initiator must be added to host;

Page 45



Are we ready for use FCoE????



Huawei Enterprise A Better Way

Date post:	01-Jul-2015
Category:	Internet
Upload:	proidea
View:	165 times
Download:	4 times

PLNOG 13: Artur Pająk: Storage w sieciach Ethernet, czyli coś o iSCSI I FCoE

Internet