WHITE PAPER

915-2505-01 Rev C October 2013www.ixiacom.com

Best Practices in DeployingConverged Data Centers

2

3

Contents

Introduction ................................................................................................. 4

Converged Data Center ............................................................................... 4

Deployment Best Practices ......................................................................... 6

Testing Best Practices ................................................................................. 8

Conclusion ..................................................................................................14

About Emulex .............................................................................................15

About Ixia ...................................................................................................15

4

IntroductionThe increasing demand and dependency on Internet resources is undeniable. High speed Internet used to be a luxury 10 to 15 years ago, but has evolved into a primary method of communication and research – from college students to professionals to friends and relatives. Web presence, social networking sites, and high definition media services grew explosively, while businesses and enterprises continued to expand existing large databases with one or more duplicate backup sites. More recently, a rapid adoption of network-based services such as desktop virtualization and cloud computing has begun to manifest as a new paradigm of network management.

These digital age trends put an unprecedented strain on servers and storage, but data centers cannot continue to grow linearly in size. With economic and social pressure to make data centers more technologically efficient and environmentally green, data centers must substantially increase service capacity and storage flexibility without significantly increasing physical footprint.

Data center networks have come a long way since the mainframe days. Today, large data centers can support thousands of rack mounted and blade servers using multiple networking infrastructures to support a variety of processes – Ethernet for high volume IP communication, fibre channel for high speed storage access and InfiniBand for high-performance computing. Each of these infrastructures is specifically designed for the type of I/O that it delivers, and this model has proven successful. However, as demand continues to climb exponentially year over year, the multiple infrastructure model becomes unmanageable.

One popular solution for data center evolution is server virtualization. Server virtualization brings many technical and economical benefits to the data center infrastructure. Data centers that leverage virtualization technologies can maximize network asset utilization, deliver computing resources on demand, and seamlessly move data to any location on the fly.

Converged Data CenterA large enterprise firm operates thousands of powerful servers designed to accommodate peak hour loading. For the majority of the day these servers are highly underutilized, but still consume network and power resources. To optimize the utilization of the data center assets and to minimize unnecessary power consumption, operators are looking at server virtualization to dynamically create and destroy virtual machines as needed and to assign hardware resources to each virtual machine as necessary.

The next major cost savings initiative is the convergence of network and storage I/O, and the consolidation of multiple switching frameworks. Two of the key technologies are fibre channel over Ethernet (FCoE) and data center bridging (DCB).

With FCoE, each fibre channel frame is fully encapsulated into an Ethernet frame. From a software stack perspective, the original fibre channel stacks from the protocol layer (FC-2) onward remain unchanged and the lower fibre channel layers (FC-0 and FC-1) are replaced by Ethernet. With this approach, FCoE inherits all link initialization and maintenance capabilities to well-defined and mature fibre channel operations without modifications.

With economic and social pressure to make data centers

more technologically efficient and

environmentally green, data centers must substantially

increase service capacity and

storage flexibility without significantly increasing physical

footprint.

5

Simply providing an Ethernet network to transport FCoE frames is not sufficient in data center networks , as fibre channel is a lossless technology, and Ethernet is a lossy transport built on maximizing low-cost, best-effort data communication.

FCoE Link

Simply providing an Ethernet network to transport FCoE frames is not sufficient in data center networks , as fibre channel is a lossless technology, and Ethernet is a lossy transport built on maximizing low-cost, best-effort data communication. Enhancements must be incorporated into today’s Ethernet switches so data centers can deploy FCoE commercially, and more importantly, reliably.

To address this need, the IEEE 802.1 working group initiated the DCB Task Group to define protocols that use Ethernet for lossless transmission. Congestion notification, enhanced transmission selection (ETS), and priority-based flow control (PFC) are three critical technologies that work cohesively to achieve lossless Ethernet. PFC is the most notable and widely supported of the three protocols. It takes the concept of the original 802.3x flow control, which prevents congestion by requesting the uplink to cease transmission on the entire link, and applies it to up to eight virtual links (PFC Queues) on each physical port. Each virtual link is identified by the VLAN priority value, and each virtual link can be pre-configured to carry a specific type of traffic requiring a different level of treatment. For example, it is a common best practice today to carry FCoE traffic over VLAN priority 3.

DCB-based deployments begin at the server with a converged network adapter (CNA) that supports 10 Gb/s Ethernet (10GbE) and FCoE on a single adapter port. For a typical scenario, a dual-port CNA is used to replace a dual-port fibre channel HBA and multiple 1 GbE NICs (often six or more for virtualized servers). The result is immediate cost savings for adapters and cabling and ongoing reductions in energy and cooling costs.

Another requirement is a bridge between the converged DCB Ethernet infrastructure and existing Ethernet LANs and fibre channel SANs. To meet this requirement, DCB switches employ ports that support enhanced Ethernet with FCoE, traditional Ethernet, and FCoE. This allows FCoE to be introduced at the edge of the data center, typically with new servers, while continuing to use the existing network and storage infrastructure.

6

Converged Data Center Network

A well designed converged data center will also have a single management pane for fibre channel and TCP/IP networks and support server virtualization to enable resource optimization by dynamically provisioning virtual machines and allocating hardware resources as necessary.

Deployment Best PracticesThe industry is beginning a very exciting and promising evolution, and standards are being ratified as a key step in the technology cycle. The standard (ANSI INCITS T11 FC-BB-5) has been ratified for FCoE and several others are making their way through maturing drafts in the IEEE 802.1 DCB Task Group. That is not to say that the converged data center is not ready or interoperable. The fact is, with common best practices, a multi vendor end-to-end converged data center is already possible with CNAs and DCB-enabled switches that are currently available for rack-mount and blade deployments. The technology has also been demonstrated in various interoperability test events sponsored by the Ethernet Alliance (EA) and Fibre Channel Industry Association (FCIA).

State of the Industry – Fibre Channel over Ethernet

The FC-BB-5 standard defines Fibre Channel over Ethernet and a protocol called FCoE Initialization Protocol (FIP) that provides discovery, initialization and maintenance tools. FC-BB-5 is well implemented among supporters, with most supporting up to draft version 1.04 at the time of this writing. Draft version 1.04 introduces the FIP VLAN Discovery protocol, and is the last major change to the FC-BB-5 specification.

State of the Industry – Lossless Ethernet

The IEEE 802.1Qbb project defines priority-based flow control, and is in draft version 2.3 at the time of this writing. PFC is well supported and interoperable among implementers, primarily because it is based on a well established flow control method (i.e. 802.3x) and most of the implementation is local behavior. The major differentiations between implementers are the number of PFC Queues a device can support and the response time in which the device can throttle transmission upon receiving a PFC PAUSE request.

With common best practices, a multi

vendor end-to-end converged data

center is already possible with CNAs

and DCB-enabled switches that are

currently available for rack-mount and blade deployments.

7

Early adoption of leading edge technologies is key to achieving market leadership in the converged data center.

The IEEE 802.1Qaz project defines an enhancement to the bandwidth allocation and management system of Ethernet switches called enhanced transmission selection (ETS), as well as a protocol called data center Bridging exchange (DCBX) to advertise and configure lossless Ethernet parameters. Most implementations today do not support the IEEE 802.1Qaz specification. Instead most implementations support two specification proposals commonly known as pre-CEE (DCBX subtype 1) and CEE (DCBX subtype 2). At the time of this writing, DCBX interoperates well among implementers for pre-CEE and CEE versions, and the draft version is 1.6. ETS is also fairly well supported, with the major differentiation among implementers being the total number of ETS Priority Groups supported by the device.

The IEEE 802.1Qau project defines a mechanism called congestion notification (CN) for an intermediate point or endpoint to request the transmitting device to throttle the transmit rate of a specific data flow. Very few devices on the market support congestion notification, and the draft version is 2.4 at the time of this writing.

Early adoption of leading edge technologies is key to achieving market leadership in the converged data center. In the absence of fully ratified standards, care must be taken to ensure the proper implementations are put in place. The following sections suggest best practices from several leaders in the converged data center industry.

Converged Network AdaptersNetwork convergence is built on a 10GbE infrastructure. In many cases, the requirement for 10GbE bandwidth will lead the transition, particularly for data centers using virtualized servers that typically require multiple 1GbE NICs to support hypervisor functions and I/O requirements for multiple virtual machines.

When appropriate, the next step will be acquisition of DCB-capable switches that connect to FCoE CNAs in the server. In order to support this two-step process, Emulex is providing a flexible “pay-as-you-grow” capability with the OneConnect™ Universal CNA platform. This allows data centers to begin with a standard OneConnect 10GbE NIC and then add FCoE support with a straight-forward license upgrade to an installed OneConnect adapter. Data centers that want to go directly to FCoE can begin with OneConnect CNAs that ship from the factor with FCoE support.

Based on the server deployment plan, data centers may want to deploy FCoE with blade and rack-mount servers. Working with long-standing OEM relationships, Emulex OneConnect technology is available in a wide range of PCI Express 2.0, blade and mezzanine adapters from leading server and storage suppliers. This allows optimized management of FCoE for all servers throughout the data center from a single console using the Emulex OneCommand Manager application that supports TCP/IP, iSCSI, FCoE and traditional HBA functions.

8

Testing Best PracticesTraditional storage test tools cannot satisfy the requirements of the converged data center because they often lack the broad technology coverage and port density required by future converged data centers. Converged data center require virtualization, network security, routing/switching technologies, FCoE, and 40/100GbE. The best practice is to test with a high performance and high density unified test platform that covers all the key network elements from the end user plane all through the storage area network. Only a truly unified test platform can effectively measure the end-to-end quality of service delivered by the converged data center.

Testing Areas for Converged Data Centers

The importance of having a test platform that is unified and delivers high performance at high port density is quite obvious. The next generation converged data centers will be:

• High-performance – aggregating millions of business critical transactions per second up to 10 Gbps and 40/100 Gbps.

• High density – forming an ultra low latency fabric consisting of hundreds to thousands of 10 Gbps Ethernet ports

• Unified – consolidating storage, network and high performance computing I/Os into a single 10 Gbps Ethernet link.

All of these key requirements must operate in unison, not as independent network elements. To truly characterize the performance, reliability, and stability of the converged data center, a unified test platform with new methodologies is the best practice.

CNA/Virtualization Use Case: Layer 2-3 Network PerformanceValidating the performance of a virtual switching fabric requires the same real world notions of throughput and frame loss that are relevant in the physical world. Obtaining a measure of the best case performance of a virtual switch as well as the physical cards connected to it will give you a high level understanding of the upper limit on performance for the system.

Traditional storage test tools cannot satisfy the

requirements of the converged data

center because they often lack the broad

technology coverage and port density

required by future converged data

centers.v

9

Layer 2-3 Network Performance

A very simple traffic configuration between virtual ports or between a combination of virtual ports and physical ports can be created extremely quickly to verify both reach-ability and performance at the Ethernet layer and the IP layer.

Layer 2-3 Network Performance Statistics

Extending this simple traffic configuration to include larger numbers of virtual ports will put strain on the virtual switching fabric. The aggregated traffic from multiple VM sources and destinations can be forwarded through a CNA and typically accelerated to physical world hardware acceleration to packets sourced from and destined to other parts of the physical network outside the virtual switch boundaries. The no drop throughput rate obtained when frame loss percentage is 0% will give the tester an accurate view of how much traffic can be pushed through the end-to-end environment bidirectionally.

10

When the no drop throughput is not a highly desired, real-time statistics make it easier to isolate faults without having to disrupt the active testing. Further real-time drill-down into each virtual port will yield the individual flows that can be tracked by IP address. For example, you can determine which specific VM activities might affected by network controls such as quality of service (QoS).

CNA/Virtualization Use Case: Layer 4-7 Network PerformanceMeasuring the maximum throughput achieved via Layer 4 protocols such as HTTP will yield an application level view of the performance that can be achieved by an application server such as a web server in delivering data to multiple clients accessing the server concurrently.

Having identified the layer 2-3 no drop rate throughput from the previous use case we have a measure of the maximum packet rate that can be achieved through the system, but we need to further refine our measurements to include the overhead of retransmissions causes by congestion, timeouts, etc.

Layer 4-7 Performance Test

By generating real world application traffic the tester can determine what level of responsiveness the actual end user will experience. This is particularly true of virtualized environments where resources such a CPU, memory, and even the CNA itself are shared resources whose performance will vary over time as load on the host machine changes.

Layer 4-7 Performance Test Results

Measuring the layer 4 throughput over long duration intervals will yield a graph of how the system behaves over the long term. One-way latency through the system can be measured by dividing the round trip latency in half.

11

Layer 4-7 Performance Test Results

CNA/Virtualization Use Case: Storage I/O Performance

The CNA assists with the off-loading of storage traffic performance and enables the virtual machine to see external storage at the operating system level. From the point of view of the application, external storage can be written to and read from as if it was directly attached. The CNA is responsible for aggregating this storage I/O traffic together with other Ethernet traffic that is inbound/output from the CNA. An important benchmark that needs to be established is what the storage I/O throughput rate is in isolation so that we can compare it with results in final use case where storage and data traffic are interleaved.

Storage I/O Performance Test

Storage requests and writes are more varied then just sending raw data traffic. Storage I/O requests are configured using a weighting percentage to more accurately reflect the real world nature of storage requests. For example, storage traffic destined to and from the virtual machine will vary in the size of the SCSI block data size requested, as well as in the type of request, read, or write operations. The virtualization test tool will be responsible for generating this average weighting between request types and block sizes while controlling other parameters like number of concurrent sessions to achieve a desired throughput objective.

12

Storage I/O Performance Test Results

Latency measurements are also a key indicator of storage performance as high latency values could correspond to a less than optimal configuration which is adding additional lag to each storage request.

Storage I/O Performance Test Results

CNA/Virtualization Use Case: Live Migration

Live migration impacts performance of the network at all levels and combines elements of the previous uses cases together with the need to identify the maximum time interval over which service is affected. Transaction times for HTTP traffic can be used to obtain a measure of the impact of doing live migration of a virtual machine from one host to another host.

Live Migration Test

13

An embedded virtual port or a physical port is assigned as an HTTP client and a separate embedded virtual port is assigned as an HTTP server. HTTP traffic is generated between client and server to establish a baseline when the HTTP server VM is in normal operation. A live migration function is then initiated for the VM acting as the HTTP server so that it is moved from one physical server to another physical server.

Live Migration Test Results

During the process of live migration, the transaction times for HTTP will be adversely affected until the live migration operation completes and the HTTP server VM assumes full operational state on the new physical server. To mitigate the impact of this transition, live migration employs using a common disk storage that is shared by both physical servers. In this manner only a fraction of the information of the live running HTTP server VM needs to be moved by live migration and the HTTP Server should resume to full state very quickly.

Live Migration Test Results

CNA/Virtualization Use Case: Isolated Load Performance Testing

The converged network adapter plays a crucial role in the converged data center, as it is responsible for delivering various business-critical I/Os to and from the servers. To verify the forwarding performance of the CNA itself, it is best to isolate the CNA from other systems in the test network and surround the CNA with test equipment.

14

CNA Load Testing

As shown in the diagram, the test emulates a set of virtual machines using embedded virtual test ports, and then emulates a converged data center network using physical test ports and source traffic from the virtual machines over the CNA all the way across the converged data center network to measure the maximum performance of the CNA.

The physical test port will emulate an FCoE switch, allowing the CNA to proceed with DCBX and FIP/FCoE processes as it would with a real FCoE switch. Once the CNA is logged in to the fabric, traffic will be generated from the virtual test ports (VMs) to the remote endpoints simulated behind the physical test port. Performance metrics of the CNA such as throughput, latency, frame loss, and jitter, among many others, can be measured. By removing other components from the test network such as a 3rd party switch improves the accuracy of the test results, as there will be latency, congestion and some level of jitter introduced by the 3rd party switches.

ConclusionThe leap into large scale server virtualization, supported by a homogeneous high speed Ethernet infrastructure that unifies the IP and storage area network (SAN), enables a truly converged data center that answers to the challenges of consolidation, increased capacity and lower total cost of ownership (TCO).

As with any technology cycle, the most exciting period is during the early research and development phase. At this stage, changes are most rapid, revolutionary ideas are proposed, and companies have the most opportunities to become market leaders. Thorough testing of converged data center technologies is also most critical at this stage, as many new prototypes, chipsets and systems are being introduced to the market place. Once testing is completed, deployment must be executed with well established best practices, because in a converged data center network, there will be multi-vendor products working at different stages of the end-to-end network.

The leap into large scale server

virtualization, supported by a

homogeneous high speed Ethernet infrastructure

that unifies the IP and storage area network (SAN), enables a truly converged data

center that answers to the challenges of consolidation,

increased capacity and lower total cost of ownership (TCO).

15

About Emulex

Emulex is the leader in converged networking solutions for the data center. The Emulex Connectivity Continuum architecture provides intelligent networking services that transition today’s infrastructure into tomorrow’s unified network ecosystem. Emulex provides a single framework that intelligently connects every server, network and storage device within the data center. Through strategic collaboration and integrated partner solutions, Emulex provides its customers with industry leading business value, operational flexibility and strategic advantage.

The Emulex OneConnect UCNA platform is based on 10 generations of field-proven Fibre Channel HBA technology. Key features and benefits include:

• Full protocol off-loads for TCP/IP, FCoE and iSCSI to optimize server efficiency and maximize server virtualization

• Industry-leading performance for networking and storage

• Integrated management of Emulex OneConnect NICs, FCoE CNAs, iSCSI adapters and LightPulse™ HBAs throughout the data center using the OneCommand Manager application

About Ixia

Ixia’s converged data center test solution is the industry’s only high-density unified platform that includes both FCoE and Fibre Channel interfaces, converged traffic performance tests, and real I/O performance testing. Ethernet interfaces can operate at speeds from 1 and 10 Gbps, while the Fibre Channel interface module’s 4 or 8 port variants can be dynamically programmed to 2, 4 or 8 Gbps on each port. Ixia’s solution includes IxANVL and IxNetwork test applications that utilize the same hardware platform:

• IxANVL validates device conformance to FCoE protocols, including FIP and DCBX.

• IxNetwork offers performance testing over Ethernet, FCoE, and Fibre Channel connections by emulating large-scale network environments, allowing end-to-end testing of converged FCoE switches, CNAs, and other mixed protocol devices.

WHITE PAPER

Ixia Worldwide Headquarters26601 Agoura Rd.Calabasas, CA 91302

(Toll Free North America)1.877.367.4942

(Outside North America)+1.818.871.1800(Fax) 818.871.1805www.ixiacom.com

Ixia European HeadquartersIxia Technologies Europe LtdClarion House, Norreys DriveMaidenhead SL6 4FLUnited Kingdom

Sales +44 1628 408750(Fax) +44 1628 639916

Ixia Asia Pacifi c Headquarters21 Serangoon North Avenue 5#04-01Singapore 554864

Sales +65.6332.0125Fax +65.6332.0127

16

WHITE PAPER

Ixia Worldwide Headquarters26601 Agoura Rd.Calabasas, CA 91302

(Toll Free North America)1.877.367.4942

(Outside North America)+1.818.871.1800(Fax) 818.871.1805www.ixiacom.com

Ixia European HeadquartersIxia Technologies Europe LtdClarion House, Norreys DriveMaidenhead SL6 4FLUnited Kingdom

Sales +44 1628 408750(Fax) +44 1628 639916

Ixia Asia Pacifi c Headquarters21 Serangoon North Avenue 5#04-01Singapore 554864

Sales +65.6332.0125Fax +65.6332.0127

915-2505-01 Rev C October 2013