The Unintended Consequences of Converged Datacenter Deployment Models
Simon Gordon, Juniper Networks
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 2 2
SNIA Legal Notice
The material contained in this tutorial is copyrighted by the SNIA unless otherwise noted. Member companies and individual members may use this material in presentations and literature under the following conditions:
Any slide or slides used must be reproduced in their entirety without modification The SNIA must be acknowledged as the source of any material used in the body of any document containing material from these presentations.
This presentation is a project of the SNIA Education Committee. Neither the author nor the presenter is an attorney and nothing in this presentation is intended to be, or should be construed as legal advice or an opinion of counsel. If you need legal advice or a legal opinion please contact your attorney. The information presented herein represents the author's personal opinion and current understanding of the relevant issues involved. The author, the presenter, and the SNIA do not assume any responsibility or liability for damages arising out of any reliance on or use of this information. NO WARRANTIES, EXPRESS OR IMPLIED. USE AT YOUR OWN RISK.
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 3 3
Abstract
The Unintended Consequences of Converged Datacenter Deployment Models
The last two years have seen FCoE move from buzz to production deployments. As this has happened many people have had to start dealing with the many of the unintended and unexpected consequences that arise as a result of the move to a converged infrastructure.
During this session we will examine these areas and review both the challenges and some potential solutions to meet the full datacenter requirements list. Customers and partners are seeing differences and challenges in a number of areas including:
Changes in the server redundant nic model due to io consolidation and CNA operating models Tension between storage dual independent rail and Ethernet/ip fully connected networks Differences in the HA assumptions around Ethernet/IP and FC platforms Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE Implications of multi protocol storage networks (mixed has iscsi fcoe big-data) Challenges in maintain the traditional FC Qualification Matrix / Firmware requirements model.
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Agenda – NOT COVERED
What about iSCSI, NAS, Big Data Much of what I say here applies there as well
What about multiprotocol storage networks Aka all of the above at the same time Again much of what I say here applies even more so
In other words this presentation is mostly about network convergence but happens to use FCoE as the example given the many companies currently using native Fibre Channel
4
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 5 5
Agenda – NOT COVERED
FCoE using FC-BB-5 & IEEE DCB works as intended For both ‘Converged Access’ and ‘End to End FCoE’ Of course it depends on what products you look at And depends on what topologies (logical & physical)
FC-BB-6 is adding more capabilities (but it still depends on product and topology specifics)
802.1P, and DCB (inc. ETS) gives separation of traffic in some sense like having 8 virtual networks PFC Gives Lossless per priority for FCoE DCBX helps DCB & FCoE Setup FC-BB-5 FC encapsulation, L2 Discovery, etc
P0
P1
P2
P3
P4
P5
P6
P7
P3
FCoE
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 6 6
Agenda
I See No Ships ! Is it ok to just go right ahead ? Should you ?
The Elephant(s) in the Room … Certifications qualifications interoperability upgrades HA
How do you know there’s an Elephant in the Fridge ? Because there are footprints in the butter Monitoring tools
We don’t know what we don’t know And always will be
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
I See No Ships !
When asked to retreat… Nelson claimed he couldn’t see the enemies ships
Battle ensued… And Nelson was victorious
Sometimes we should get engaged despite the fact that we know there WILL be challenges…
7
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 8 8
I See No Ships
Fibre Channel over Ethernet IS NOT PERFECT We should not claim it is
HOWEVER Fibre Channel is not perfect Ethernet is not perfect, nor is IP perfect
BUT FCoE DOES WORK within well understood boundaries EVEN End to End FCoE works well as its ‘nothing special’ FCoE can be problematic if done badly (as can FC) Just so long as you apply best practices all will be well
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 9 9
I See No Ships
Is it the right time to deploy ? When are your natural replacement cycles ? What are the current focuses for investment ? What are your short medium and long term goals ?
Most companies do not have Enough people or enough budget
To do everything they might like to do
Work out what gives the best bang for the buck Does FCoE deployment align with other activities ?
Moving towards cloud / virtualization / orchestration
Don’t try to do ‘unnatural acts’ just for the sake of it ? Just because you need to replace some kit it may not be the right time for you
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 10 10
I See No Ships
Maybe now is not the right time for you But start to understand so that you are ready when it is Choose not to converge rather than accidental…
Or you want to use convergence to increase potential connectivity to 100% attach
Aka converged access to on going dedicated SAN
Maybe its cap manage and wait for the next steps Aka converged access to ongoing dedicated SAN
Or maybe its cap and actively migrate Either to FCoE end to end Or to other Ethernet/IP storage protocols
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 11 11
The Elephant(s) in the Room…
Qualification Firmware Versions Interoperability High Availability Supported topologies
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 12 12
The Elephant in the Room
Qualification / Certification has value Verifies unidimensional interoperability Multidimensional feature testing Scale & performance testing End to End Solution testing Define best practices to make sure it works well Fibre Channel has been successful in part due to this testing
But it was never a perfect world Early on in Fibre Channel maybe we followed the qualification guidelines mostly because we bought the solution from one vendor These days as most of you have servers/storage from multiple vendors you end up breaking at least one vendors guidelines
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
The Elephant in the Room
The new FC hardware/software is tested by The manufacturer
The hardware/software development team The systems test team The solutions teams The partner pre-qualification team
The server & storage vendor Qualification test team
The systems integrator SI test lab
The customer Customer test lab vs production floor deployment
Yet sometimes you still find bugs in production…
13
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 14 14
The Elephant in the Room
Interop’ testing also done for Ethernet & IP Verifies unidimensional interoperability Multidimensional feature testing Scale & performance testing End to End Solution testing (though more limited) Define best practices (but customers diverge more) All vendors forced to support much broader interoperability Sheer volume of deployed ports leads to greater hardening App’s & high level protocols designed to run on unreliable networks Formal qualification / certification matrixes don’t tend to exist in for format Fibre Channel people are used too
Even for iSCSI and NAS in many cases Though you can argue even for iSCSI/NAS they should…
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 15 15
The Elephant in the Room
What are the issues ? What boxes need what testing
FC-FCoE Switch (FC-SW) / FCoE Forwarder (FCF) N_Port Virtualizing Gateway FIP Snooping Bridge (FSB)
When should a vendor retest ? Even when new code is not intended to affect FCoE ?
How deeply should they test ? More protocol complexity, new features, new standards, new topologies
How broad should they test Lots of features and hard to know all the interactions There are ALWAYS unintended or unanticipated consequences
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 16 16
The Elephant in the Room
Solutions… We all do risk analysis
Work out what we do or don’t need to test Work out best practices on how to deploy Prioritize where to do the testing and who with
Frankly we always did Its nothing new This is how the FC world always works This is how the Ethernet world works as well
You must still do this And I am sure you all do There is a lot of good information on FCoE best practices
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 17 17
The Elephant in the Room
Even more solutions… SAN and LAN team cooperation & coordination
Just like for dc connectivity Product selection Datacenter design Operational practices
Best practices Just because you can do something doesn't mean you should
– Remember there is a fine line between clever and stupid
Think about the implications as you come up with your design
Don’t forget most of the protocols are mature both Fibre Channel and Ethernet are mature Even DCB is very well defined and no ambiguity Testing can be focused on very specific areas of challenge
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Oh No – ANOTHER Elephant…
18 18
Different views on High availability: Single Rail Vs Dual Rail Network Design
FC Single Rail vs Ethernet Dual Rail
Product HA Characteristics Fixed & Modular - Ethernet vs FC
Implications of L2 Domain on HA Physical L2 Domain
– full separation even from untagged broadcast
Logical L2 Domain (VLAN) – partial separation as untagged broadcast may cross over
To cover this properly would take entire separate session
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Oh No –ANOTHER OTHER Elephant…
19 19
Implications of LAG LAG (LLCP) assumes packets can be sent to any link in the LAG
That is fine when the upper level protocol is AFTER the LAG
However for some devices the FCoE traffic is peeled off from the single physical interfaces before it is remerged in to the logical interface
LAG based NIC teaming can be problematic with HW offload CNA Any device that has the logic on the physical port not the logical port
Of course LAG is NOT the only load balancing or link level redundancy mechanism
There are other options that do not have these challenges
To cover this properly would take entire separate session
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 20 20
How do you know that there’s an elephant in the fridge ?
Because you can see footprints in the butter
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 21 21
Clearly Use the existing SAN monitoring and management tools Use the existing LAN monitoring and management tools
But are they complete ? Do the SAN tools properly understand the Ethernet piece Do the LAN tools give the data the SAN team needs
Not to mention dual admin Are there challenges with two sets of people using two sets of tools to monitor and manage a single set of equipment The tools, and indeed the standards, are still a work in progress
How do you know that there’s an elephant in the fridge ?
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 22 22
We Don’t Know what we Don’t Know
But then that’s always the case We do know there is value in convergence…
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
What are the scalability implications
Fibre Channel Networks getting larger And more complex
23 23
Ethernet Networks getting larger And more complex
Fibre Channel over Ethernet Merging networks means even larger networks
But better economies of scale And less diversity of equipment
That’s why FC-BB-6 is addressing scalability/topologies FCF/FDF Distributed switches VN2VN for direct layer 2 initiator to target traffic
We Don’t Know what we Don’t Know
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Implications of multi protocol storage networks (mixed NAS iSCSI FCoE big-data)
24 24
FC SAN A
FC SAN B
Big Data Storage
End of Row Storage
In Rack Storage
In Server Storage
Data De-duplication
We Don’t Know what we Don’t Know
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE
Bandwidth & Latency may be an issue for end to end converged networks Firewalls & Routers definitely get in the way of end to end converged networks
25 25
Ethernet
Servers NAS
Multi Tier Architecture
1 2 3 4
Red NetworkGreen Network
We Don’t Know what we Don’t Know
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Simplify your network to an L2/L3 Fabric of some form
Avoid L2 scale issues Avoid criss crossing the fabric for VLAN Routing
MAN/WAN L3 pushed to the edge
L3 Scale & performance
Move Firewalls to the edge and only redirect those flows needing firewall services
Plus gateway services in hypervisor
26
Firewall
Storage
MAN WAN
Fin Mkt Eng Servers
Firewall Service NO Firewall Service
Core Routers
Datacenter Fabric
Implications in the L2/L3/Firewall deployment model resulting from end to end FCoE
We Don’t Know what we Don’t Know
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved.
Many other changes underway Ethernet Fabrics
And what is the trade of between FCF vs FSB in the devices that make up the fabric
Network Virtualization & Overlay Networks What’s the impact on FCoE Far too early to tell
SDN What is the impact on FCoE May simplify here just as elsewhere
27
We Don’t Know what we Don’t Know
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 28 28
Summary
This stuff works and does brings benefits So you should plan your route forwards Go at the pace that makes sense for you Look to where you get the greatest value
However there are challenges Learn - before, during, and after - each step Keep your options open as to the final destination
The Unintended Consequences of Converged DC Deployment Models © 2012 Storage Networking Industry Association. All Rights Reserved. 29 29
Attribution & Feedback
Please send any questions or comments regarding this SNIA Tutorial to [email protected]
The SNIA Education Committee would like to thank the following individuals for their contributions to this Tutorial.
Authorship History
Original Author: Simon Gordon (Fall 2012) Updates:
Additional Contributors
Joseph White Erik Smith / J Metz