Matthew Slavin – mslavin@cisco - e-TechServices to Cisco CBS switches in the IBM BladeCenter...

Introduction to Cisco CBS switches in the IBM BladeCenter

Matthew Slavin – [email protected] Engineer – Strategic Alliances

© 2007 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1

Consulting Engineer Strategic Alliances

Cisco SystemsApril 2008 Rev A1

AgendaAgenda

Current Cisco IBM Blade Options

Virtual Blade Switch Introduction

Looking at some specifics

Mi ll I f tiMiscellaneous Information

© 2007 Cisco Systems, Inc. All rights reserved. 2

Current Cisco IO Portfolio for IBM BladeCenters

Intelligent Ethernet SwitchingIntelligent Ethernet SwitchingIntelligent Ethernet SwitchingIntelligent Ethernet Switching

s

Intelligent Server SwitchingIntelligent Server SwitchingIntelligent Server SwitchingIntelligent Server Switching

Serv

ers

IBM EoSIBM EoSIBM EoSIBM EoSIBM EoSIBM EoSIBM EoSIBM EoS

1X HCA1X HCA1X HCA1X HCA

I t lli t SAN S it hiI t lli t SAN S it hiI t lli t SAN S it hiI t lli t SAN S it hi

Bla

de

4X HCA4X HCA4X HCA4X HCAIntelligent SAN SwitchingIntelligent SAN SwitchingIntelligent SAN SwitchingIntelligent SAN Switching 4X HCA4X HCA4X HCA4X HCA


4G Fibre Channel Switch4G Fibre Channel Switch4G Fibre Channel Switch4G Fibre Channel Switch

Recent Evolution of Blade EnclosuresIBM BladeCenter E HP p-ClassDell 1855/1955IBM BladeCenter E HP p ClassDell 1855/1955

7U6U*7U

9U 10U10U

HP c-ClassIBM BladeCenter H

Implications for the I/O connections offer higher Larger enclosures means fewer servers per rack over last

Dell M1000e(only IBM maintained compatibility between generations)


DataCenterspeeds and greater numbers over last generation

servers per rack over last generation

But how am I going to manage all of these switches!!

AgendaAgenda






Who benefits when blades are deployedCustomer server teams?

Increase in CPU’s per square foot

Rapid deployment of new servers

Far better MTTR over stand alone servers

Customer facilities teams?

Massive reduction in cabling requirements

Reduction in power and cooling per CPU

Customer network teams?

Imagine the network team going to the server team and saying “Hey, I’ve got this great new networking solution, all I need is for the server team to install and administer


all I need is for the server team to install and administer 100’s of extra servers.” = Goes over like a lead balloon

How is the network team impacted?G f i l f l i h 100’ f ll bl dGoes from managing a couple of large switches to 100’s of small blade

switchesHave to install and troubleshoot 100’s of discrete switches

Have to keep track of configs, patches, security, etc. for 100’s of extra switches

Dramatically increases points of failure in the network

Confusion over the boundary between network and server teamsConfusion over the boundary between network and server teams

Leads to increased time to deploy and maintain = TCO goes up for networking

Blade switches still lack some important/desired features

Example - Full PVLAN – often important for HIPAA

Almost all large customers have to redesign their network to accommodate these small switchesthese small switches

Redesign and testing can take months and require many resources

Requires a significant investment in manpower to do right


This is why network teams frequently push for pass-thruPass-thru = No extra switches to manage, no need to change design

What is Cisco doing to address this?The Next Generation – Cisco Virtual Blade Switch!

True switch virtualization via switch stackingStacking provides up to 9 to 1 reduction in switches managed

More cable flexibility over any of the current gen solutions

True plug and play on failure replacement – Reduces MTTRTrue plug and play on failure replacement Reduces MTTR

More feature rich solution then any of the competitioni.e. full PVLAN support

Provides greater flexibility in design choicesEasier to deploy in existing network designs

L3 i ti l ti t th bl d l lL3 is now a practical option at the blade level

Servers get standards based NIC load balancingCurrent crop of switches only work well with Active/Standby


Current crop of switches only work well with Active/Standby

No other vendor can even come close to this solution!

Introducing VBSBased on Cisco 3750E chip setBased on Cisco 3750E chip set

Real field-proven technology!StackWise Plus = 64G stacking links (AKA VBS cabling)

Cables offered in .5, 1 and 3 meter lengthsStacked switches look like modules in a single modular switch

Rich feature setUsable L3 support, Full PVLAN, Cross-stack EtherChannel, Flexlink support, etc.

Three SKUs to be offeredCBS 3110G – 4 x 1G uplinks with stacking (RJ45)CBS 3110X – 1 x 10G uplink with stacking (X2)

X2 module support at FCS - CX4, LR and LRMCBS 3012 – 3rd SKU to be released shortly thereafter

4 x 1G uplinks without stacking (Entry level targeted at BC S)VBS (stacked model) supported in all BladeCenters (except S)


( ) pp ( p )Requires an Advanced Management Module in BladeCenter to use these new blade switches in older BladeCenters

VBS = Real EvolutionPhysical Next

GenerationLogical Next

GenerationPrevious Generation


Don’t confuse this with the competition

Competition Daisy Chained 10G

Cisco Virtual Blade Switching

Cisco offers true switch virtualization

High performance clustering of

=g p g

switches

Truly acts as a single switch –for both management and data flow purposes

Physically looks similar –But…

U t h l ! flow purposes

Based on real world, proven technology!

Unproven technology!

Only 10G cross-connects!

Many unknowns in the realNo one else comes close to the flexibility and power offered by VBS

Many unknowns in the real world


VBS = True switch virtualizationProvides up to 9 to 1 reduction in switches to manage

Can stack up to 9 switches together – looks and acts like a single switchSi l IP f t kSingle IP for stackSingle login to manage all switches in the stack

More cable flexibility over any of the current gen solutionsCross-stack EtherChannelSupport for Cisco TwinGig to ease migration from GigE to 10GigEDove-tails into Cisco VSS strategy on 6500Can mix 10G and 4x1G in same stack

Can use unused 1G uplinks as optional connections for standalone servers or other devices

True plug and play on failure replacementReplacement automatically gets config from other stack membersReplacement automatically gets desired IOS from other stack members


Next-Gen Stacking Rack Switch ArchitectureSt k f S it h t

Stack PHYSwitch Fabric

Two Cables

• Stack of Switch acts as Single Switch

•Distributed L2/ MAC learning•Centralized L3 learning

SDRAM

CPU

PHY

FlashSerial

Port ASIC

Port ASIC

10/10

• Each switch consists of• Switch Fabric• Port Asics (downlink & uplink ports) 10/10

0

VBS cables

uplink ports)•Stackwise Plus (for stacking)

• One Master Switch in stack •1:N Resiliency for Master



•1:N Resiliency for Master• L2/L3 reconvergence is sub 200 msec

• High Speed StackWise Stack PHYSwitch Fabric

• High Speed StackWise Plus (64 Gbps)

• Dual counter rotating rings• Still functions if not fully wrapped (half speed)


wrapped (half speed)

High Performance, Scalable and Manageable Architecture

Cisco Catalyst Blade Switch 3000 SoftwareSoftware Packaging and OfferingsSoftware Packaging and Offerings

Advanced IP Advanced IP

IP ServicesIP Services

ServicesServices

IP Base +

IP Services + IPv6 (OSPFv3 and RIP-NG)Software Packaging Software Packaging

Aligns with CiscoAligns with Cisco®®

CatalystCatalyst®® FamilyFamily

IP BaseIP Base

IP ServicesIP Services IP Base +Comprehensive Layer 3 (OSPF, EIGRP, and BGP)

IP BaseIP Base Comprehensive Layer 2(VLANs, Spanning Tree, Trunking, QoS, Security, Management, etc.)Basic Layer 3Basic Layer 3 (Static routes, RIP, EIGRP Stub)


Software: IP Base image included in each part numberIP Services and Advanced IP Services images available only for VBS switches (extra charge)

Software: IP Base image included in each part numberIP Services and Advanced IP Services images available only for VBS switches (extra charge)

Greater design flexibilityEasier to deploy in existing network designs

Single virtual switch looks like a single physical switch to the upstreamReduces design issues with redundant L2 connections

L3 is now a practical option at the blade level!Prior to VBS, L3 was very rarely desired/utilized in blade switchesPrior to VBS, L3 was very rarely desired/utilized in blade switches

Problems with any technology that needed L2 adjacency (clustering, teaming, etc, all common in blade environments) Very difficult to manage many small L3 domains/IP subnetsAll current generation switches have this issue

With VBS, all servers in the stack get L2 adjacencyCan now have up to 56 servers in an L2 blade network, with L3 on the p ,upstream

Allows L2 adjacency to the servers with L3 on the uplinksEliminates Spanning-tree issues on the upstream


More intelligent routing protocols can now make decisions

“Multiple Deployment Options for Customers” Some Examples for Different Customer NeedsSome Examples for Different Customer Needs

1

3

2

4

1

3

2

4

1 3

2 4

3 4

Multi-NIC Scenario

1 3

2 4

Common ScenarioSingle Virtual Blade switch

Redundant ToR ScenarioSeparate VBS

Multi NIC ScenarioSame VBS for all NICs Possible 4 port

C

Multi-NIC ScenarioSeparate VBS for paired NICs e g HA VMware


Cost Effective HA More resilientp

EtherChannel = Max server bandwidth

e.g. HA VMware

Example showing L2 to servers and L3 to uplinks

Distribution Layer

L3L3 in the uplinks = no issues with Spanning Tree

Access Layer

L3

L2

L3 in the uplinks no issues with Spanning Tree


All servers in VBS stack have L2 adjacency

Example showing L2 on uplinks of CBS

Di t ib ti L

L3

Distribution Layer L2

Could use STP or Can use anyCould use STP or FlexLinks between

port channels

Can use any combination of uplinks in the

stack

Access Layer


Standards based NIC load balancingCurrent crop of switches only work well with Active/Standby

It can be done, but does not load balance well and is a nightmare to troubleshoottroubleshoot

VBS will permit true NIC aggregation using EtherChannel type support in the team

Current GenG0/1

Current GenG0/1

Virtual Blade SwitchG1/1

Virtual Blade SwitchG2/1

Teamed - Active/StandbyTeamed – EtherChannel

True Active/Active

NIC 1 NIC 2

BladeServer 1 BladeCenter

Logical NIC Interface

NIC 1 NIC 2

BladeServer 1 BladeCenter

Logical NIC Interface


Current gen – to different switchesCan not aggregate links

VBS – Stacked = same switchCan aggregate links

Cisco Catalyst Virtual Blade Switch Topology Highlighting Key BenefitsTopology Highlighting Key Benefits

Distribution LayerAccess Layer (Virtual Blade Switch)

Mix-n-match GE & 10GEGE & 10GE switches

Local Traffic doesn’t go to distribution

switch

With VSS on Cat

switch

Higher Resiliency With VSS on Cat

6K, all links utilized

Si l S it h / N d

Resiliency with

Etherchannel

Only Cisco provides


Single Switch / Node (for Spanning Tree or

Layer 3 or Management)

Greater Server BW –via Active-Active

Server Connectivity

Only Cisco provides maximum flexibility

and scalability!

“Operational Consistency” for Customers Achieving Lower TCOAchieving Lower TCO

Customer BenefitEliminates retraining of IT staff

Achieves efficiency from consistent management and

Cisco® GOLD and EEMSimilar Diagnostics and Cisco® GOLD and EEMSimilar Diagnostics and

diagnostics

Lowers OpEx

TroubleshootingTroubleshooting

Consistent End-to-End FeaturesSame Services for ApplicationsConsistent End-to-End FeaturesSame Services for Applications

Cisco Network AssistantFor Small-Scale Deployments

Cisco Network AssistantFor Small-Scale Deployments

CiscoWorks Management ToolConsistent Management Tools

CiscoWorks Management ToolConsistent Management Tools

Familiar Cisco IOS® Software CLI and SNMP MIBs Familiar Cisco IOS® Software CLI and SNMP MIBs


Eliminates Retraining CostsEliminates Retraining Costs

Command Line Interface Access

CLI access via console portIn stacked mode, all console ports connect to master switch CLI

CLI access via telnetNot available until user ID and password configured via Device Manager (http)

CLI is the tool most will use to configure these switches

CLI guides for 3750E are very close to 3XXX switches

Certain commands have been enhanced for use within a Bl d C tBladeCenter

show switch chassis-mgmt shows UUID of BC and slot of switch


Device Manager View of a Stacked SwitchAccessed by pointing your browser at the IP address of the stack

If non-stacked, point browser at IP address of individual switch

Switch selector UUID of BladeCenter

Slot in BC


CNA View of a StackConfigure CNA to point at the IP address of the stack

Good free GUI to configure various basic and advanced options


GOLD (Generic Online Diags) coming with VBS product

Online Switch Diagnostics for • Hardware Components How to Use:• Switch Interfaces

When Do you Run

• Invoke from Stack Master• Schedules on stack members• Resilient to Stack Master failure

Switch(config)#[no] diagnostic monitor interval { switch <1-9> } test { test-id | test-id-range | all } hh:mm:ss { ms <0-999> } { days <0-20> }

Health-Monitoring (Run-time) Offers Basic Diagnostic TestsTwo Types of Tests

Disruptive Tests

y

20 }

Switch#diagnostic start {switch <1:9>} test {test-num | test range |

On-Demand

• Disruptive Tests• Non-disruptive Tests

S diff ith C t 6K# g { } { | g |all | basic | non-disruptive }

Switch(config)#[no] diagnostic schedule { switch <1-9> } test

Scheduled

Some differences with Cat 6K (e.g. No Boot-up test, No Stop Command)


{ test-id | test-id-range | all } daily {hh:mm}

Embedded Event Manager with VBS productsVery Flexible & Powerful

Capability to Automate Troubleshooting & Mgmt

Program automatic actions based on events

1. Set policies (scripts) using• Applets (IOS CLI)• Tcl

2. Event Detectors watch for eventsevents

3. EEM Server is notified –carries out the actions for the policies

Feature Consistent with Cat 6KFeature Consistent with Cat 6KPotential application in Blade Environment

Notify network admin on insertion of new blade


insertion of new blade

What can EEM do for me?What can EEM do for me?Embedded Event Manager – Very powerful scripting tool!Define policies to take specific actions when IOSDefine policies to take specific actions when IOS recognizes certain events through the Event DetectorsExample: Could use in place of Trunk Failover, but more fl ibl d f lflexible and powerfulOther examples: Automate tasks, program actions to take based on certain events, write custom syslog messages,based on certain events, write custom syslog messages, send alerts or e-mail to inform operations personnelFor more information, see:

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6550/prod_qas0900aecd803b1169_ps6815_Products_Q_and_A_Item.htmlhttp://www.cisco.com/en/US/products/ps6815/products_ios_protocol_


group_home.html

AgendaAgenda






Sales and support

Primary sale of solution is via IBM and IBM partners

Can be sold by Cisco under special cases, but not the normal/desired path

Support can come from either IBM or Cisco, depending on what support contract customer has

IBM ff b i RMA d d d t tIBM offers basic RMA and more advanced contracts

Cisco can offer full SmartNet, which includes RMA and more advanced supportmore advanced support


Model and Part NumbersDescription Cisco Product ID IBM Option P/N IBM FRU P/NCisco Catalyst Switch Module 3012 for yIBM BladeCenter WS-CBS3012-IBM-I 43W4395

Cisco Catalyst Switch Module 3012 for IBM BladeCenter WS-CBS3012-IBM-F 43W4404

Cisco Catalyst Switch Module 3012 for IBM BladeCenter WS CBS3012 IBMfor IBM BladeCenter WS-CBS3012-IBM

Cisco Catalyst Switch Module 3110G for IBM BladeCenter WS-CBS3110G-S-I 41Y8523

Cisco Catalyst Switch Module 3110G for IBM BladeCenter WS-CBS3110G-S-F 41Y8519

Cisco Catalyst Switch Module 3110G for IBM BladeCenter WS-CBS3110G-SCisco Catalyst Switch Module 3110X for IBM BladeCenter WS-CBS3110X-S-I 41Y8522

Cisco Catalyst Switch Module 3110X for IBM BladeCenter WS-CBS3110X-S-F 41Y8518

Cisco Catalyst Switch Module 3110X for IBM BladeCenter WS-CBS3110X-SSW U d li f CBS3110 t IPSW Upgrade license for CBS3110 to IP Services 3110-IPS-LIC-I 43W4434

SW Upgrade license for CBS3110 to Adv IP Services 3110-AISK9-LIC-I 43W4435

Console cable (USB-RS232) 02R9365


Co so e cab e (US S 3 ) 0 9365

.5 Meter Stack Cable CAB-STK-IBM-0.5M=

1 Meter Stack Cable CAB-STK-IBM-1M= 44R8302

3 Meter Stack Cable CAB-STK-IBM-3M=

What Will Ship With Each Switch?

Each CBS 3110 (stacking solution) 1 x 1 meter stack cable

1 x USB/Serial cable

1 x Getting started CD1 x Getting started CD

E h CBS 3012 ( t ki l ti )Each CBS 3012 (non-stacking solution) 1 x USB/Serial cable

1 x Getting started CD


1 x Getting started CD

Some hardware detailsSome hardware detailsVBS has 256M RAM, 64M flash

All have 14 GigE links toward blade server

3110G and 3012 has four RJ45 GigE uplinks

3110X has one X2 slot (10G) – need to purchase x2 module separately Cisco X2 Wavelength

(nm)Cable Type Core Size

(microns)Modal Bandwidth

(MHz*km)Cable Distance*

X2-10GB-CX4 N/A CX4 (copper) N/A N/A 15m

X2-10GB-LRM 1310 MMF 62.5 500 220m

*Minimum cabling distance for -LR, -SR, -LX4, -ER modules is 2m, per IEEE 802.3ae standard, and minimum cabling f

50.050.0

400500

100m220m

X2-10GB-LR 1310 SMF G.652 - 10 km

distance for -LRM modules is 0.5m.

X2 module is hot swappable

Depending on model and feature set – much can change


g gBase feature set will probably satisfy a large majority of customers

Components of the CBS 3012

1 - Cisco Catalyst Switch Module 3012 6 – Release Latch2 - Console port (SERIAL, NOT USB)3 – Mode Button


4 – Switch LEDs5 - 10/100/1000 Ethernet ports (15, 16, 17 and 18)

Components of the CBS 3110

1 Cisco Catalyst Switch Module 3110G 6 10/100/1000 Ethernet ports1 - Cisco Catalyst Switch Module 3110G 6 - 10/100/1000 Ethernet ports2 - Cisco Catalyst Switch Module 3110X 7 - Switch LEDs3 - 10-Gigabit X2 Ethernet module slot 8 - Mode button 4 – StackWise+ (VBS) ports 9 - Console port (SERIAL NOT USB)


4 – StackWise+ (VBS) ports 9 - Console port (SERIAL, NOT USB)5 - Release latch

Faceplate View – CBS 3110 Modules

4x1G1x10GCBS 3110X CBS 3110G

USB/Serial consoleLEDs and Mode

button

4 x 1G RJ45

button

10G (X2 module installed)

Stacking ports(D t d)


(Dust caps removed)

Mode buttonThe mode button can be used for multipleThe mode button can be used for multiple purposes

Requires a small pointed object to activate

P d l

Example of the Mode button on a 3110G

Press and releaseTurns on the “!” LED on all switches in the stack, and then begins to flash the stack member number on the MBR LED on all switches in the stack for 30on the MBR LED on all switches in the stack for 30 secondsFor example, for stack member 4, it would flash the MBR LED 4 times in a row, then pause briefly, then flash it again 4 times and repeat this for 30flash it again 4 times, and repeat this for 30 seconds

Press and hold down for more then 15 or so secondsseconds

If non-stacked – Resets the switch back to factory default and reloads the switchIf stacked – Resets the entire stack back to factory


If stacked Resets the entire stack back to factory default and reloads the all switches in the stack

X2 Optics Support for CBS3110XX2 Optics Support for CBS3110X

Copper Multi-mode Fiber Single-mode Fiber

X2 X2-10GB-CX4 (15m) X2-10GB-SR – (26m)

X2-10GB-LRM – (220m)

X2-10GB-LX4 (300m)*

X2-10GB-LR – (10km)*

X2-10GB-LX4 – (300m)

LX4 and LR modules will not be supported at FCS. We are currently planning on support for these Post FCS

IBM has decided to not carry these parts, need to be ordered by Cisco or Cisco reseller


TwinGig Support for CBS3110X

Copper Multi-mode Fiber Single-mode FiberCopper Multi mode Fiber Single mode FiberSFP GLC-T=

RJ45 connector, 1000Base-T

GLC-SX-MM=

LC connector SX transceiver

GLC-LH-SM=

LC connector LX/LH transceiver1000Base T transceiver transceiver

GLC-ZX-SM=

LC connector ZX transceiver (70 km)


( )

Not supported at FCS

CBS ConnectionsPort DescriptionPorts 1 – 14* Internal Gigabit Ethernet 1000BASE-X downlink ports that

connect to the blade servers.

Ports 15 to 18(CBS 3110G and 3012)

External 10/100/1000BASE-T copper Gigabit Ethernet uplink ports that support auto-MDIX and auto-negotiation.

Port 1 (X2) External 10-Gigabit Ethernet module slot for use with Ci X2 t i d lPort 1 (X2)

(CBS 3110X only) Cisco X2 transceiver modules.

Internal 100BASE-T Ethernet port

The Ethernet management port (Fa0) is used only for switch module management traffic, not for data traffic. It is connected to the aMM through the blade enclosureconnected to the aMM through the blade enclosure midplane connector. Traffic to and from this port is isolated from the switch ports. This port operates in 100/full-duplex

StackWise Plus ports – Only on CBS 3110 modules

Stacking cable ports (2).

* Th b f i t l t i d t i d b th bl d l d l S

CBS 3110 modulesConsole Port CBS USB-RS232 management port


* The number of internal ports is determined by the blade enclosure model. See the blade enclosure documentation for more information about internal port numbering.

The Stacking Cable

Comes in .5, 1 (default) and 3 meter lengths

Will not stack with 3750E’s

Can only stack between other same endor blade s itches

Ordered from Cisco or Cisco partner:CAB-STK-E-0.5M= (0.5-meter cable)

Keyed, can only be

same-vendor blade switches

IBM will RMA defective 1M stack cable, but does not sell any cables

( )CAB-STK-E-1M= (1-meter cable)CAB-STK-E-3M= (3-meter cable)

inserted one way

Locking mechanism, Use pull tab to remove cable

( d tl


controlled by pull tab(grasp and gently pull tab straight

back)

Stacking ConsiderationsThe stack cables are different then the cables on current 3750E switches

Can not stack blade switches to non-blade switches (different connectors)

Technically not supposed to stack between blade vendors, but this may or may not be y pp y yblocked via software control (still being considered before FCS)

You do not have to have a complete ring for operationPartial ring will provide half the throughput but will still operateg p g p p

Non-complete ring will result in a design with single points of failure

Not recommended for normal production

Maximum distances can be achieved with 3 meter cablesMaximum distances can be achieved with 3 meter cablesIf desired, every cable can be a 3 meter cable

Probably not practical as some links will have a lot of cable left over

When a new switch is added into a stack, it will reload to sync up with the stackNeed to keep this in mind as it will disrupt traffic for the reloading switch


Physical Stacking in the IBM BladeCenter

On the surface, rules for non-VBS stacked switches would seem to apply!But wait…

What makes this different than standalone stacked switches?Standalone stacked switches are not powered and controlled in pairs

Greater chance of two VBS’s being powered off at the same time, creatingGreater chance of two VBS s being powered off at the same time, creatingdouble fault in ring, and ending in separate rings under certain physical stacking

designs (leading to management and other issues)

Standalone stacked switches can be (and usually are) stacked very close together, VBS stacks by their nature of being enclosed within a blade chassis are always stretchedstacks, by their nature of being enclosed within a blade chassis, are always stretched out over the rack

VBS has greater possibility of cable distance issues with the default 1M cable shipped


Examples of good and not so goodEach offer a full ring could be built with 1 meter cables and looks similar But:

No Yes

Each offer a full ring, could be built with 1 meter cables, and looks similar – But:Certain designs could lead to a split ring if an entire BladeCenter is powered down in the stack

For example, in the 4 BC “No” stack

No Yes

BC4

p ,example, if BC 3 had power removed (or an aMM shut down all switch modules in the enclosure), you would end up with two rings, one made up of the switches in BC 1 and 2, and one made up of the switches in BC4 No Yes

BC3

and one made up of the switches in BC4. This, at a minimum would leave each stack contending for the same IP address, and remote switch management would become difficult

No YesThe “Yes” examples also have a better chance of

i t i i BC2maintaining connectivity for the servers in the event a ring does get completely split due


BC1

completely split due to multiple faults in the stack

Rules for Stack PlanningScalable/Robust stacking design example for IBM BladeCentersScalable/Robust stacking design example for IBM BladeCenters1) Build from bottom to top, or vise versa - Easy to expand2) A single BC being powered off will not break the ring3) Can be done with all 1 meter VBS cables

Two basic rules for success:1) Stack one link between pairs in an enclosure 2) Connect the second link on each VBS to a different blade chassis (except if only a single BC, in which case, just stack the two VBS’s back to back in a 2 node ring)

Note: This is not theNote: This is not the preferred order to build the stack. The preferred order to build the stack is to start at one end (top or bottom) a o e e d ( op o bo o )and work your way up or down, but still following the two basic rules above


Note1: These are not the only possibilities, but serve as examples of good stack cable planningNote2: BC-HT may require 3M cables to go from bay 1 to bay 2 if you want dressed cables

Example Alternate Stacking Options

Off f ll i dBC4

Offers full ring and no split if a BladeCenter is powered off, but will require purchasing a 3 Meter VBS cable for the one long run (VBS ships

BC3

g ( pwith a 1 M cable)

BC2

BC1


Some base features of all 3XXX switchesSome base features of all 3XXX switchesUp to 128 instances of Spanning Tree supported4K VLAN IDs supported4K VLAN IDs supported1K simultaneous VLANs468 Routed (L3) ports per stack468 Routed (L3) ports per stackUp to 12K MAC addresses supported (depends on template applied)p pp )1K SVI’sSupport for up to 48 EtherChannel ports per switch/stackpp p p pAll of the features of the IGESM – UDLD, CDP, VTP, DTP, RPVST+, Trunk Failover, 9216 byte jumbo frames, etc, etc etc


etc, etc

Supporting Advanced OptionsCBS now offers many routing and advanced functions that the IGESM did not

OSPF, EIGRP, BGP, HSRP, IPV6 support, etc

This will be important to some customers

When discussing routing features – consider engaging your Cisco Systems Engineer

Most server teams will have no clue about this stuff and it will be a lotMost server teams will have no clue about this stuff and it will be a lot of effort to try to educate them on why they might need it, when to use it, and how to deploy it – This is truly network team territory

E th t t k tEngage the customers network team

Engage Cisco field Systems Engineers or Advanced Services


Software Licenses – Basic DifferencesSoftware Licenses Basic Differences

Feature IP Base IP Services Adv IPS

Standard L2+ Feature Set X X X

IP Source Guard and Dynamic ARP Inspection

X X Xp

Rip/Static Routing, EIGRP Stub X X X

IPv6 Manageability X X X

Multicast Routing OSPF BGP EIGRP X XMulticast Routing, OSPF, BGP, EIGRP X X

WCCP X X

IPv6 Routing Xg

IPv6 ACLs X

All come default with IP Base, other licenses can be added as desired


All come default with IP Base, other licenses can be added as desired

Some Comparisons Between Code BasesIP Advanced IP

Policy Based Routing (PBR) No Yes Yes

OSPF No Yes Yes

Feature Name IP Base Services Services

OSPF Fast Convergence No Yes Yes

BGP 4 No Yes Yes

Enhanced IGRP (EIGRP) No Yes Yes

EIGRP St b R ti Y N NEIGRP-Stub Routing Yes No No

EIGRP to Multi VRF CE (VRF Lite) No Yes Yes

RIPv1, RIPv2, Static Routing Yes Yes Yes

Routed Jumbo Frame Support Yes Yes YesRouted Jumbo Frame Support Yes Yes Yes

Multi VRF CE (VRF Lite – 26 VRFs) No Yes Yes

VRF Aware IP Directed Broadcast No Yes Yes

Inter-VLAN Routing Yes Yes YesInter VLAN Routing Yes Yes Yes

Equal Cost Routing Yes Yes Yes

DHCP Snooping Yes Yes Yes

DHCP Option 82 Data Insertion Yes Yes Yes


p

DHCP Option 82 Pass Through Yes Yes Yes

Some Comparisons Between Code Bases (cont)

Feature Name IP BaseIP

ServicesAdvanced IP

Services

DHCP Option 82 - Configurable Remote ID and Circuit ID Yes Yes Yes

DHCP Snooping Statistics and SYSLOG Yes Yes Yes

CEF - Cisco Express Forwarding Yes Yes Yes

PIM (SM,DM, SDM) Yes Yes Yes

DVMRP Tunneling No Yes Yes

NSF Awareness No Yes Yes

Fallback Bridging for Non-IP Traffic between VLANs No Yes Yes

WCCPV2 No Yes Yes

PIM Stub in IP Base Yes Yes YesVRF Aware IP Services (ARP, ping, traceroute, SNMP, syslog, HSRP, IP SLAs, TFTP, FTP, GLBP, uRPF) Note: GLBP and URPF available with E Series only No Yes Yes


IPV6 featuresIP IP Advanced IP

Feature NameIP

BaseIP

ServicesAdvanced IP

Services

IPv6 Routing - Unicast Routing “RIPng” and “OSPFv3” No No Yes

IPv6 Neighbor Discovery Throttingly No No YesIPv6 Neighbor Discovery Throttingly No No Yes

IPv6 MLDv1 & v2 Snooping Yes Yes Yes

IPv6 Host support (- IPv6 support: Addressing; IPv6: Option processing, Fragmentation, ICMPv6 TCP/UDP over IPv6; Applications: Ping/Traceroute/VTY/SSH/TFTP SNMPICMPv6, TCP/UDP over IPv6; Applications: Ping/Traceroute/VTY/SSH/TFTP, SNMP for IPv6 objects) Yes Yes Yes

IPv6 ACLs No No Yes

IPv6 Management Services (CDP over IPv6, SSHv2 over IPv6) Yes Yes Yes

EIGRP for IPv6 No No Yes

HTTP, HTTP(s) over IPv6 Yes Yes Yes

SNMP over IPv6 Yes Yes Yes

SysLog over IPv6 Yes Yes Yes

IPv6 Stateless Auto Config Yes Yes Yes

DHCP-based Auto Config (Auto Install) and Image download Yes Yes Yes


Switch Resources Controlled by TemplatesTemplates let you tune the switch to the environment – reduces wasted resourcesUses Switch Database Management (SDM) Templates to allocate resources depending on desired environmentdepending on desired environment


PVLAN – Cisco-only feature

Benefit A Private VLAN is a way to provide layer 2 isolation between target hosts in the same subnet … (segmentation of IP space can waste lot of addresses)

Deployment Scenario: Servers belonging to different departments or customers can be isolated inside the same blade chassisinside the same blade chassis

Promiscuous portCan communicate with all interfaces, including the isolated and community ports within a PVLANa PVLAN.

Isolated portAn isolated port has complete Layer 2 separation from the other ports within the same PVLAN, but not from the promiscuous ports. PVLANs block all traffic to isolated ports except traffic from promiscuous ports. Traffic from isolated port is forwarded only to promiscuous ports.

Community portCommunicate among themselves and with their promiscuous ports. These interfaces are separated at Layer 2 from all other interfaces in other communities or isolated ports within their PVLAN

For more info on PVLAN:http://www.cisco.com/en/US/tech/tk389/tk814/tk840/tsd_technology_support_sub-

within their PVLAN.


p _ gy_ pp _protocol_home.html

What is Flexlink?

An alternative to Spanning Tree for redundant L2 connections

Provides link level redundancy without Spanning Tree Protocols

STP is automatically disabled on Flexlink ports

Does not need to be configured on both sides

Fast convergence time – can be less then 50msSpeed of convergence not impacted by number of VLANs/STP instances running on linkg

Configured in pairs (pair of physical ports or pair of EtherChannel ports)

Configured using switchport backup interface xyz command on interface

Use show interface switchport backup command to view


Use show interface switchport backup command to view

Considerations for EtherChannel on uplinksConsiderations for EtherChannel on uplinksStacking virtualizes the physical switches providing more options then previously possible

This does not mean that you can hook anything to anything and expect it to work

BladeCenter

6500 in VSS mode

VSS

6500 in VSS mode

VSS Links

BladeCenter

Optimal – no single points of failure

Example of best solutionUsing virtualized upstream

Can mitigate single points of failure by i d d t

This will not work and is not supported for cross


Using virtualized upstream using redundant sups and multiple line cards

not supported for cross-stack EtherChannel

Considerations for EtherChannel on server facing ports

Only possible when switches are stackedIn stacked mode, new options are available for server teaming

802.3AD LACP802.3AD LACPGeneric Trunking (Broadcom term for static aggregation)

To support these new options, you must also configure the ports on the switch facing the serverNote that VBS’s that are NOT stacked can not use these forms of teamingforms of teaming


VMware ESX and Trunking (IEEE 802.1Q)When 802.1Q trunking to ESX virtual switches

Need to address Native VLAN operation between CBS and ESXESX t ll VLAN h i d t1ESX tags all VLANs when using dot1qCisco leaves “native” VLAN untagged by default for dot1qResult: ESX VMs unable to use the Native VLAN to communicateResult: ESX VMs unable to use the Native VLAN to communicate

Solution when trunking between CBS and ESX vSwitchesDo not use the Native VLAN for a VLAN in ESX For example, if Native VLAN on CBS is 1, do not use VLAN 1 in ESXSet Native VLAN to unused VLAN on CBSswitch(config if)#switchport trunk native vlan x (x = unused VLAN)switch(config-if)#switchport trunk native vlan x (x = unused VLAN)Tag the Native VLAN (global command so need to make sure ALL

trunked interfaces support and are configured for this on both sides)


switch(config)#vlan dot1q tag native

Some Important things to considerM t l h Ad d M t M d l t d t t CBSMust always have Advanced Management Module at proper code to support CBS

The non-Advanced MM that came with the original BC E will not work

When switches are un-stacked, fa0 interface is up and management of switch is via MM uplink by defaultMM uplink by default

You can manually override this by setting switch to Protected ModeIn PM, switch expects to be managed over its own uplinks, not the MMPM i th tti PM d f IGESM i ilPM requires other settings – PM doc for IGESM similarhttp://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/WP101034

As soon as two switches are stacked, switch management via the MM is no longer possible for stack memberspossible for stack members

Fa0 interface gets shut down on all members of the stack and can not be manually brought up

Certain traffic (SoL and cKVM) can still use the fa0 interfaceMust configure management VLAN over uplinksMust put management IP on VLAN interface of switch

When plugging in to the console port of a stacked switch, console session is actually


p gg g p , yto the stack master, regardless of what console port used

Some things to consider for users migrating from IGESMs to 3XXX

Currently no SFP version of IBM 3XXX availableCurrently no SFP version of IBM 3XXX availablePorts are numbered differently

IGESM ports named g0/XCBS ports named gY/0/X – Where Y = switch number in the stack

For non-stacked (CBS 3012), will always be g1/0/X3XXX has no default user name and passwordp

Must set via console port or http session prior to telnet3XXX defaults to VTP Server mode (IGESM set to VTP Transparent)IGESM management ports 15/16 replaced with 3XXX port Fa0IGESM management ports 15/16 replaced with 3XXX port Fa0

IP address set on MM GUI goes on the Fa0 interfaceYou can not associated a VLAN with Fa0Gi E li k t CBS b i t 15GigE uplink ports on CBS now begin at 15

IGESM uplinks = g0/17 through g0/20CBS 3110G and CBS 3012 GigE uplinks = gX/0/15 through gX/0/18


CBS 3110X Ten Gigabit port = teX/0/1

CBS Support in the BladeCenter Epp

CBS 3110G, CBS 3110X and CBS 3012 supported in bays 1, 2, 3 and 4

Using in bay 3 and 4 requires appropriate

13

Using in bay 3 and 4 requires appropriate Ethernet daughter card on server

BladeCenter E needs to have an Advanced MM to support CBS switches

Up to 4 CBS switches per BladeCenter EAny combinations of CBS’s in bays 1 – 4

24


CBS Support in the BladeCenter Tpp


Using in bay 3 and 4 requires appropriate

12


BladeCenter T needs to have an aMM (AKA cMM2 on BC-T) to support CBS switchesswitches

Up to 4 CBS switches per BladeCenter TAny combinations of CBS’s in bays 1 – 4

34


Basic BladeCenter E and T Port Connections for CBS


CBS Support in the BladeCenter H

CBS 3110G, CBS 3110X and CBS 3012 d i b 1 2 3 d 4supported in bays 1, 2, 3 and 4

Using in bay 3 and 4 requires appropriate Ethernet daughter card on serverCBS 3012 i l t d i b 7/8CBS 3012 is also supported in bays 7/8 or bay 9/10, if an MSIM is installed and appropriate daughter card on the serverAt this time, each MSIM only supports aAt this time, each MSIM only supports a single 3012 – Working on getting support for 3110 in MSIM as soon as possibleUp to 6 CBS switches per BladeCenter H

Currently any combinations of CBS’s in bays 1 – 4 and a 3012 in each MSIM if installed


CBS Support in the BladeCenter HT



CBS 3012 is also supported in bays 7/8 or b 9/10 if MSIM i i t ll d dbay 9/10, if an MSIM is installed and appropriate daughter card on the server

At this time, each MSIM only supports a single 3012 – looking for 3012 support as g g ppsoon as possible

Up to 6 CBS switches per BladeCenter HTCurrently any combinations of CBS’s in bays 1

4 and a 3012 in each MSIM if installed– 4 and a 3012 in each MSIM if installed


Basic BladeCenter H and HT Port Connections for CBS

Example CBS3110G

ConnectionsBladeCenter HT

Bays 1 to 4BladeCenter H

Bays 1 to 4

4 uplink ports10/100/1000 Mbps RJ45for external network connections

Connections

gX/0/1gX/0/2gX/0/3gX/0/4gX/0/5gX/0/6

gX/0/15gX/0/16gX/0/17gX/0/18

BladeServerSlot1BladeServerSlot2BladeServerSlot3BladeServerSlot4BladeServerSlot5Bl d S Sl t6

Bays 7 and 9 Bays 7 and 9BladeServerSlot1BladeServerSlot2BladeServerSlot3BladeServerSlot4BladeServerSlot5Bl d S Sl t6

On BC HT, optional interswitch link interposer gX/0/6

gX/0/7gX/0/8gX/0/9gX/0/10gX/0/11gX/0/12gX/0/13

BladeServerSlot6BladeServerSlot7BladeServerSlot8BladeServerSlot9

BladeServerSlot10BladeServerSlot11BladeServerSlot12

Not used

Stack 1Stack 2

To other stack

members



interswitch link interposer can be installed that ties ports 13 and 14 in bay 1 to 13 and 14 in bay 2 (same for bays 3 and 4). This is not supported with Cisco switches

1 portUSB-RS232 Console Port

gX/0/13gX/0/14

Fa0

Not usedNot used

aMM1aMM2

Serial

X in X/0/Y = Stack member numberf ( / / )

BladeServerSlot13BladeServerSlot14

SwBay 1 and 2 - Reserved for Ethernet type connectionsSwBay 3 and 4 - depends on daughter card on serverSwBay 7 and 9 – depends on daughter card on server* * For bays 7 and 9 you will need an MSIM to support

Cisco switches


CBS3012 has same connections as 3110G, but no Stack ports

CBS3110X has same connections as 3110G except the uplinks (gX/0/15 - 18) are replaced with a single 10GE (teX/0/1)

If not stacked, this will be 1 (1/0/Y)This will always be 1 on CBS3012

For bays 7 and 9, you will need an MSIM to support CBS and only the CBS 3012 is supported in MSIM

Example of CBS3110G and X Connectivity in Bays 1 and 2 of BladeCenter H


CBS Support in the BladeCenter S

Only CBS 3012 supported in BladeCenter S

Bay 1 connects to both onboard NICs on each server

Using in bay 3 and 4 requires1 3


Bay 3 connects to NIC 1 on serversy

Bay 4 connects to NIC 2 on servers

Currently bay 2 is reserved for future use

2 4

use

Up to 3 CBS 3012 switches per BladeCenter S


Basic BladeCenter S Port Connections for CBSExample

Cisco Systems CBS3012

ConnectionsBladeCenter S

Bay 1 and 2BladeCenter S Bays 3 and 4

BladeServerSlot1(a)BladeServerSlot1(b)BladeServerSlot2(a)BladeServerSlot2(b)Bl d S Sl t3( )

Bay 1 and 2Bays 3 and 4

12345

BladeServerSlot1BladeServerSlot2BladeServerSlot3BladeServerSlot4Bl d S Sl t5

4 uplink ports 10/100/1000 Mbps RJ45for external network connections

g1/0/1g1/0/2g1/0/3g1/0/41/0/5

g1/0/15g1/0/16g1/0/17g1/0/18

BladeServerSlot3(a)BladeServerSlot3(b)BladeServerSlot4(a)BladeServerSlot4(b)BladeServerSlot5(a)BladeServerSlot5(b)

56789

10

BladeServerSlot5BladeServerSlot6

Not ConnectedNot ConnectedNot ConnectedNot Connected

g1/0/5g1/0/6g1/0/7g1/0/8g1/0/9g1/0/10BladeServerSlot5(b)

BladeServerSlot6(a)BladeServerSlot6(b)

(SAS) Sw Bay 3(SAS) Sw Bay 4

1011121314

Not ConnectedNot ConnectedNot ConnectedNot ConnectedNot Connected 1 port

USB-RS232

g1/0/10g1/0/11g1/0/12g1/0/13g1/0/14

SerialaMM1 Console

PortFa0

Serial

CBS3110G and CBS3110X not supported in BladeCenter S

SwBay 1 Reserved for EthernetSwBay 2 = futureSwBay 3 and 4 = depends on daughter card on server


pp

If Ethernet switches in bays 3 and 4, ports 13 and 14 from bay 1 and 2 have no connectivityPort 13 and 14 from bay 1 and 2 are only used if SAS controllers are present in bay 3 and/or 4

BladeCenter S – Ethernet I/O connectivityVery different from otherVery different from other BladeCenters

Both on board NICs go to same switch bay (bay 1) in

pairs (Server 1 gets ports 1 and 2, server 3 gets

ports 3 and 4, etc

CBS 3012 is the only CBS switch slated for support in the BladeCenter S

Server daughter cards go to bays 3 and 4 and getbays 3 and 4, and get

sequential port numbering

S it h b 2 i tl d fi d


Switch bay 2 is currently defined as future but should have connections similar to bay 1

AgendaAgenda






3XXX Manuals and Other DocsCisco Software Activation and Compatibility Document

Instructions on applying license fileshttp://www cisco com/en/US/docs/switches/lan/catalyst3750e 3560e/softhttp://www.cisco.com/en/US/docs/switches/lan/catalyst3750e_3560e/software/release/12.2_40_se/configuration/note/swactivn.html

3XXX Software Configuration GuideTo be added

3XXX Command ReferenceTo be addedTo be added

3XXX Message Guide (all error messages)To be added

3XXX Install GuideTo be added


3XXX Getting Started GuideTo be added

Links to More Information3750E stacking white paper

Not exactly VBS, but close and good resource on operation of stackhttp://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps5023/prod_white_paper09186a00801b096a.html

3750E feature comparison/supportCompare 3750E IP Base, IP Services and Advanced IP Serviceshttp://www.cisco.com/en/US/prod/switches/ps5718/ps7077/networking_solutions_products_genericcontent0900aecd805f0eaf.html

CBS Code Download and Release NotesRequires CCO ID (free)

Go to cisco.com->Support->Download Software->Switches Software->Blade Switches->…Blade Switch 3000 Series for IBM

Q&A for 3750EQhttp://www.cisco.com/en/US/products/ps7077/products_qanda_item0900aecd805bbea5.shtmlLink to more information on Generic OnLine Diagnostics (GOLD)

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd801e659f htmlf.html

Link to more information on Embedded Device Manager (EMM)http://www.cisco.com/en/US/products/ps6815/products_ios_protocol_group_home.html

Link to Cisco pages on 3XXX switches


(need to add when available)

Important general linksWhere to go to get helpWhere to go to get helpNetpro forums can be used to post questions (Select Server Networking)

http://forum.cisco.com/eforum/servlet/NetProf;jsessionid=C70A9FD6A9BB7250361055D6812E4F75.SJ1B?page=Data_Center_discussion

Cisco Network Assistant 5 3Cisco Network Assistant 5.3GUI config tool for IGESMhttp://www.cisco.com/go/cna

CiscoView Package DownloadTo be added

CiscoWorks IDU's to support the IGESM To be added

Link to VLAN best practicesLink to VLAN best practiceshttp://www.cisco.com/en/US/products/hw/switches/ps708/products_white_paper09186a008013159f.shtml

Link to BladeCenter troubleshooting documentshttp://www-304.ibm.com/jct01004c/systems/support/supportsite.wss/docdisplay?lndocid=MIGR-57086&brandind=5000020

Free TFTP Server (SolarWinds) for upgrading IOShttp://www.solarwinds.net/Download-Tools.htm

Ethereal/Wireshark – Open source network sniffing tool


http://www.wireshark.orghttp://www.ethereal.com


Date post:	08-Jul-2018
Category:	Documents
Upload:	vanminh
View:	219 times
Download:	0 times

Matthew Slavin – mslavin@cisco - e-TechServices to Cisco CBS switches in the IBM BladeCenter...

Documents