Nexus FamilyVirtual Port Channel
Cisco Tech-Know DayFrankfurt 2009
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 1
Dieter HadwigerSystems Engineer Team Finance Germany
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 2
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
vPC Feature Overview & Terminology
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 3
� Allow a single device to use a port channel across two upstream switches
� Eliminate STP blocked ports� Uses all available uplink bandwidth Logical Topology without vPC
Feature Overview & TerminologyvPC Definition
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 4
bandwidth� Dual-homed server operate in active-active mode
� Provide fast convergence upon link/device failure
� Reduce CAPEX and OPEX� Available on current and future hardware for M1 and D1 generation cards.
Logical Topology with vPC
� vPC peer – a vPC switch, one of a pair� vPC member port – one of a set of ports (port channels) that form a vPC
� vPC – the combined port channel between the vPC peers and the downstream device
� vPC peer-link – Link used to synchronize state between vPC peer devices, must be 10GbE
� vPC peer-keepalive link – the keepalive vPC peer
vPC peer-keepalive link
CFS protocol
vPC peer-link
Feature Overview & Terminology vPC Terminology
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 5
� vPC peer-keepalive link – the keepalive link between vPC peer devices, i.e., backup to the vPC peer-link
� vPC VLAN – one of the VLANs carried over the peer-link and used to communicate via vPC with a peer device.
� non-vPC VLAN – One of the STP VLANs not carried over the peer-link
� CFS – Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPC peer devices
vPCnon-vPC device
vPC member port
vPCvPC
member port
vPC Design Guidance & Best Practices
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 6
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 7
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
Building a vPC DomainConfiguration Steps
Following steps are needed to build a vPC (Order does Matter!)1. Configure globally a vPC domain on both vPC devices2. Configure a Peer-keepalive link on both vPC peer switches (make sure is operational)
NOTE: When a vPC domain is configured the keepalive must be operational to allow a vPC domain to successfully form.
3. Configure (or reuse) an interconnecting port-channel between the vPC peer switches4. Configure the inter-switch channel as Peer-link on both vPC devices (make sure is
operational)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 8
operational)5. Configure (or reuse) Port-channels to dual-attached devices6. Configure a unique logical vPC and join port-channels across different vPC peers
vPC vPC member port
vPC peer-keepalive link
vPC peer-link
Standalone Port-channel
vPC peer
vPC Configuration Commands� configure vPC, and start the peer-keepalive link on both peers:(config)# feature vpc(config)# vpc domain 1(config-vpc-domain)# peer-keepalive destination x.x.x.x source
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 9
(config-vpc-domain)# peer-keepalive destination x.x.x.x source y.y.y.y vrf management(conifg)# int port-channel 10(config-int)# vpc peer-link
� Move any port-channels into appropriate vPC groups(config)# int port-channel 20(config-int)# vpc 20
Building a vPC DomainPeer Link
� Definition:�Standard 802.1Q Trunk�Can Carry vPC and non vPC VLANs*�Carries Cisco Fabric Services messages (tagged as CoS=4 for reliable communication)
�Carries flooded traffic from a vPC peer�Carries STP BPDUs, HSRP Hellos, IGMP updates, etc.
vPC peer-link
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 10
�Carries STP BPDUs, HSRP Hellos, IGMP updates, etc.� Requirements:
�Member ports must be 10GE interfaces one of the N7K-M132XP-12 modules
�Peer-link are point-to-point. No other device should be inserted between the vPC peers.
� Recommendations (strong ones!)�Minimum 2x 10GbE ports on separate cards for best resiliency.�Dedicated 10GbE ports (not shared mode ports)� use udld on vpc peer links *It is Best Practice to split vPC and non-vPC
VLANs on different Inter-switch Port-Channels.
� Common Nexus 7000 configuration:1x 10G, 7x 1G cards
� vPC recommendation is 2 10G cards� Potential problem occurs if Nexus 7000 is L3 boundary with single 10G card
Building a vPC DomainPeer Link with Single 10G Module
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 11
single 10G card� Use Object Tracking Feature available in 4.2� More information on CCO:http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-os/interfaces/configuration/guide/if_vPC.html#wp1529488
Scenario:� vPC deployments with a single N7K-M132XP-12 card, where core and peer-link interfaces are localized on the same card.
� This scenario is vulnerable to access-layer isolation if the 10GE card fails on the primary vPC.
e1/…e1/…
e1/…e1/…
e1/… e1/… e1/… e1/…vPC PLL3
L2
Building a vPC DomainPeer Link with Single 10G Module – Object Tracking
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 12
the primary vPC.vPC Object Tracking Solution:� Leverages object tracking capability in vPC (new CLI commands are added).
� Peer-link and Core interfaces are tracked as a list of boolean objects.
� vPC object tracking suspends vPCs on the impaired device, so traffic can get diverted over the remaining vPC peer.
e1/… e1/…
vPCPrimary
e2/… e2/… vPCSecondary
vPC PKLL2
rhs-7k-1(config-vpc-domain)# track <object>
Building a vPC DomainCisco Fabric Services (CFS)
� Definition/Uses:�Configuration validation/comparison�MAC member port synchronization�vPC member port status�STP Management�HSRP and IGMP snooping synchronization�vPC status
CFS Messaging
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 13
�vPC status� Characteristics:
�Transparently enabled with vPC features�CFS messages encapsulated in standard Ethernet frames delivered between peers exclusively on the peer-link
�Cisco Fabric Services messages are tagged as CoS=4 for reliable communication.
�Based on CFS from MDS product development�Many years in service, robust protocol
Building a vPC DomainPeer-Keepalive (1 of 2)
� Definition:�Heartbeat between vPC peers�Active/Active (no Peer-Link) detection�Messages sent on 2 second interval�3 second hold timeout on peer-link loss�Fault Tolerant terminology is specific to VSS and deprecated in vPC.
� Packet Structure:
vPC peer-keepalive link
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 14
�UDP message on port 3200, 96 bytes long (32 byte payload), includes version, time stamp, local and remote IPs, and domain ID.
�Keepalive messages can be captured and displayed using the onboard Wireshark Toolkit.
� Recommendations:�Should be a dedicated VRF and link (1Gb is adequate)�Should NOT be routed over the Peer-Link�Can optionally use the mgmt0 interface (along with management traffic)�As last resort, can be routed over L3 infrastructure
Building a vPC DomainPeer-Keepalive (2 of 2)
Cautions/Additional Recommendations:� When using supervisor management interfaces to carry the vPC peer-keepalive, do not connect them back to back between the two switches.
� Only one management port will be active a given point in time and a supervisor switchover may break keep-alive connectivity
� Use the management interface only if you have an out-of-band management network (management switch in between).
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 15
management network (management switch in between).
vPC1 vPC2
vPC_PL
Management Network Standby Management
InterfaceActive Management Interface
Management Switch
vPC_PKvPC_PK
� Definition:�Port-channel member of a vPC peer.
� Requirements:�Configuration needs to match other vPCpeer’s member port config.
Building a vPC DomainvPC Member Port
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 16
�In case of inconsistency a VLAN or the entire port-channel may suspend (i.e. MTU mismatch, inconsistent set of Vlans, values and config).
�Number of member ports on both vPCpeers is not required to match.
�Up to 8 active ports between both vPCpeers (16-way port-channel can be build with multi-layer vPC)
vPC member portvPC
member port
� vPC works seamlessly in any VDC based environment.� One vPC domain per VDC is supported, up to the maximum number of VDCs supported in the system.
� It is still necessary to have a separate vPC peer-link and vPC Peer-Keepalive Link infrastructure for each VDC deployed.
Can vPC run between VDCs on the same switch?
Building a vPC DomainVDC Interaction
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 17
Can vPC run between VDCs on the same switch?� This scenario should technically work, but it is NOT officially supported and has not been extensively tested by our QA team.
� Could be useful for Demo or hands on, but It is NOT recommended for production environments. Will consolidate redundant points on the same box with VDCs (e.g. whole aggregation layer on a box) and introduce a single point of failure.
� ISSU will NOT work in this configuration, because the vPC devices can NOT be independently upgraded.
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 18
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
Attaching to a vPC domainThe One and Only Rule…
ALWAYSdual attach
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 19
dual attach devices to a vPC
Domain!!!
� Definition:�Port-channel for devices for devices dual-attached to the vPC pair.
�Provides local load balancing for port-channel members
�STANDARD 802.3ad port channel� Access Device Requirements
Attaching to a vPC DomainIEEE 802.3ad and LACP
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 20
� Access Device Requirements�STANDARD 802.3ad capability�LACP Optional
� Recommendations:� Use LACP when available for better failover and mis-configuration protection (config consistency check)
vPC member port
vPC
RegularPort-channel port
Attaching to a vPC Domain”My device can’t be dual attached!”
Recommendations (in order of preference):1. ALWAYS try to dual attach devices using vPC (not applicable for routed links).
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Ensures full redundant active/active paths through vPC.CONS: None
2. If (1) is not an option – connect the device via a vPC attached access switch (could use VDC to create a “virtual access switch”). PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Availability limited by the access switch failure.
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 21* VLAN that is NOT part of any vPC and not present on vPC peer-link
CONS: Need for an additional access switch or need to use one of the available VDCs. Additional administrative burden to configure/manage the physical/Virtual Device3. If (2) is not an option – connect device directly to (primary) vPC peer in a non-vPC VLAN* and provide for a separate interconnecting port-channel between the two vPC peers.
PROS: Traffic diverted on a secondary path in case of peer-link failover CONS: Need to configure and manage additional ports (i.e. port-channel) between the Nexus 7000 devices.
4. If (3) is not an option – connect device directly to (primary) vPC peer in a vPC VLANPROS: Easy deployment CONS: VERY BAD. Bound to vPC roles (no role preemption in vPC) , Full Isolation on peer-link failure when attached vPC toggles to a secondary vPC role.
Attaching to a vPC DomainvPC and non-vPC VLANs (i.e. single attached .. )
SSP P
2. Attached via VDC/Secondary Switch
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 22
Orphan PortsOrphan PortsS
SPP
1. Dual Attached 2. Attached via VDC/Secondary Switch
3. Secondary ISL Port-Channel 4. Single Attached to vPC Device
Primary vPCSecondary vPCS
P
Attaching to a vPC Domain”My device only does STP!”
Recommendations (in order of preference):1. ALWAYS try dual attach devices using vPC
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Ensures full redundant active/active paths through vPC.CONS: None
2. If (1) is not an option – connect the device via two independent links using STP. Use non-vPC VLANs ONLY on the STP switch.*PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 23* Run the same STP mode as the vPC domain. Enable portfast/port type edge on host facing ports
PROS: Ensures minimal disruption in case of peer-link failover and consistent behavior with vPC dual-active scenarios. Ensures full redundant Active/Active paths on vPC VLANs.CONS: Requires an additional STP port-channel between the vPC devices. Operational burden in provisioning and configuring separate STP and vPC VLAN domains. Only Active/Standby paths on STP VLANs.
3. If (2) is not an option – connect the device via two independent links using STP. (Use vPC VLANs on this switch)PROS: Simplify VLAN provisioning and does not require allocation of an additional 10GE port-channel. CONS: STP and vPC devices may not be able to communicate each other in certain failure scenarios (i.e. when STP Root and vPC primary device do not overlap). All VLANs carried over the peer-link may suspend until the two adjacency forms and vPC is fully synchronized".
Attaching to a vPC DomainvPC and non-vPC VLANs (STP/vPC Hybrid)
SS
PP
PRSR
Non vPC port-channel
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 24
SP
1. All devices Dual Attached via vPC 2. Separate vPC and STP VLANs
3. Overlapping vPC and STP VLANs
Primary vPCSecondary vPCPrimary STP RootSecondary STP Root
SP
SRPR
SR PR
� Multi-Layer vPC can join 8 active ports port-channels in a unique 16-way port-channel*
� vPC peer side load-balancing is LOCAL to the peer
� Each vPC peer has only 8 active
Nexus 7000
16-way port
Attaching to a vPC Domain16-way Port-Channel (1 of 2)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 25
� Each vPC peer has only 8 active links, but the pair has 16 active load balanced links Nexus
5000
* Possible with any device supporting vPC/MCEC and 8-way active port-channels
16-way port channel
� 16 active ports between 8 active port-channel devices and 16 active port-channel devices?
� vPC peer side load-balancing is LOCAL to the peer
Nexus 7000
Attaching to a vPC Domain16-way Port-Channel (2 of 2)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 26
� Each vPC peer has only 8 active links, but the pair has 16 active load balanced links to the downstream device supporting 16 active ports
� D-series N7000 line cards will also support 16 way active port-channel load balancing, providing for a potential 32 way vPC port channel!
Nexus 5000
Nexus 5000 16-port port-channel support introduced in 4.1(3)N1(1a) release
16-port port-channel
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 27
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
� Use separate L3 links to hook up routers to a vPC domain is still standing.� Don’t use L2 port channel to attach routers to a vPC domain unless you can statically route to HSRP address
� If both, routed and bridged traffic is required, use individual L3 links for routed traffic and L2 port-channel for bridged traffic
Layer 3 and vPCRecommendations
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 28Router
7k1 7k2
Switch
Po1
Po2
Router
Switch
L3 ECMP
Po2
vPC view Layer 2 topology Layer 3 topology
7k1 7k2 7k1 7k27k vPC
Layer 3 and vPCWhat can happen… (1 of 3)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 29
Port-channel looks like a single L2 pipe. Hashing will decide which link to chose
Layer 3 will use ECMP for northbound traffic
7k1 7k2
R
7k1 7k2
RR
R could be any router, L3 switch or VSS
building a port-channel
1) Packet arrives at R2) R does lookup in routing table and sees 2 equal paths going north (to 7k1 & 7k2)
3) Assume it chooses 7k1 (ECMP decision)4) R now has rewrite information to which router it needs to go (router MAC 7k1 or
SPo2
Layer 3 and vPCWhat can happen… (2 of 3)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 30
router it needs to go (router MAC 7k1 or 7k2)
5) L2 lookup happens and outgoing interface is port-channel 1
6) Hashing determines which port-channel member is chosen (say to 7k2)
7) Packet is sent to 7k28) 7k2 sees that it needs to send it over the peer-link to 7k1 based on MAC address
R
7k1 7k2
Po1
9) 7k1 performs lookup and sees that it needs to send to S
10) 7k1 performs check if the frame came over peer link & is going out on a vPC.
11) Frame will only be forwarded if outgoing interface is NOT a vPC or if outgoing vPC doesn’t have active interface on
SPo2
Layer 3 and vPCWhat can happen… (3 of 3)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 31
vPC doesn’t have active interface on other vPC peer (in our example 7k2)
R
7k1 7k2
Po1
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 32
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
Spanning Tree RecommendationsOverview – STP Interoperability
� STP Uses:• Loop detection (failsafe to vPC)• Non-vPC attached device• Loop management on vPC addition/removal
� Requirements:• Needs to remain enabled, but doesn’t dictate vPC member port state
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 33
member port state• Logical ports still count, need to be aware of number of VLANs/port-channels deployed!
� Best Practices:• Not recommended to enable Bridge Assurance feature on vPC channels (i.e. no STP “network” port type)• Make sure all switches in you layer 2 domain are running with Rapid-PVST or MST (IOS default is non-rapid PVST+), to avoid slow STP convergence (30+ secs)• Remember to configure portfast (edge port-type) on host facing interfaces to avoid slow STP convergence (30+ secs)
vPCvPCSTP is running to manage loops outside of vPC’s direct domain, or before initial vPC configuration
Spanning Tree RecommendationsPort Configuration Overview
Aggregation
Data Center CoreB
L
R
N
E
BPDUguard
LoopguardRootguard
Network portEdge or portfast port type
- Normal port type
N N
Layer 3Secondary
RootSecondary
Root
HSRPSTANDBY
HSRPSTANDBY
PrimaryPrimary
HSRPACTIVEHSRPACTIVE
PrimaryvPC
SecondaryvPC
vPCDomainvPC
Domain
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 34
Access
B
RR- - -
-
-
- - -RRRRRR
--
BE
BBE
BE
Layer 2 (STP + Rootguard)
Layer 2 (STP + BPDUguard)
L
E
RootRootPrimaryRoot
PrimaryRoot
E
-
Spanning Tree RecommendationsSTP interaction on double failure
� On a peer-link and peer-keepalivesymultaneous failure, Active/Active mode may occur
� Both vPC peers forward BPDUs with same bridge IDs (NEW as of 4.2(x)), this resolves the need to disable the etherchannel guard
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 35
the need to disable the etherchannel guard feature on downstream devices
� Before 4.2(x) BPDUs are beeing sent due to dual active from both N7k with different Bridge ID which results in legacy Ethernet Guard feature (enabled by default) to kick in and disabling the portchannel -> you would be needed to disable portchannel guard feature
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 36
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
Long DistanceDC 1 DC 2
CORE
CORE
vPC domain 21vPC domain 11
Rootguard
B
F
N
E
BPDUguardBPDUfilter
Network portEdge or portfast port type
- Normal port type
R
-- F F- -
N
NN
N
Data Center InterconnectMulti-layer vPC for Agg and DCI
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 37
AGGR
ACCE
SS
Server Cluster
AGGR
ACCESS
Server Cluster
Key Recommendations� vPC Domain id for facing vPC layers should be different� No Bridge Assurance on interconnecting vPCs� BPDU Filter on the edge devices to avoid BPDU propagation� No L3 peering between DCs (i.e. L3 over vPC)
vPC domain 10 vPC domain 20
E E
- -
--
- -
-- F F-
- - --
BB
N N NN
RR
-
RRRR
RR
Nexus 7010 Nexus 7010DC-1 DC-2
vPCvPC
Data Center InterconnectEncrypted Interconnect
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 38
Nexus 7010 Nexus 7010
CTS Manual Mode (802.1AE 10GE line-rate
encryption)No ACS is required
� Validated TrustSec between Nexus 7000 connected back to back.
� Validated TrustSec across EoMPLScloud with ASR 1000 routers and Catalyst 6500s terminating EoMPLS. DCI Dark Fiber
Data Center InterconnectReferences
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 39
Catalyst 6500s terminating EoMPLS.
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 40
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
� Support for all FHRP protocols in Active/Active mode with vPC
� No additional configuration required
� Standby device communicates with vPC manager to determine L3
L2
HSRP/VRRP “Standby”: Active for
shared L3 MAC
HSRP/VRRP “Active”: Active for
shared L3 MAC
HSRP with vPCFHRP Active/Active
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 41
with vPC manager to determine if vPC peer is “Active” HSRP/VRRP peer
� General HSRP best practices still applies.
� When running active/active aggressive timers can be relaxed (i.e. 2-router vPC case)
L2
L3 CORE
Cautions:� Not recommended using HSRP link tracking in a vPC configuration� Reason: vPC will not forward a packet back on a vPC once it has crossed the peer-link, except in the case of a remote member port failure
HSRP with vPCDo NOT use Object Tracking
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 42
L2/L3 Aggregation
ACTIVE HSRP STANDBY HSRP
GW GWGW
VLAN 100 VLAN 200VLAN 100
VLAN 200
VLAN 100, 200
L3 CORE
� Use an OSPF point-to-point adjacency (or equivalent L3 protocol) between the vPC peers to establish a L3 backup path to the Core through in case of uplinks failure
� A single point-to-point VLAN/SVI will suffice to establish a L3 neighborship.
OSPF
HSRP with vPCL3 Backup Routing
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 43
L3L2
PrimaryvPC
SecondaryvPC
OSPF
OSPF
VLAN 99
Scenario:� Provide L2/L3 interconnect between L2 Pods, or between L2 attached Datacenters (i.e. sharing the same HSRP group).
� A vPC domain without an active HSRP instance in a group would not be able to forward traffic. Active Standby Listen Listen
HSRP with vPCDual L2/L3 Pod Interconnect
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 44
be able to forward traffic. Multi-layer vPC with single HSRP:� L3 on the N7K supports Active/Active on one pair, and still allows normal HSRP behavior on other pair (even across different vPCdomains we support all in one HSRPgroup)
� L3 traffic will run across Intra-pod link for non Active/Active L3 pair
Active Standby Listen Listen
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 45
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
vPC and ServicesvPC Services Integration� Services deployed as part of Catalyst 6500 Service chassis
� Investigation ongoing with standalone services (ASA, ACE)
� Appliance based services that do not support port-channel may L3
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 46
do not support port-channel may require additional peer-link connections to deal with the additional traffic forced across the peer-link
� More information will be posted as soon as more scenario are verified – keep in touch w/ your responsible Cisco SE
L3L2
vPC and ServicesCatalyst 6500 Services Chassis w. Services VDC SandwichTwo Nexus 7000 Virtual Device Contexts used to “sandwich”
services between virtual switching layers• Layer-2 switching in Services Chassis with transparent
services• Services Chassis provides Etherchannel capabilities for
interaction with vPC• vPC running in both VDC pairs to provide Etherchannel for
both inside and outside interfaces to Services ChassisDesign considerations:
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 47
Design considerations:• Access switches requiring services are connected to sub-
aggregation VDC• Access switches not requiring services may be connected to
aggregation VDC• May be extended to support multiple virtualized service
contexts by using multiple VRF instances in the sub-aggregation VDC
Design Cautions:• Be aware of the Layer 3 over vPC design caveat. If Peering at
Layer 3 is required across the two vPC layers an alternative solution should be explored (i.e. using STP rather than vPC to attach service chassis)
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 48
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
Several enhancements to vPC: �vPC Object Tracking�vPC Peer-Gateway�vPC Delay Restore�Multi-layer vPC with single HSRP group�vPC unicast ARP handling
vPC Latest EnhancementsSummary
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 49
�vPC Exclude Interface-VLAN�vPC single attached device Listing�vPC Convergence and Scalability
For more details:� 4.2 Release Notes
http://www.cisco.com/en/US/docs/switches/datacenter/sw/4_2/nx-os/release/notes/42_nx-os_release_note.html#wp218085
vPC PL
vPC PKL
L3L2
Scenario:� Interoperability with non RFCcompliant features of some NAS devices (i.e. NETAPP Fast-Path or EMC IP-Reflect) � NAS device may reply to traffic using the MAC address of the sender device rather than the HSRP gateway.
Local Routing for peer router –mac Traffic
vPC Latest EnhancementsvPC Peer-Gateway for NAS interoperability
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 50
rather than the HSRP gateway.� Packet reaching vPC for the non local Router MAC address are sent across the peer-link and can be dropped if the final destination is behind another vPC.vPC Peer-Gateway Solution:� Allows a vPC switch to act as the active gateway for packets addressed to the peer router MAC (Non disruptive CLI command added in the vPC global config)
N7k(config-vpc-domain)# peer-gateway
4.2(1) vPC EnhancementsvPC Delay Restore convergence improvement
Problem/Impact:� After a vPC device reloads and come back up routing protocol needs time to reconverge. vPCs may blackholerouted traffic from access to core until layer 3 connectivity is reestablished
vPC Delay restore solution: vPC PLL3
OSPF
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 51
� Delays vPCs bringup after a vPCdevice reload (SVI bring-up timing is unchanged),
� Allows for Layer 3 routing protocols to converge for a more graceful restoration.
� Enabled by default with a vPCrestoration default timer of 30 seconds
� Timer can be tuned according to a specific layer 3 convergence baseline.
vPCPrimary
vPCSecondary
vPC PKL
L3L2
4.2(1) vPC EnhancementsvPC unicast ARP handlingProblem/Impact: � Lack of interoperability with BigIP (F5devices) using Unicast ARP requests to monitor gateway liveness
� Due to the hashing mechanism the unicast ARP requests for the HSRPvirtual IP may reach the secondary
vPC PL
vPC PKL
L3L2
StandbyHSRP
ActiveHSRP
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 52
virtual IP may reach the secondary HSRP device. If that is the case they get punted to the Sup and dropped –due to NOT the active control plane
vPC unicast ARP handling solution:� 4.2(1) achieve interoperability forwarding unicast ARP requests via the peer-link to the active HSRPinstance.
� No additional configuration Is required to enable the functionality.
vPCPrimary
vPCSecondary
vPC PKLL2
4.2(1) vPC EnhancementsvPC Exclude Interface-VLAN
Problem/Impact:� When a dual active condition is detected SVIsand vPC ports on the secondary vPC peer are suspended and therefore Single homed devices on secondary peer suffer due to loss of gateway
� Only the primary vPC peer continues data plane and control plane functionalities
vPC PL
vPC PKL
L3L2
SVI
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 53
vPC exclude interface-VLAN solution:� The vPC exclude interface-VLAN feature ensures that a configurable list of SVIs are not suspended on the secondary vPC peer
� Consequently Layer 3 connectivity is maintained even in a dual active condition for a restricted selection of interfaces
� Other option : configure separate VLAN(s) for single attached devices (recommended)
vPCPrimary
vPCSecondary
vPC PKLL2
N7K (config-vpc-domain)# dual-active exclude interface-vlan ?<1-3967,4048-4093> Set allowed interface vlans
vPC PL
vPC PKL
L3L2
4.2(1) vPC EnhancementsvPC single attached device Listing
Problem/Impact:� Single attached devices that are not connected via a vPC but still carry vPC VLANs are also known as orphan ports.
� In case of a peer-link shut or restoration, an orphan port's
Port #1 Port #2
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 54
vPCPrimary
vPCSecondary
vPC PKLrestoration, an orphan port's connectivity may be bound to the vPC failure or restoration process.
vPC single attached device listing:� For this reason, NX-OS Release 4.2(1) introduces a show command to check and list single attached devices in the system along with impacted VLANs.
N7K (config-vpc-domain)# show vpc orphan-ports
Agenda� Nexus 7000 vPC Feature Overview & Terminology� Nexus 7000 vPC Design Guidance & Best Practices
�Building a vPC domain�Attaching to a vPC domain�Layer 3 and vPC�Spanning Tree Recommendations�Data Center Interconnect (& Encryption)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 55
�Data Center Interconnect (& Encryption)�HSRP with vPC�vPC and Services�vPC latest enhancements�ISSU
� Nexus 7000 vPC Convergence and Scalability� Nexus 7000 vPC Roadmap and Reference Material� Nexus 5000 / 2000 vPC design considerations
� ISSU is still the recommended system upgrade in a multi-device vPC environment� vPC system can be independently upgraded with no disruption to traffic.� Upgrade is serialized and must be run one at a time (i.e. config lock will prevent synchronous upgrades)� Configuration is locked on “other” vPC peer during ISSU.
4.1(3) 4.1(3)4.2(1)
4.1(3)
4.2(1)
4.2(1)
In-Service Software Upgrade (ISSU)vPC System Upgrade/Downgrade
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 56
ISSU.� No card reloads or port flaps, even different releases during interim condition
Begin End Caveats4.1(x) 4.2(x) None4.2(x) 4.1(x) None
vPC Convergence &Scalability
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 57
L2/L3 Aggregation
L3 CoreNexus 7000
N7K-1 N7K-2
20 flows @1000 pps
OSPF
OSPF
4.2(1) vPC EnhancementsConvergence Topology
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 58
vPC Peer Link LACP Channel (2x10 GigE)
vPC Peer-Keepalive (GigE)
AggregationNexus 7000 vPC
L2 AccessNexus 5000
Po10
20 flows @1000 pps20 flows @1000 pps
Po20Po160
16-way port-channel 4-way port-channel
vPC on Nexus 7000Convergence Numbers- Disclaimer: without engagement
Failover case Failure Topology Convergence Time Failure Restoration
Failure of secondary vPC peer*
4.1(4)North-Bound: ~700 msSouth-Bound: ~2.5 sec
4.1(4)North-Bound: ~3 secSouth-Bound: ~3.4 sec
4.2(1)North-Bound: ~50 ms.South-Bound: ~100 ms
4.2(1)North-Bound: 100 – 900 msSouth-Bound: 1.2 -2 s
Failure of a primary vPC peer*
4.1(4)North-Bound: ~150 ms
4.1(4)North-Bound:~4.5 secs
P S
P S
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 59
North-Bound: ~150 msSouth-Bound: ~3 sec
North-Bound:~4.5 secsSouth-Bound: ~5 secs
4.2(1)North-Bound: ~50 msSouth-Bound: ~100 ms
4.2(1)North-Bound: ~400 ms-1.5 sSouth-Bound: ~1.5 s
Failover of the vPC Peer Link
4.1(4)North-Bound: ~1.3 sSouth-Bound: ~1.8 s
4.1(4)North-Bound: ~900 ms South-Bound: up to 10+ s (CSCsz88998)
4.2(1)North-Bound: 100-300 msSouth-Bound: 50-500 ms
4.2(1)North-Bound: 150 - 900 msSouth-Bound: ~ 900 ms–1.5 s
NOTE: Convergence numbers may vary depending on the specific configuration (i.e. scaled number of VLANs/SVIs or HSRP groups) and traffic patterns (i.e. L2 vs L3 flows).
P S
P S
vPC on Nexus 7000Scalability Number Improvements
Release Supported Scalability4.1(5) 192 vPC’s (2-port) with the following,
200 VLANs200 HSRP Groups40K MACs & 40K ARPs10K (S,G) w. 66 OIFs (L3 sources)3K (S,G) w. 34 OIFs (L2 sources)
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 60
3K (S,G) w. 34 OIFs (L2 sources)LatestAnkara 4.2(2a)
256 vPC’s (4-port) with the following,260 VLANs200 SVI/HSRP Groups40k MACs & 40K ARPs10K (S,G) w. 66 OIFs (L3 sources)3K (S,G) w. 64 OIFs (L2 sources)
NOTE: Supported numbers of VLANs/vPCs are NOT related to an hardware or software limit but reflect what has been currently validated by our QA (data-points). The N7k BU is planning to continuously increase these numbers as soon as new data-points become available. Please contact your responsible Cisco team if you
have particular VPC scaling requirements.
vPC Roadmap and Reference Material
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 61
Roadmap and Reference MaterialvPC Plan of ActionDisclaimer: without engagement – subject to correction
•vPC scalability, new data-point targets: 50 vPCs-2Ports and 1000 VLANs
•vPC scalability, new data-point targets: 768 vPC-2ports and 300 VLANs
•vPC scalability, new data-point targets: 2000 FEX hosts-2ports and 300 VLANs
Bogota Cairo Delhi•vPC scalability, new data-point targets: 3072 FEX hosts-2ports and 200 VLANs
Future
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 62
300 vPCs-4ports and 300 VLANs•Enhanced vPC dual Active support
•vPC over D1ports•16-port vPC on D1 modules with N5K downstream•Port-Security over vPC
VLANs•PVLANs over vPC•Config sync for vPC•vPC for FEX Host Ports
VLANs
1HCY’10 2HCY’10 1HCY’11
CCd and ECd Not CCd Not CCd Not CCd
L2/L3 AggregationNexus 7000 vPC
L3 Core
N7K-1 N7K-2
Physical Logical
Roadmap and Reference MaterialvPC/VSS Interop Test Details
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 63
Nexus 7000 vPC
L2 Access6500 VSS
E1/25
Te2/2/1
E1/26
Te1/2/1
Po10
vPC Peer Link LACP Channel (2x10 GigE)vPC Peer-Keepalive (GigE)
Po100
VSS VSL Channel (2x10 GigE)
6K-26K-1
Po100
� The following scenarios were tested:• VSS and vPC member failover and convergence• Dual active scenarios and behavior• Best practice guidelines for STP, L3 (NSF), Multicast
� Catalyst 6500/Nexus 7000 interoperability:
Roadmap and Reference MaterialvPC/VSS Interop Test Details
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 64
� Catalyst 6500/Nexus 7000 interoperability:� Enterprise Solutions Engineering:http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DC_3_0/DC-3_0_IPInfra.html
Please refer to CCO for more detailed information or refer to your CiscoSE
Datacenter designs with Nexus 5000/2000
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 65
NX-OS 4.1(3)N1(1)� Support for 512 HW (but SW allows 507 maximum) VLANs (minus number of VSANs)� Supports 12 Fabric Extenders � Supports 16 Hardware Ethernetport channels (12 Ethernet and 4 Fiber Channel supported
Fabric Ports
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 66
Fiber Channel supported concurrently as well as just 16 Ethernet Portchannels and zero FC port-channels)� Supports the use of the GEM� Supports vPC
5020 = 52 Fabric Ports & 16 Port Channels
5010 = 26 Fabric Ports & 16 Port Channels
4x10G uplinks
CoreLayer
L3L2
VSS
Nexus 2000 Fabric ExtenderNetwork Topology – Physical vs. LogicalPhysical Topology Logical Topology
CoreLayer
L3L2
VSS
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 67
FE4x10G uplinksfrom each rack
Rack-1 Rack-2 Rack-3 Rack-4 Rack-12
Servers
Nexus 5020
FEX
Rack-5
Nexus 5020
FEX FEX FEX FEX FEX
Servers
Nexus 5020Nexus 5020
12 FEX
Rack-1 Rack-N Rack-1 Rack-N
12 FEX
Fabric Extended Terminology� Fabric Links: connect Nexus 5000 to Fabric Extender (switchport mode fex-fabric)
� Host Interfaces (HIF)
n5k01
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 68
� FEX connectivity between Nexus 5000 and Nexus 2000 (FEX) can leverage either (static) pinning or port-channels
� FEX: N2148T-1GE(48x1GE + 4x10GE)
FEX100FEX101
FEX102
Port-Channeling� With Static Pinning if a fabric uplink port fails, the associated HIFs are beeing shut down
� With Port-Channeling if a fabric uplink fails then HIFsuse the remaining fabric
N5k01
A
1,2,3,4
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 69
use the remaining fabric uplinks
� Port-channeling is therecommended design method
1-48
N2k01
pinning max-links 1
Fabric Ports
Host Ports
1,2,3,4
What is Nexus 2000 Single Homed(aka Straight Through)
Nexus 2000 Straight-through deployment
n5k01max 4 “fabric links”
Typical Redundant straight-throughdeployment as of 4.0(1a)
n5k01 n5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 70
FEX100FEX101
FEX102
max 12 = 576 ports
http://www.cisco.com/en/US/partner/products/ps9670/products_installation_and_configuration_guides_list.html
Active/Standby
FEX100FEX101
FEX102 FEX120FEX121
FEX122
max 12 x 2 = 576 ports x 2
vPC peer link
nexus5k01 nexus5k02
Fault Tolerantor peer keepalivelink
mgmt0 vrf
mgmt0 mgmt0
vPC Terminology (NX-OS 4.1(3))
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 71
vPC member port
nexus5k01 nexus5k02
Peer KeepalivePeer Link/ MCT
vPC Member Port
vPC
Virtual Port-ChannelTerminology
� vPC peer – a vPC switch, one of a pair� vPC member port – one of a set of ports (port channels) that form a vPC
� vPC – the combined port channel between the vPC peers and the downstream device
� vPC peer link – Link used to synchronize state between vPC peer devices, must be 10GbE. Also carries
vPC peer keepalive link vPC peer link
vPC peer
vPC
5k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 72
be 10GbE. Also carries multicast/broadcast/flooding traffic and data traffic in case of vpc member port failure
� vPC peer keepalive link – the peer keepalive link between vPC peer switches. It is used to carry heartbeat packets
� CFS – Cisco Fabric Services protocol, used for state synchronization and configuration validation between vPCpeer devices
vPCmember port
Orphan PortOrphan PortOrphan PortOrphan Port
Virtual Port-ChannelvPC Peer Link
vPC Peer Link
� Peer Link carries both vPC data and control traffic between peer switches
� Carries any flooded and/or orphan port traffic
� Carries STP BPDUs IGMP updates, etc. 5k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 73
5020 (config)# interface port-channel 105020 (config-if)# switchport mode trunk5020 (config-if)# switchport trunk allowed <BETTER TO ALLOW ALL VLANS>5020 (config-if)# vpc peer-link5020 (config-if)# spanning-tree port type network
etc.� Carries Cisco Fabric Services messages (vPC control traffic)
� Minimum 2 x 10GbE ports
5k01 5k02
STP implementation Virtual Port-Channel vPC Roles
� Two Nexus 5000s running vPCappear as a single STP entity
� vPC Role defines which of the two vPC peers processes BPDUs
� Role matters for the behavior
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 74
Primary Role
� Role matters for the behavior with peer-link failures!
� Role is defined under the domain configuration
� Lower priority wins - if not, lower system MAC wins
Secondary Role
5k01 5k02
5k01
4+ Ports vPCs2-Ports vPCs
vPC on the Nexus 5000
5k025k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 75
access
eth2/1,2/2 eth2/3,2/4 eth2/1 eth2/2
Peer KeepalivePeer Link
vPC Member Port
vPC
Max 16 HW-Port Channel As many as the number of ports on the 5k
vPC
vPC with FEXNexus 2000 Single-homed vPC Nexus 2000 active/active
(or dual homed)
Peer-link
FT link (can be routed) FT link (can be routed)
primary
mgmt network mgmt network
mgmt0 mgmt0mgmt0 mgmt0
Peer-link
secondaryprimary secondary
Peer KeepalivePeer Link/ MCT
vPC Member Port
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 762-GigE ports host port channel
vPC
2 ports
FEX120FEX100 vPC 1 vPC 2
FEX120FEX100HIF HIF
HIF HIF
“fabric links” “fabric links” 5k01 5k02 5k01 5k02
n5k02n5k01
Nexus 2000 straight-through with vPC
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 77
max 24 FEXes = 1152 (24 x 48GE ports)
max 480 vPCs (each vPC has 2 ports)
vPC Primary vPC SecondaryPo10
5k01 5k02
Nexus 2000 dual-homed
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 78
max 12 FEXes
Host and Switch Port-channels to 5k� The 4.1(3)N1 release enables the configuration of virtual port-channels from switches connected to Nexus 5000� It also enables port-channels from servers connected redundantly to the Nexus 5000� It enables both 2-ports port-channels and 4+ ports port-channels� Maximum 16 4+ ports portchannels are vPC member ports
Mgmt network
primary secondary
5k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 79
� Maximum 16 4+ ports portchannels are possible (minus the number of FC port-channels)� Any of the 52 ports of the 5020 or the 26 ports of the 5010 can be utilized (i.e. can also use the GEM modules)
vPC member ports
2-portsswitch port channel
2-portshost port channel
4+ portsswitch port channel
4+ portshostport channel
vPC member portPeer Keepalive or FT linkvPC Peer Link aka MCT
vPC
vPC Mixed Topology equally work
Management Network
primary secondary
mgmt0 mgmt05k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 80
2-GigE ports host port channel
FEX120FEX100
FEX101 FEX121
single attached servers and/or A/S
vPC on the N7kN7k01 N7k02 N7k01 N7k02
DESIGN 1 DESIGN 2Double-sided vPC between Nexus 7000 and Nexus 5000
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 81
vPC on the N5kN5k01 N5k02 N5k01 N5k02
1 2 3 1 3
1 2 3 4 1 2 3 4
2
Max 16 Ports
vPC on the N7kN7k01 N7k02 N7k01 N7k02
DESIGN 3 DESIGN 4Double-sided vPC between Nexus 7000 and Nexus5000 and Nexus 2000
Max 16 Ports
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 82
vPC on the N5kN5k01 N5k02 N5k01 N5k02
N2k01 N2k02
1 2 3
1 2 3 4 1 2 3 4
5 6 7 8 5 6 7 8
N2k01 N2k02
1 2 3
vPC on the N7k
N7k01 N7k02
1 2 3 4
DESIGN 5 DESIGN 6
N7k01 N7k02
Double-sided vPC between Nexus 7000 and Nexus5000 and FEX A/A
Max 16 Ports
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 83
N2k01 N2k02
1 3
N5k02N5k01
1 2 3 4
vPC on the N5k
1 2 3 4
N5k02N5k01
N2k01 N2k02
1 3
16-ports Port-Channel
� Each vPC peer has only 8 active links, but the pair has 16 active load balanced links
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 84
16 x 10 GigE ports
You can still use TLB with FEX A/Ayou cannot just use 802.3ad or static port-channel with FEX A/A
vPC 1 vPC 2
Peer-linkprimary secondary
“fabric links”
5k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 85
vPC 1 vPC 2
FEX120FEX100HIF HIF
How Many Paths?
vPC
� In a typical vPC deployment, e.g. in FEX A/A you want to tune the traffic to use all the available paths.
� Remember that there are 3 components involved:
5k01 5k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 86
vPC Primary
vPC Secondary
Po10Nexus 5k which can load balance based on L2/L3/L4 information
FEX (which can only load balance based on L2/L3 information)
Teaming software (which can be configured for various load balancing options e.g. tcp connections) TLB
5k01
core1 core2
5k02
core1 core2
vPC Forwarding Behavior
5k01 5k02
vPC peer link almostunutilized
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 87
5k01
acc1 acc2 acc3
5k02
acc1 acc2 acc3
Summary Checklist� Ensure MST region is configured for the NXOS VLAN range
� Use pathcost method long� Assign roots/secondary roots as usual (regardless of primary/secondary roles)
N7k01 N7k02
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 88
primary/secondary roles)� Create a single Port-channel leveraging LACP
� Trim VLANs that are used for VSANs� Do not forget that putting a VLAN on a vPC requires that that VLAN be on the Peer-link too
� Make sure the configuration is not causing Type-1 Inconsistencies
1 2 3 4
N5k02N5k01
N2k01 N2k02
1 3
L2
L3
L3
IP Cloud
Core
AggregationvPC
OTV Inter-POD Connectivity across L3Failure Boundary Preservation
Failure Boundary
Feature Overview & TerminologyIntelligent L2 Domains POD Evolution
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 89
L2
L2 vPCAccess
Servers
vPC
vPC
STP+ vPC/VSS Cisco L2MPSTP EnhancementsBridge Assurance
NIC TeamingSimplified loop-free trees
2x Multi-pathing
16x ECMPLow Latency / Lossless
MAC ScalingOperational Flexibility
L2MP
Networkers at Cisco Live 2010 - Barcelona
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 90
Registrieren Sie sich hier: www.cisco.com/go/networkersregister
© 2009 Cisco Systems, Inc. All rights reserved. Cisco Confidential 91