Date post: | 19-Mar-2019 |
Category: |
Documents |
Upload: | truongcong |
View: | 225 times |
Download: | 0 times |
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Virtual Machine Fabric Extension (VM-FEX) Bringing the Virtual Machines Directly on the Network
BRKCOM-2005
Dan Hanson, Technical Marketing Manager, Data Center Group, CCIE #4482
2
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Agenda
Fabric Extension Technology Overview
Virtual Machine Fabric Extender (VM-FEX) Introduction
VM-FEX Operational Model
VM-FEX General Details on UCS and Nexus 5500
VM-FEX Implementation with VMware on UCS and Nexus 5500
VM-FEX Implementation with KVM on UCS
VM-FEX Implementation with Hyper-V on UCS
Easy-VMFEX tool usage
Performance Review
Summary
What This Session Will Cover
3
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Fabric Extension (FEX) Concept Virtualising the Network Port
LAN LAN Switch port extended over Fabric Extender
Lo
gic
al S
wit
ch
Switch
Switch
Legacy multi-tier architecture FEX architecture
Switch
FEX
Collapse network tiers, fewer network management points 5
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Nexus 5000 + Fabric Extender Single Access Layer
=
Distributed Modular System
+
Nexus 5000 Parent Switch
Cisco Nexus® 2000 FEX
Over 6000 production customers Over 5 million Nexus 2000 ports deployed
Distributed Modular System Nexus 2000 FEX is a Virtual Line Card to the Nexus 5000
Nexus 5000 maintains all management & configuration
No Spanning Tree between FEX & Nexus 5000
LAN
N7000/ C6500
MDS
SAN
Access Layer N5000
1 12
N2232 N2232
6
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VNTAG mimics forwarding vectors inside a switch
D: Direction, P: Unicast/Multicast, L: Loop
Policy associated with the Virtual Interface NOT port
VLAN member ship, QoS, MTU, Rate limit etc
VNTAG Ether type
Destination Virtual Interface
Source Virtual Interface ver
P
R
Application Payload
TCP IP
Ethernet VNTAG
FEX architecture
Switch
FEX
LAN
Frame
VNTAG Frame
Key Architectural Component #1: VNTAG “Intra-Chassis” Bus Header
L
D
7
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
FEX Data Forwarding
Constellation Bus had 32 byte header for fabric switching ‒ Vast majority of modular switch vendors have an internal “Tag” for fabric
communications
Originally, Centralized forwarding ASICs ‒ Line cards fed into these ASICs directly
When we needed higher performance – we added faster Switch Fabrics, and Distributed Forwarding Capabilities to system
What this really meant – adding more ASIC forwarding capacity to the system to minimize the number of devices a flow had to traverse
Revisiting Traditional Modular Switches (Example Catalyst 6500)
8
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
FEX Data Forwarding
Think the original C6k Satellite Program for VSL and RSL
The Constellation Bus now is smaller header – 6 Byte VNtag header
‒ Core to FEX technology and being standardized as 802.1BR
‒ This is NOT a 1:1 mapping to VEPA/802.1bg which is designed to offer an enhanced forwarding mechanism between peer devices via a single upstream device
Keep the ASIC counts for high performance but put them on the Central controlling switch instead of all these line cards
‒ Latency and bandwidth were more a function of the layers of ASICs to traverse in a tree – rather than the location of these ASICs (the fiber/copper paths for a packet to propagate)
Add protocols for configuration and firmware management of these remote cards (Satellite Control Protocol, Satellite Discovery Protocol)
‒ Allows us to get away from manual firmware code management per (remote) line-card
Move from Store-and-Forward behavior to Cut-Through switching to make latency actually better
Decoupling the Modular Switch
9
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
LAN
Parent Switch
SAN
FEX
Parent Switch + FEX Decoupling the Modular Switch
10
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
FEX Technology for Unified I/O
Virtual Switch Ports, Cables, and NIC Ports
Mapping of Ethernet and FC Wires over Ethernet
Service Level enforcement Multiple data types (jumbo, lossless,
FC)
Individual link-states Fewer Cables
Multiple Ethernet traffic co-exist on same cable
Fewer adapters needed Overall less power Interoperates with existing Models
Management remains constant for system admins and LAN/SAN admins
Possible to take these links further upstream for aggregation
Individual Ethernets
DCB Ethernet
Individual Storage (iSCSI, NFS, FC)
Blade Management Channels (KVM, USB, CDROM, Adapters)
11
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Extending FEX Architecture to the VMs Cascading of Fabric Extenders
Lo
gic
al S
wit
ch
Virtualized Deployment
Switch
FEX
Hypervisor vSwitch
App
OS
App
OS
App
OS
LAN
Lo
gic
al S
wit
ch
VM-FEX architecture
Switch
FEX
Hypervisor
LAN
App
OS
App
OS
App
OS
VM-FEX
Switch port extended over cascaded Fabric Extenders to the Virtual Machine
L
og
ical
Sw
itch
13
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Key Architectural Component #2: UCS VIC UCS Virtual Interface Card Family
256 PCIe devices
Devices can be vNICs or vHBAs
Each device has a corresponding switch interface
Bandwidth 2x4x10 Gb
Uses 4x10 Ether Channel, HW 40Gb Capable
vNICs/vHBAs NOT limited to 10Gb
PCIe Gen-2 x 16
Mezzanine and PCIe
14
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS VM-FEX Distributed Modular System Removing the Virtual Switching Infrastructure to a FEX
=
Distributed Modular System
VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer
VM-FEX is a Virtual Line Card to the parent switch
Parent switch maintains all management & configuration
Virtual and Physical traffic treated the same
LAN
N7000/ C6500
MDS
SAN
Access Layer UCS 6100
1 160
UCS VIC UCS VIC
App
OS
App
OS
App
OS App
OS
App
OS
App
OS
App
OS
App
OS
App
OS App
OS
App
OS
App
OS
UCS IOM UCS IOM
+
UCS Fabric Interconnect Parent Switch
Cisco UCS VIC
UCS IOM-FEX
+
15
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Nexus 5000/2000 VM-FEX Distributed Modular System Removing the Virtual Switching Infrastructure to a FEX
=
Distributed Modular System
VM-FEX: Single Virtual-Physical Access Layer Collapse virtual and physical switching into a single access layer
VM-FEX is a Virtual Line Card to the parent switch
Parent switch maintains all management & configuration
Virtual and Physical traffic treated the same
LAN
N7000/ C6500
MDS
SAN
Access Layer Nexus 5500
1 160
UCS VIC UCS VIC
App
OS
App
OS
App
OS App
OS
App
OS
App
OS
App
OS
App
OS
App
OS App
OS
App
OS
App
OS
Nexus 2000 Nexus 2000
+
Nexus 5500 Parent Switch
Cisco UCS VIC
Nexus 2000 FEX
+
16
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Software Based Virtual Access Layer Out of the Box Deployments
Physical Network
Virtual Network
Hyp
ervi
sor
Hyp
ervi
sor
VM VM VM VM VM VM VM VM
VETH
VNIC
18
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operations Model
Step 1: Preboot
‒ UCS defined PCIe devices and enumerations
‒ Host discovers PCIe devices
Pre-Boot Configuration
Hyp
ervi
sor
Hyp
ervi
sor
19
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Step 1: Preboot
‒ UCS defined PCIe devices and enumerations
‒ Host discovers PCIe devices
Step 2: Port Profile
‒ Folder of Network Policy defined
Defining “Port Profiles” on the UCS or Nexus 5000
Hyp
ervi
sor
Hyp
ervi
sor
Port Profiles Definition
WEB Apps
HR
DB
Compliance
VLAN Web VLAN HR
VLAN DB VLAN Comp
UCSM or Nexus 5500
20
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Step 1: Preboot
‒ UCS defined PCIe devices and enumerations
‒ Host discovers PCIe devices
Step 2: Port Profile
‒ Folder of Network Policy on UCS or Nexus 5500 defined
Step 3: Port Profile Export
‒ Port Profile name list exported to virtualization manager
Pushing Port Profiles to the Hypervisor System
Hyp
ervi
sor
Hyp
ervi
sor
VLAN Web VLAN HR
VLAN DB VLAN Comp
Hypervisor Manager
UCSM or Nexus 5500 exports Port Profiles
UCSM or Nexus 5500
21
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Step 1: Preboot
‒ UCS defined PCIe devices and enumerations
‒ Host discovers PCIe devices
Step 2: Port Profile
‒ Folder of Network Policy on UCS or Nexus 5500 defined
Step 3: Port Profile Export
‒ Port Profile name list exported to virtualization manager
Step 4: VM Definition
‒ Named Policy in VM
Mapping of Port Profiles to VM Virtual Adapters
Hyp
ervi
sor
Hyp
ervi
sor
VLAN Web VLAN HR
VLAN DB VLAN Comp
Hypervisor Manager
Network Manager
VM
VM VM VM
UCS or Nexus 5500 exports Port Profiles
22
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Modes of Operation Enumeration vs. Hypervisor Bypass
Emulated Mode PCIe Pass-Thru or VMDirectPath
Standard (Emulated) Mode Each VM gets a dedicated PCIe
device
~12%-15% CPU performance improvement
Appears as distributed virtual switch to hypervisor
LiveMigration supported
High Performance Mode
Co-exists with Standard mode
Bypasses Hypervisor layer
~30% improvement in I/O performance
Appears as distributed virtual switch to hypervisor
Currently supported with ESX 5.0 only
vMotion supported
23
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model vMotion with Hypervisor Bypass (VMDirectPath with VM-FEX)
Temporary transition from VMDirectPath to
standard I/O
• VM Sending TCP stream (1500MTU) • UCS B200 M2 blades with UCS VIC card
0
2500
5000
7500
10000
19:0
6:19
19:0
6:23
19:0
6:27
19:0
6:31
19:0
6:35
19:0
6:39
19:0
6:43
19:0
6:47
19:0
6:52
Mb
ps
Time (secs)
vMotion to secondary host
1 sec silent period
24
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model Simplifying the Access Infrastructure
Physical Network
Virtual Network
Hyp
ervi
sor
Hyp
ervi
sor
VM VM VM VM VM VM VM VM
VETH
VNIC
Unify the virtual and physical network ‒ Same Port Profiles for various
hypervisors and bare metal servers
Consistent functions, performance, management
25
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Programmable Access Layer ‒ Basic Starting Points
Software Definable Networking (SDN)
Standardize Troubleshooting ‒ SPAN usage, Performance Trending
Traffic Engineering
Traffic Management Physical Network
Virtual Network
VM VM VM VM VM VM VM VM
VLANs
26
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Removing performance dependencies from VM location
Offloading software switching functionalities from host CPU
More on this in upcoming slides
Traffic Forwarding
Physical Network
Hyp
ervi
sor
Hyp
ervi
sor
VM VM VM VM VM VM VM VM
VETH
VNIC
27
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Operational Model
Simpler Deployments ‒ Unifying the virtual and physical network
‒ Consistency in functionality, performance and management
Robustness ‒ Programmability of the infrastructure
‒ Troubleshooting, traffic engineering virtual and physical together
Performance ‒ Near bare metal I/O performance
‒ Improve jitter, latency, throughput and CPU utilization
Security ‒ Near bare metal I/O performance
‒ Improve jitter, latency, throughput and CPU utilization
Contrasting VM-FEX to Virtualised Switching Layers
28
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS System Components Foundation
UCS 6100
UCS 6200
UCS 2208 IOM
UCS 2104 IOM
UCS Manager Embedded – Manages entire system
UCS Fabric Interconnect – UCS 6100 • 20x 10GE Ports – 1 RU • 40x 10GE Ports – 2 RU • Ethernet or FC Expansion Modules
UCS Fabric Interconnect – UCS 6200 • 48x Unified Ports (Eth/FC) – 1 RU • 32x base and 16x expansion
UCS Fabric Extender – UCS 2104 • 8x 10GE Downlinks to Servers • 4x 10GE Uplinks to FIs
UCS Fabric Extender – UCS 2208 • 32x 10GE Downlinks to Servers • 8x 10GE Uplinks to FIs
UCS Blade Server Chassis Flexible Bay Configuration Houses blades based on Industry-standard architecture
Adapters - M81KR VIC, M71KR, etc.
• Up to 2x 10GE ports
• M81KR: Up to 128 virtual interfaces
Adapter - UCS VIC 1280
• Up to 8x 10GE ports
• Up to 256 virtual interfaces
30
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS System Architecture Foundation
2x 4 Link 80 Gbps per Chassis
2x 8 Links 160 Gbps per Chassis
2x 2 Link 40 Gbps per Chassis
2x 1 Link 20 Gbps per Chassis
Wire once for bandwidth, not connectivity Policy-driven bandwidth allocation Integrates as a single system into your data center
31
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Nexus 5500 and 2200 Components for VM-FEX Foundation
Nexus 5548
Nexus 5596
Nexus 2232 FEX
Nexus 5548 • 48x Unified Ports (Eth/FC) max – 1 RU • 32x base and Universal GEM expansion for 16x • 1 expansion slot
Nexus 5596 • 96x Unified Ports (Eth/FC) max – 2 RU • 32x base and universal GEM expansion for 64x • 4 expansion slots
Nexus 2232 • 32x 10GE Downlinks to Servers • 8x 10GE Uplinks to Fis
Adapters – P81E
• Up to 2x 10GE ports
• Up to 128 virtual interfaces
32
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS VM-FEX System View Deploying on a UCS B or C Series Infrastructure
3 4
1 3 4 5 6 7 8 Chassis IO Module A
1 2
Server Ports
3 4
1 3 4 5 6 7 8
1 2
Server Ports
VN Tag @ 10Gbe
2 2 Chassis IO Module B
Internal Connections
vfc0
2 3 4 5 6
0 1
vNIC1(s) vNIC2(s)
vfc1
2 3 4 5 6
ESX 4.0u1+ / RHEL KVM 6.1+ / MS Windows 8 Server
VM
-FEX
UCS 6x00 Physical Ports
Chassis IOM Ports
UCS 6x00 Physical Ports
Chassis IOM Ports
VIC CPU
Virtual Interface Control Logic
Virtual Interface Control Logic
vCenter Controlled interfaces on VMs
with forwarding rules enforced on dynamic
adapters and signaled on Private
Interfaces
HBA 0 vHBA0
HBA 1 vHBA1
veth10
1 1
8 7 2 1 Fiber Channel Uplink Ports
8 7 2 1 Fiber Channel Uplink Ports
6 5 6 5
veth1 veth2 veth3 veth4 veth1 veth2 veth3 veth4 veth10 0
Mgmt Uplink
0
Mgmt Uplink
CIMC KVM etc.
Cisco Adapter
UCS B or C Series Server
UCS Fabric Interconnect B (port profiles) UCS Fabric Interconnect A (port profiles)
ESX Kernel Module / Libvirt / HyperV Extendable Switch
33
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS VM-FEX System View Deploying on a UCS B or C Series Infrastructure
3 4
1 3 4 5 6 7 8 Chassis IO Module A
1 2
Server Ports
3 4
1 3 4 5 6 7 8
1 2
Server Ports
VN Tag @ 10Gbe
2 2 Chassis IO Module B
Service Console Kernel
Internal Connections
vfc0
2 3 4 5 6
0 1
d-vNIC1 vNIC1(s) d-vNIC2 vNIC2(s) d-vNIC3
vfc1
2 3 4 5 6
VM
-FEX
UCS 6x00 Physical Ports
Chassis IOM Ports
UCS 6x00 Physical Ports
Chassis IOM Ports
VIC CPU
Virtual Interface Control Logic
Virtual Interface Control Logic
vCenter Controlled interfaces on VMs
with forwarding rules enforced on dynamic
adapters and signaled on Private
Interfaces
ESX Kernel Module / Libvirt / HyperV Extendable Switch
Attempts by Guest OS to improperly mark
traffic blocked
HBA 0 vHBA0
HBA 1 vHBA1 d-vNIC4
veth10
1 1
8 7 2 1 Fiber Channel Uplink Ports
8 7 2 1 Fiber Channel Uplink Ports
6 5 6 5
veth1 veth2 veth3 veth4 veth1 veth2 veth3 veth4 veth10 0
Mgmt Uplink
0
Mgmt Uplink
CIMC KVM etc.
UCS Fabric Interconnect B (port profiles) UCS Fabric Interconnect A (port profiles)
Cisco Adapter
UCS B or C Series Server
ESX 4.0u1+ / RHEL KVM 6.1+ / MS Windows 8 Server 34
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS VM-FEX System View Deploying on a UCS C Series with Nexus 5500 Infrastructure
8
1 3 4 5 6 32 2232 FEX A
1 2
2232 Server Ports
8
1 3 4 5 6 32
1 2
2232 Server Ports
VN Tag @ 10Gbe
2 2 2232 FEX B
Internal Connections
vfc0
2 3 4 5 6
0 1
vNIC1(s) vNIC2(s)
vfc1
2 3 4 5 6
ESX 4.0u1+
VM
-FEX
Nexus 55xx Physical Ports
2232 Fabric Ports
Nexus 55xx Physical Ports
2232 Fabric Ports
VIC CPU
Virtual Interface Control Logic
Virtual Interface Control Logic
vCenter Controlled interfaces on VMs
with forwarding rules enforced on dynamic
adapters and signaled on Private
Interfaces
ESX Kernel Pass Through Module HBA 0 vHBA0
HBA 1 vHBA1
veth10
1 1
8 7 2 1 Fiber Channel Uplink Ports
8 7 2 1 Fiber Channel Uplink Ports
veth1 veth2 veth3 veth4 veth1 veth2 veth3 veth4 veth10 0
Mgmt Uplink
0
Mgmt Uplink
CIMC KVM etc.
Cisco P81E Adapter
UCS C Series Server
47 47
48 48
vPC Connections (veth’s not a vPC at FCS)
Nexus 55xx A (port profiles) Nexus 55xx B (port profiles)
35
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS VM-FEX System View Deploying on a UCS C Series with Nexus 5500 Infrastructure
8
1 3 4 5 6 32 2232 FEX A
1 2
2232 Server Ports
8
1 3 4 5 6 32
1 2
2232 Server Ports
VN Tag @ 10Gbe
2 2 2232 FEX B
Internal Connections
vfc0
2 3 4 5 6
0 1
vNIC1(s) vNIC2(s)
vfc1
2 3 4 5 6
VM
-FEX
Nexus 55xx Physical Ports
2232 Fabric Ports
Nexus 55xx Physical Ports
2232 Fabric Ports
VIC CPU
Virtual Interface Control Logic
Virtual Interface Control Logic
vCenter Controlled interfaces on VMs
with forwarding rules enforced on dynamic
adapters and signaled on Private
Interfaces
HBA 0 vHBA0
HBA 1 vHBA1
veth10
1 1
8 7 2 1 Fiber Channel Uplink Ports
8 7 2 1 Fiber Channel Uplink Ports
veth1 veth2 veth3 veth4 veth1 veth2 veth3 veth4 veth10 0
Mgmt Uplink
0
Mgmt Uplink
CIMC KVM etc.
Cisco P81E Adapter
UCS C Series Server
47 47
48 48
Service Console Kernel
d-vNIC1 d-vNIC2 d-vNIC3 d-vNIC4
6 5 6 5
Nexus 55xx A (port profiles) Nexus 55xx B (port profiles)
Attempts by Guest OS to improperly
mark traffic blocked
ESX 4.0u1+
ESX Kernel Pass Through Module
vPC Connections (veth’s not a vPC at FCS)
36
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #1: Creating Dynamic vNICs
Policies are to automatically provision dynamics on Servers
Dependent on the number of Fabric Interconnect to IO Module connections ‒ (# IOM to FI links * 63) - 2
Setting a Dynamic Adapter Policy Up
37
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #2: Building Service Profile
2 Statics – 1 to each UCS Fabric
Change dynamic vNIC connection policy to setup dynamics
Adding the Dynamic Policy and Static Adapters
38
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #3: Building Port Profiles
Creating Port Profiles Includes: ‒ VLAN(s)
‒ Native and/or Tagging allowed
‒ QoS Weights and Flow Rates
‒ Upstream Ports to always use
Creating Folders of Network Access Attributes
39
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #4: Building Port Profiles
Selecting High Performance will only Impact VMware deployment today
No problem if selected and used on other hypervisors
Enhanced Options like VMDirectPath with VM-FEX
40
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #5: Communication with Manager
Same Plug-in Method used in Nexus 1000v
Tool discussed later to simplify the whole integration process
8 Separate managers today
Establishing Communication to Hypervisor Manager
41
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS General Baseline #6: Publishing Port Profiles
Publish Port Profiles to Hypervisors and virtual switches within
4 Separate virtual switch per manager today
Exporting Port Profiles to these to Hypervisor Manager
42
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #1: Dynamic vNICs
Enable NIV (Network Interface Virtualization) on the P81E Adapter in the CIMC ‒ CIMC is the management Interface for the Cisco C2xx servers
‒ Choose the number of dynamics to configure (next slide)
Setting a Group of Dynamics on C2xx Servers
43
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #1: Dynamic vNICs
Enable vNICs then view the VM-FEXs Tab in the CIMC ‒ CIMC is the management Interface for the
Cisco C2xx servers
‒ UCS Standalone CIMC version 1.4 or greater required
‒ Minimum of 2 static vNICs defined
‒ Numbers of VM FEX’s (dynamic vNICs) are dependent on links from 5500 to 2232 if using FEX (Limit at 96 today)
‒ Nexus 5500 version 5.1(3)N1(1) or later required
Setting a Group of Statics and Dynamics on C2xx Servers
44
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #2: N5k Port Profiles nexus5500-1(config)# install feature-set virtualization
nexus5500-1(config)# feature-set virtualization
nexus5500-1(config)# feature vmfex
nexus5500-1(config)# feature npiv
nexus5500-1(config)# vethernet auto-create
nexus5500-1(config)# interface Ethernet100/1/27 << P81E Port
nexus5500-1(config-if)# switchport mode vntag
nexus5500-1(config)# port-profile type vethernet VM_VLAN_6
nexus5500-1(config-port-prof)# switchport access vlan 6
nexus5500-1(config-port-prof)# high-performance host-netio
nexus5500-1(config-port-prof)# port-binding dynamic
nexus5500-1(config-port-prof)# dvs-name all << To publish to the VM-FEX DVS
nexus5500-1(config-port-prof)# state enabled
Enabling VMFEX, and Creating Folders of Network Access Attributes
45
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #3: P81E Statics
Configuring a static profile for the fixed interfaces
One to each N5k in a pair
Select the vNIC, and the port profile to assign to it
These will be initially in the vSwitch in the out of box VMware configuration
Configuring Adapter FEX interfaces for Startup/Shutdown Operations
46
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #4: vCenter plugin
nexus5500-1(config)# feature http-server
Downloading and registration of plug-in per the other VM-FEX topologies
Enabling VMFEX, and Creating Folders of Network Access Attributes
47
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
UCS C2xx / N5500 General Baseline #5: Comm with Manager Establishing Communication to Hypervisor Manager
nexus5500-1(config)# svs connection vCenter1
nexus5500-1(config-svs-conn)# protocol vmware-vim
nexus5500-1(config-svs-conn)# extension-key UCSTME-Nexus5000-VMFEX-DVS
nexus5500-1(config-svs-conn)# remote ip address 172.25.177.227 port 80 vrf management
nexus5500-1(config-svs-conn)# dvs-name UCS_C2XX_VMFEX_DVS_1
nexus5500-1(config-svs-conn)# vmware dvs datacenter-name UCSTMELAB
nexus5500-1(config-svs-conn)# connect
Do this on Both Nexus 5500’s in the vPC Pair
nexus5500-1(config-svs-conn)# show svs connections
48
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: Infrastructure Requirements
8 vCenters can be joined from UCS
Each vCenter can have 4 DVS’ within
Plug-In download and install method (unless Easy VM-FEX tool is used)
Hosts then use VUM Depot’s to install ESX module when bringing host into UCS DVS (unless Easy VM-FEX tool is used)
Enterprise+ required (as is for any DVS) on Host
Standard and above is required for vCenter
VMotion fully supported
VMDirectPath with VM-FEX is possible (Hypervisor Bypass)
‒ Cisco UCS with VM-FEX linked to in vSphere 5 networking guide – page 42
‒ vCenter VM->Properties->Resources – need to reserve all guest memory
Versions, Licenses, etc.
50
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: vCenter View
Uplinks from ESX hosts shown on right ‒ These are the statics for overhead
VM vNICs shown in port groups on left ‒ Port Groups are from Port Profiles
sent in from UCSM or Nexus 5500
Normal view of VM vNICs, MAC, Port numbers, etc.
View from the Administrator
51
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: vCenter View
Match the Memory Reservation with the Limit
Fundamental requirement of all DirectPath deployments
Only Supported Guests will get DP with VM-FEX ‒ Windows Server 2008 SP2, Windows Server 2008 R2, RHEL 6.x, SLES11 SP1
View from the VM Settings to Get DirectPath with VM-FEX
52
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: vCenter View
Simply add vNICs
Define normal VMXNET3 type
Select Port Group to put adapter into
Displays if the DirectPath with VM-FEX is active
Other Adapters can remain in emulated mode if desired
View from the VM Settings
53
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: vCenter View View from the vCenter
54
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: Administrator View View from the UCSM
55
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: Administrator View
User can see where the VM’s map to the Veth ports on the 5500’s
Normal NX-OS command set to view/manage/SPAN etc as in physical ports
View from the Nexus 5500
56
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
RHEL KVM VM-FEX: Infrastructure Requirements
VM-FEX is available for KVM on only UCS Managed Deployments Today
Install Red Hat as Virtualization Host
Unlike VMware no VEM to load (utilizes libvirt)
Works with live migration feature for VMs
Scripted nature of configuration at FCS ‒ No current RHEV-M for RHEL KVM 6.x
‒ RHEV-M 3.0 will have RHEL 6.2 hooks for VM-FEX configuration assistance
MacVTap has 3 distinct modes ‒ Bridge mode for normal end points connecting together within host (Adapter FEX has value here)
‒ 802.1Qbg “VEPA” for traffic hair-pinning on next upstream device
‒ 802.1Qbh “private-mode” for traffic always passing to controlling bridge (UCS FI)
VM-FEX uses private-mode
Virtual Machine interface management via editing of VM domain XML file
Versions, Licenses, etc.
58
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
RHEL KVM VM-FEX: Virt-manager View
RHEL virt-manager to do simple VM operations
Can start, stop, open, migrate VM’s with VM-FEX connections also
RHEV-M will be able to present VM-FEX port profiles natively with 3.x
View from the Administrator
59
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
RHEL KVM VM-FEX: CLI View
Virsh set of commands to control VM’s
Create, Start, Stop, Migrate, etc. of VM’s with VM-FEX included
View from the Administrator
60
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
RHEL KVM VM-FEX: VM vNIC View
Adapters can be created in virt-manager wizard and MAC assigned
Edit the domain.xml file to make the adapters VM-FEX
Bring in the port profile here
Then VM will operate as normal
No reliance on bridge domains
View from the Administrator
61
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: Administrator View
Same port profiles can be used to KVM VM’s
Live Migration fully supported
View from the UCSM
62
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
HyperV VM-FEX: Infrastructure Requirements
VM-FEX is available for HyperV on only UCS Managed Deployments at shipment of Windows 8
HyperV Role Enabled on Windows 8 Servers
For Live Migration, MS Cluster built with shared storage
‒ VM-FEX with Live Migration fully supported
HyperV Networks defined as shown here
‒ Through HyperV Manager GUI
‒ Through PowerShell Applets
Cisco Extension to the HyperV Extensible Switch infrastructure
Systems Center Virtual Machine Manager (SCVMM) 2012 version with Windows 8 Server support will be needed for manager integration
‒ Fully supported via PowerShell until the SCVMM ships
Versions, Licenses, etc.
64
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
HyperV VM-FEX: VM vNIC View
Install the VIC device
Install the Cisco VmFex Forwarding Extension
‒ .\vmfextool.exe -i “E:\VmFex\ReleaseDrivers\cscovmfext\cscovmfext.inf" "Csco_FEXT"
Create a IOV Switch
‒ New-VMSwitch -Name "Cisco-Network" -NetAdapter $externalnic -SwitchType External -AllowManagementOS 1 -Notes "Cisco Private Network Traffic Switch" -EnableIov 1
‒ Enable-VMSwitchExtension -VMSwitchExtensionName "Cisco VmFex Forwarding Extension
Insert the Cluster UUID and Name to match the UCSM
‒ .\Cisco_SwitchProp-Add.ps1 "Cisco-Network" "38c79463-e5d3-46f6-ba47-b35719f15c70" "Msft-clus"
Add an SR-IOV adapter to the VM with the command
‒ .\PortProfileSetting-Add.ps1 "VMName" "PortProfileId" "PortProfileName" "NicID"
Key PowerShell Components
65
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
HyperV VM-FEX: Failover Cluster Manager View View from the Administrator
66
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
HyperV VM-FEX: VM vNIC View
Virtual Network Manager to configure the switch
PS scripts to configure the Cisco extension to the HyperV switch
View from the Administrator
67
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
HyperV VM-FEX: VM vNIC View
VM is created first
PS scripts add the network policy from our port-profiles behind Adapters
View from the Administrator
68
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VMware VM-FEX: Administrator View View from the UCSM
69
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Easy VM-FEX Tool
VMware solution only today with UCS
Quick System Bringups
Assumption of 1 management interface per ESX host
‒ Optional vMotion / FT logging also handled
All supported versions of VMware that VM-FEX supports
‒ Enterprise+ or Evaluation
Can define some defaults in text file
vCenter folders OK
Server needs Dynamic vNICs on Service Profile (will check)
Deployment name limited to 8 characters in tool
UCSM respository for ESX kernel model, or separate tool to pull from VMware online to a dedicated directory locally
Tool Usage
71
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Easy VM-FEX Tool Tool View
72
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Performance Review
Virtual Switch, CPU at ~65%
VM-FEX, CPU at ~ 37%
Test of 10 VM’s running HTTP and FTP Gets with IxLoad
73
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
VM-FEX Performance Review
3 IxVM’s, sending fixed 3.33G flows to a 4th VM All on same ESX host to maximize the differences in operations
‒ Virtual switch with VMs on same host – flows contained within ESXi 5 host ‒ VM-FEX with VMs on same host – flows traverse the chassis IOM/FI, or N2k/5k ‒ Higher Bandwidth AND Lower Latency between VM’s
Test of 4 IxVM’s – Virtual Switching CPU @ 88%, VM-FEX CPU @ 52%
74
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Summary
FEX technologies can reduce managed device count
FEX technologies will greatly reduce cabling overhead
VM-FEX is terminating these virtual links directly on the VMs
Closely maps to the physical server model for operations and management
Multiple Hypervisors are supported with advanced features
Bandwidth can be engineered identically to physical infrastructures today
Latency can surpass local virtualized switching by moving away from virtual switching store and forward buffering, “tree’s” of ASIC traversals, to a uniform port controller and switch fabric model
76
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
BRKCOM-2005 Recommended Viewing
www.YouTube.com/ciscodatacenter Playlist UCS Technical Videos http://www.youtube.com/ciscodatacenter#p/c/F04A2C6AA04DF055
Overview Cisco UCS Advantage http://www.youtube.com/watch?v=IW4zHXIjpPU
UCS Advantage Videos on YouTube
77
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Category Title URL UCS server Service Profiles and Templates http://www.youtube.com/watch?v=JW-YtVN75R0
UCS server Organizations and Roles http://www.youtube.com/watch?v=tb-L0zv3If
UCS server Extended Memory Technology http://www.youtube.com/watch?v=kS3ehPRcVDo
UCS server Server Pre-Provisioning http://www.youtube.com/watch?v=o7BuEE3hNPE
UCS server BIOS Policies http://www.youtube.com/watch?v=Pr6EptC9JXQ
UCS server RAID Policies http://www.youtube.com/watch?v=Vcs56wjUWuI
UCS server Firmware Policies http://www.youtube.com/watch?v=vjj8Xz0NqI4
UCS server Server Pools and Qualification Policies http://www.youtube.com/watch?v=KTw7M3T-VOw
UCS server Maintenance Policies http://www.youtube.com/watch?v=QQTlm98NgTI
UCS server High Availability During Upgrades http://www.youtube.com/watch?v=57HXMGn88HA
UCS server Monitoring with BMC BPPM http://www.youtube.com/watch?v=mdoEZf7tM5E
UCS server Microsoft Hyper-V on UCS http://www.youtube.com/watch?v=G3x_YOYK-Fo
BRKCOM-2005 Recommended Viewing
78
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
BRKCOM-2005 Recommended Viewing
Category Title URL
UCS I/O Adapter Templates http://www.youtube.com/watch?v=KpVEn3DhfOM
UCS I/O Network Interface Virtualization http://www.youtube.com/watch?v=njjbCEblxVc
UCS I/O Adapter Fabric Failover http://www.youtube.com/watch?v=tlu8RSq6T_M
UCS I/O Extend the Network to the Virtual Machine http://www.youtube.com/watch?v=Ylizxq18yxE
UCS I/O Traffic Analysis of All Servers http://www.youtube.com/watch?v=PHTdXy_8Zdg
UCS I/O Ethernet Switching Modes http://www.youtube.com/watch?v=roX8MRN66UM
UCS I/O Fibre Channel and Switch Modes http://www.youtube.com/watch?v=VSetsgOYYCo
UCS I/O FC Port Channels and Trunking http://www.youtube.com/watch?v=PpzKPguRTXc
79
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Category Title URL
UCS Infrastructure
Lights-Out Management http://www.youtube.com/watch?v=QEO1d_1vTxs
UCS Infrastructure
Easy VM-FEX Deployment http://www.youtube.com/watch?v=0aAuj80cNvg
UCS Infrastructure
Server Power Grouping http://www.youtube.com/watch?v=EgoFe33YoD8
UCS Infrastructure
Blade and Rack-Mount Management http://www.youtube.com/watch?v=aOsx4YMiOho
UCS Infrastructure
Manager Platform Emulator http://www.youtube.com/watch?v=ZNNrs2e0wvk
UCS Infrastructure
Cisco Developer Network and Sandbox http://www.youtube.com/watch?v=Syhl6SAiwew
BRKCOM-2005 Recommended Viewing
80
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Complete Your Online Session Evaluation Give us your feedback and you
could win fabulous prizes. Winners announced daily.
Receive 20 Passport points for each session evaluation you complete.
Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live Virtual account for access to all session material, communities, and on-demand and live activities throughout the year. Activate your account at the Cisco booth in the World of Solutions or visit www.ciscolive.com.
81
© 2012 Cisco and/or its affiliates. All rights reserved. BRKCOM-2005 Cisco Public
Final Thoughts
Get hands-on experience with the Walk-in Labs located in World of Solutions, booth 1042
Come see demos of many key solutions and products in the main Cisco booth 2924
Visit www.ciscoLive365.com after the event for updated PDFs, on-demand session videos, networking, and more!
Follow Cisco Live! using social media: ‒ Facebook: https://www.facebook.com/ciscoliveus
‒ Twitter: https://twitter.com/#!/CiscoLive
‒ LinkedIn Group: http://linkd.in/CiscoLI
82