© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 1
Carlos Campos
UCS Consulting Systems Engineer – Latam
CCIE#16993 Storage/R&S
Virtual Machines Fabric Extender, UCS Fabric and VM's - Extending FEX direct to VM's in UCS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 2
Virtualization/Fabric Extender Overview
Virtual Machine Fabric Extender (VM-FEX)
Benefits of VM-FEX
Forwarding and Latency considerations with FEX Technologies (NEW)
VM-FEX UCS Generals
VM-FEX VMware on UCS
VM-FEX KVM on UCS
Security and Segmentation of with VM-FEX
Operations Model with VM-FEX (Stats and SPAN)
Summary
Agenda
What we hope to share
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 3
Vmware/Hyper-V and virtualization
Virtualized environments are the natural transiton to 10G
Reduce cabling
Reduce total number of ports
Reduce virtual machine oversubscription
Resources underutilized
Several NICs used nowadays (VMKernel, console, data, backup) and not all of them are as used as data ones
Inconsistent responsabilities/configurations
Network configurations now also being part of servers area (vSwitch) which leads to inconsistent configurations
Uplink validation (security/QoS) might be needed before Vmotion
Monitoring is not possible for saturation, DoS attacks, etc within VMWare
No IPS, IDS information exporting
Virtual-machine flexibility
Diversity in virtual-machines is not possible due to Vmnic sharing and no QoS policing enforced
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 4
Hyperv
isor
Hyp
erv
iso
r
Hyperv
iso
r
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
Phase I Virtualization Consolidated Environment
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 5
Hyperv
isor
Hyp
erv
iso
r
Hyperv
iso
r
App
OS
App
OS
App
OS
vSwitch vSwitch vSwitch
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
Phase I Virtualization More virtual infrastructure needed
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 6
Hyperv
isor
Hyp
erv
iso
r
Hyperv
iso
r
vSwitch vSwitch
App
OS
vSwitch
App
OS
Net State Net State Net State
App
OS
Phase 2 Virtualization: Mobility Network policies tied to VSwitch: Not mobile
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 7
Hyperv
iso
r
Hyp
erv
iso
r
Hyperv
iso
r
App
OS
App
OS
vSwitch vSwitch
App
OS
vSwitch
Net State Net State Net State
Cisco VN-link: Virtualization Aware Network Network Policies tied to Virtual Machine
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 8
Hyperv
isor
Hyp
erv
iso
r
Hyperv
isor
vSwitch vSwitch
App
OS
vSwitch
App
OS
Net State Net State Net State
App
OS
vNetwork Distributed Switch VN-Link Domain
Cisco VN-link: Virtualization Aware Network Network policies trascend server boundaries
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 9
Before/after VMWare considerations
Before
After
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 10
Before/after VMWare considerations
Before
After
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 11
Before/after VMWare considerations
Before
After
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 12
Cisco Nexus 1000V Industry First 3rd Party Virtual Distributed Switch
Nexus 1000V provides enhanced VM switching for VMW ESX environments
Features VN-Link capabilities:
Policy-based VM connectivity
Mobility of network and security properties
Non-disruptive operational model
Ensures visibility and continued connectivity during VMotion
Enabling Acceleration of Server Virtualization Benefits
VMW ESX
Server 1
VMware vSwitch Nexus 1000V
VMW ESX
VMware vSwitch Nexus 1000V
Server 2
Nexus 1000V
VM
#4
VM
#3
VM
#2
VM
#1
VM
#8
VM
#7
VM
#5
VM
#5
VM
#2
VM
#3
VM
#4
VM
#5
VM
#6
VM
#7
VM
#8
VM
#1
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 13
Cisco Nexus 1000V Components
VMW ESX
Server 3
VM
#9
VM
#12
VM
#11
VM
#10
VEM
VMW ESX
Server 2
VM
#5
VM
#8
VM
#7
VM
#6
VEM
VMW ESX
Server 1
VM
#1
VM
#4
VM
#3
VM
#2
VEM
Virtual Ethernet Module(VEM)
Replaces existing vSwitch
Enables advanced switching capability on the hypervisor
Provides each VM with dedicated “switch ports”
Virtual Supervisor Module(VSM)
CLI interface into the Nexus 1000V
Leverages NX-OS 4.01
Controls multiple VEMs as a single network device
Virtual Center
Nexus 1000V
VSM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 14
Nexus 1000V ‘Virtual Chassis’ Model
One Virtual Supervisor Module managing multiple Virtual Ethernet Modules
•Dual Supervisors to support HA environments
A single Nexus 1000V can span multiple ESX Clusters
SVS-CP# show module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ----------
--
1 1 Supervisor Module Cisco Nexus 1000V active *
2 1 Supervisor Module Cisco Nexus 1000V standby
3 48 Virtual Ethernet Module ok
4 48 Virtual Ethernet Module ok
--More--
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 15
Single Chassis Management
Upstream-4948-1#show cdp neighbor
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone
Device ID Local Intrfce Holdtme Capability
Platform Port ID
N1KV-Rack10 Gig 1/5 136 S Nexus
1000V Eth2/2
N1KV-Rack10 Gig 1/10 136 S Nexus
1000V Eth3/5
N1KV-Rack10 Gig 1/12 136 S Nexus
1000V Eth21/2
A single switch from control plane and management plane perspective
Protocols such as CDP operates as a single switch
XML API and SNMP management appears as a single ‘virtual chassis’
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 16
Virtual Supervisor Options
VSM
VSM
VSM Virtual Appliance
ESX Virtual Appliance
Special dependence on CPVA server
Supports up to 64 VEMs
VMW ESX
Server 3
VM
#9
VM
#12
VM
#11
VM
#10
VEM
VMW ESX
Server 2
VM
#5
VM
#8
VM
#7
VM
#6
VEM
VMW ESX
Server 1
VM
#1
VM
#4
VM
#3
VM
#2
VEM
VSM Physical Appliance
Cisco branded x86 server
Runs multiple instances of the VSM virtual appliance
Each VSM managed independently
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 17
Virtual Ethernet Module
VEM is a light weight (~10MB RAM) module that provides switching capability on the ESX host
Single VEM instance per ESX host
Relies on the VSM to provide configuration
Stores basic configs locally (system VLANs, Domain ID, etc…)
Can run in last known good state without VSM connectivity
Some will not work (Vmotion) in this state
Must have VSM connectivity upon reboot to switch VM traffic
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 18
Switching Interface Types
Physical Ethernet Ports
- NIC cards on each server
- Appears as ‘Eth’ interface on a specific module in NX-OS
Example – ‘Eth10/7’
- Static assignment as long as the module ID does not change
- Up to 32 per host
Virtual Ethernet Ports
- Virtual Machine facing ports
- Appears as ‘Veth’ within NX-OS.
- Not assigned to a specific module to simplify VMotion
Example – ‘Veth68’
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 19
Cisco Nexus 1000V Scalability @ FCS
A single Nexus 1000V
• 66 modules (2x Supervisors and 64x Ethernet Modules)
• 4x64 modules in Nexus 1010
Virtual Ethernet Module:
• 32 physical NICs
• 256 virtual NICs
Limit Per Nexus 1000V
• 512 Port Profiles
• 2048 physical ports
• 8,192 virtual ports (vmknic, vswif, vnic)
Virtual Supervisor - Standby
VEM
VEM
VEM
VEM
VEM
VEM
VEM
VEM
VEM
VEM
Virtual Supervisor - Active
Nexus 1000V
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 20
Virtual Supervisor to Virtual Center
One way API between the VSM and Virtual Center
Certificate (Cisco self signed or customer supplied) ensures secure communications
Connection is setup on the Supervisor
N1K-CP# show svs connections
Connection VC:
IP address: 10.95.112.10
Protocol: vmware-vim https
vmware dvs datacenter-name: PHXLab
ConfigStatus: Enabled
OperStatus: Connected
Nexus 1000V
VSM
Virtual Center
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 21
Introducing the Cisco Nexus 1010
• Allows network administrators to manage the Nexus 1000V Virtual Supervisor Module (VSM) as a standard Cisco® switch, with all Nexus 1000V features and with Virtual Services Nodes (VSN)
• Physical appliance for virtual network services (VSM, NAM, etc.).
• Supported by CiscoWorks LAN Management Solution (LMS).
• Cisco Nexus® 1010 is a networking appliance that can host four Nexus 1000V Virtual Supervisor Modules.
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 22
Supervisor to Ethernet Module
Two distinct virtual interfaces are used to communicate between the VSM and VEM
•Control
• Carries low level messages to ensure proper configuration of the VEM.
• Maintains a 2 sec heartbeat what the VSM to the VEM (timeout 6 seconds)
•Packet
•Carries any network packets between the VEM and the VSM such as CDP/LLDP
Must be on two separate VLANs
Supports both L2 and L3 designs
VMW ESX
VM
#1
VM
#4
VM
#3
VM
#2
VEM
Nexus 1000V
VSM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 23
Introduction to Port Profiles
Port Profiles are a collection ‘interface’ commands
i.e.:
switchport mode access
switchport access vlan 57
no shutdown
Applied at the interface level using to either physical or virtual interfaces
Dynamic configuration
Port Profile changes are propagated immediately to all ports using that profile
Interfaces can be configured manually in conjunction with a profile
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 24
VMW ESX
Server
Nexus 1000V - VEM
VM
#1
VM
#4
VM
#3
VM
#2
What Can A Profile Contain?
Policy definition supports:
VLAN, PVLAN settings
ACL, Port Security, ACL
Redirect
Cisco TrustSec (SGT)
NetFlow Collection
Rate Limiting
QoS Marking (COS/DSCP)
Remote Port Mirror (ERSPAN)
Nexus 1000V
VSM
Virtual Center
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 25
Port Profiles Propagation
Port profiles are pushed via the Virtual Center API
Upon connection/reconnection with Virtual Center the VSM re-verifies the correct port profile configuration exists within Virtual Center
Port profile ‘state’ and ‘type’ must be set for propagation to occur
N1K-CP(config-port-prof) state enable
N1K-CP(config-port-prof) vmware port-group (optional name)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 26
Network Administrator View
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 27
VMware Administrator View
Consistent Workflow: Continue to select Port Groups when configuring a VM in VMware Virtual Infrastructure Client
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 28
Port Profile Mobility – Simplified VMotion
VMW ESX
Server 2
Nexus 1000 -—VEM
VMW ESX
Server 1
Nexus 1000V—VEM Nexus 1000V
VM
#5
VM
#8
VM
#7
VM
#6
VM
#1
VM
#4
VM
#3
VM
#2
VM
#1
Nexus 1000V
VSM
VM
#1
Virtual Center
Fabric Extender Overview
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 30
LAN LAN Switch port extended
over Fabric Extender
Lo
gic
al
Sw
itch
Collapse networking tiers, reduce network management points!!!
Switch
Switch
Legacy multi-tier architecture FEX architecture
Switch
FEX
Fabric Extender (FEX) Concept
Switch
FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 31
Over 6000 Nexus 5K/2K customers Over 5400 UCS Customers
LAN
Parent Switch
Parent Switch + FEX
Single Access Layer
SAN
FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 32
Over 6000 Nexus 5K/2K customers Over 5400 UCS Customers
LAN
Parent Switch + FEX
Single Access Layer
SAN
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 33
FEX Building Block #1: VNTAG
1. VNTAG
D: Direction, P: Unicast/Multicast, L: Loop
Virtual addresses switch local
VNTAG Ether type
Destination Virtual Interface
Source Virtual Interface ver
D P
L R
Application
Payload
TCP
IP
Ethernet
VNTAG
FEX architecture
Switch
FEX
LAN
Frame
VNTAG
Frame
Layer 2 Tag To Mimic Forwarding Vector
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 34
FEX Building Block #2: Policy
FEX architecture
Switch
FEX
LAN
Frame
VNTAG
Frame
Policy Associated with Virtual Interfaces NOT Physical
1. VNTAG
D: Direction, P: Unicast/Multicast, L: Loop
Virtual addresses switch local
2. Policy
VLAN, QoS, Rate limit
VNTAG Ether type
Destination Virtual Interface
Source Virtual Interface ver
D P
L R
Application
Payload
TCP
IP
Ethernet
VNTAG
Virtual Machine Fabric Extender
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 36
LAN
Parent Switch
Parent Switch + FEX + Virtual Switch
Physical and Virtual Infrastructure
SAN
FEX
App
OS
App
OS
App
OS
vSwitch
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
Physical Network
Virtual Network
Separate virtual and physical infrastructures!!!
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 37
Lo
gic
al
Sw
itch
Virtualized Deployment
Switch
FEX
Hypervisor vSwitch
App
OS
App
OS
App
OS
LAN
Lo
gic
al
Sw
itch
VM-FEX architecture
Switch
FEX
Hypervisor
LAN
App
OS
App
OS
App
OS
VM-FEX
Cascaded Fabric Extenders
Lo
gic
al
Sw
itch
Extending FEX Architecture to VMs Cascaded Fabric Extenders
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 38
LAN
Parent Switch
Parent Switch + FEX + VM-FEX
Single “Physical-Virtual” Access Layer
SAN
FEX
App
OS
App
OS
App
OS
VM-FEX
App
OS
App
OS
App
OS
Collapse virtual and physical networking tiers!!!
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 39
LAN
Parent Switch + FEX + VM-FEX
Single “Physical-Virtual” Access Layer
SAN
App
OS
App
OS
App
OS
App
OS
Connect 1000s of VM to Distributed Modular Switch
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
App
OS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 40
FEX Building Block #3: PCIe Virtualization
FEX architecture
Switch
FEX
LAN
Frame
VNTAG
Frame
UCS Virtual Interface Card
1. VNTAG
D: Direction, P: Unicast/Multicast, L: Loop
Virtual addresses switch local
2. Policy
VLAN, QoS, Rate limit
3. PCIe Virtualization
Ability to create interfaces inside the host
Hypervisor
App
OS
App
OS
App
OS
VM-FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 41
Flexibility and Performance
UCS Virtual Interface Card 1280
Flexibility
256 PCIe devices
Devices can be vNICs or vHBAs
Each device has a corresponding switch interface
VM-FEX with VMDirectPath Capability
Performance
Dual 40Gb
vNICs/vHBAs NOT limited to 10Gb
PCIe Gen2 x 16
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 42
Physical Network
Virtual Network
Hyp
erv
iso
r
Hyp
erv
iso
r
VM VM VM VM VM VM VM VM
VETH
VNIC
VM-FEX Operational Step: Background
Server
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 43
VM-FEX Operational Step #1: Pre Boot Config
Hyp
erv
iso
r
Hyp
erv
iso
r
Step1: Pre boot config
Anonymous PCIe Enumerations
Host discovers PCIe devices
Server Server
VM-FEX
Switch
VETH
VNIC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 44
VM-FEX Operational Step #2: Define Policy
Hyp
erv
iso
r
Hyp
erv
iso
r
Step1: Pre boot config
Anonymous PCIe Enumerations
Host discovers PCIe devices
Step2: Policy Definition
Networking Policy definition
VLAN, QoS, Rate limit, etc
Server Server
VM-FEX
Switch
Port Profiles
Definition
WEB Apps
HR
DB
Compliance
Network
Manager
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 45
VM-FEX Operational Step #3: Export Policy
Hyp
erv
iso
r
Hyp
erv
iso
r
Step1: Pre boot config
Anonymous PCIe Enumerations
Host discovers PCIe devices
Step2: Policy Definition
Networking Policy definition
VLAN, QoS, Rate limit, etc
Step3: Port Profile Export
Policy exported to Hypervisor manager
Server Server
VM-FEX
Switch
Network
Manager Hypervisor
Manager
Export Port Profiles
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 46
VM-FEX Operational Step #4: VM Association
Hyp
erv
iso
r
Hyp
erv
iso
r
Step1: Pre boot config
Anonymous PCIe Enumerations
Host discovers PCIe devices
Step2: Policy Definition
Networking Policy definition
VLAN, QoS, Rate limit, etc
Step3: Port Profile Export
Policy exported to Hypervisor manager
Step4: VM Creation
Policy resolution
Server Server
VM-FEX
Switch
Network
Manager Hypervisor
Manager
Export Port Profiles
VM
VM VM VM
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 47
Emulated Mode VMDirectPath
Standard Mode
12%-15% CPU performance improvement
vMotion supported
High Performance Mode
Co-exists with Standard mode
Bypasses Hypervisor layer
30% improvement in I/O performance
vMotion supported with ESX 5.0
Modes of VM-FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 48
0
2
4
6
8
10
12
0 10 20 30 40 50 60 70 B
and
wid
th (
Gb
ps)
Time (sec)
Temporary transition
from VMDP to
standard I/O
vMotion to secondary
host
• 8GB VM, sending UDP stream using pckgen (1500MTU)
• UCS B200 blades with UCS VIC card
• vSphere 5 technology preview
VMDirectPath Operations
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 49
Physical Network Simplicity
One infrastructure
Consistent features, performance and management
Benefits of VM-FEX #1: Simplicity
Virtual Network
Hyp
erv
iso
r
Hyp
erv
iso
r
VM VM VM VM VM VM VM VM
VETH
VNIC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 50
Physical Network
Virtual Network
VM VM VM VM VM VM VM VM
VLANs
Simplicity
One infrastructure
Consistent features, performance and management
Robustness
Programmability
Trouble shooting
Traffic engineering
Benefits of VM-FEX #2: Robustness
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 51
Simplicity
One infrastructure
Consistent features, performance and management
Robustness
Programmability
Trouble shooting & Traffic engineering
Performance
Near bare metal I/O performance
Benefits of VM-FEX #3: Performance
VMDirectPath
Forwarding and Latency considerations with FEX Technologies
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 53
LAN
Parent Switch
Parent Switch + FEX
De-Coupling of the Modular Switch
SAN
FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 54
Virtualizing the Switchports with Unified I/O Mapping of Ethernet and FC
Wires over Ethernet
Service Level enforcement
Multiple data types (jumbo, lossless, FC)
Individual link-states
Fewer Cables
oMultiple Ethernet traffic co-exist on same cable
Fewer adapters needed
Overall less power
Interoperates with existing Models
oManagement remains constant for system admins and LAN/SAN admins
Possible to take these links further upstream for aggregation
Individual
Ethernets
DCB Ethernet
Individual
Storage (iSCSI, NFS, FC)
Access Layer Consolidation & Virtualization
Blade Management
Channels (KVM, USB,
CDROM, Adapters)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 55
FEX Data Forwarding Is Switching local to Blades Always Better?
High variance in server to server bandwidth metrics
High variance in server to server latency metrics
Need to build out multiple fabrics
This led to static server designs and we needed to fix the locations and engineer the infrastructure
Very stateful in nature
Latency can be minimized – but challenged in store-and-forward
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 56
FEX Data Forwarding Is Intra-Host Switching Always Better?
VM-VM data switching within host will consist of mapping memory between structures
Latency will be optimal – but only for VMs on same host and can vary greatly
Moving memory structures of packets between emulated ports
Store and Forward handling of full packets through virtual switch
With VM-FEX and multiple port ASICs on Fabric Interconnects we get forwarding closer to a physical or virtual server (not a centralized fabric)
With cut-through switching on our port ASICs latency is greatly reduced
VM-FEX UCS Generals
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 58 2009
IOM links to FI decide number of interfaces available for the host (# IOM links * 63) – 2 (v2.x Hardware)
(# IOM links * 15) – 2 (v1.x Hardware)
Set a “Dynamic Policy” for the server in the UCS
Policy include: Number of dynamic VIF’s
Adapter policy (interrupt coalescing timers, buffering, etc.)
vCon policy (which adapter in a multi-adapter blade to use)
Dynamic vNICs are Protected
VM-FEX Dynamic vNIC Policies with VIC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 59
Double-wide blades have multiple Mezzanine slots
UCS Administrator Sets the physical adapter that vNIC and vHBA’s will map to
vCon is an Adapter
vNIC/vHBA can be A, B, A-B, B-A on the given vCon (not between)
Typical option is just to allow automatic system mapping
Can define a policy to control
Service Profiles with Full width Blades
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 60 2009
• Makes use of vCon first if separation by adapter is required
• UCS Administrator Sets the Dynamic vNIC Connection Policy
• 1 static vNIC to fabric A and 1 static vNIC to fabric B, for redundancy
• Actual VLAN, QoS, Pinning, etc. configuration done in port profiles
VM-FEX Service Profile with VIC
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 61 2009
• UCS Administrator Creates Profiles
• Sets Trunk VLANs, Access VLAN, etc.
• Profiles available for use on multiple vCenter defined DVS
VM-FEX Port Profiles
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 62 2009
• UCS Administrator Sets Policy per profile
• Sets CoS, Rate Limiting (VM to fabric direction only), Network Control, Uplink Connectivity, etc.
UCS Port Profiles (QoS, Control, Pinning)
VM-FEX VMware on UCS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 64
Configuration for VMDirectPath2
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 65 2009
ESX VM-FEX for VMware: Easy VM-FEX Tool
Eliminate Virtual Networking in minutes
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 66 2009
• Connections for up to 8 vCenter DVS’ within a UCS
• Keying for each of the connections
• Define DVS within the vCenter provider
ESX VM-FEX for VMware: UCSM/vCenter Connection
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 67
ESX VM-FEX: vCenter View
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 68
ESX VM-FEX: View from VM Settings
VM-FEX KVM on UCS
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 70
VM-FEX with KVM Architecture
Uplink Port
Virtual ports
PF VF2 VFn
eth0 eth2 ethn
Macvtap 2
User
Kernel
MacVTap
Interfaces Netdev
Interface …
…
Libvirt
Management
Tools
Netlink Socket
Guest
Userspace
virtio-net
Guest OS
KVM
UCS Fabric
Interconnect
vhost-net
Guest
Userspace
Guest OS
virtio-net
veth2
VIC
adapter
veth1
VF1
eth1
Macvtap 1
vhost-net
Port Profile1:
QoS1, VLAN1 Port Profile2:
QoS2, VLAN2
QoS2, VLAN2 QoS1, VLAN1
Physical Port
IO Module
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 71
Install Red Hat as Virtualization Host
Unlike VMware no VEM to load (utilizes libvirt)
Works with live migration feature for VMs
Scripted nature of configuration at FCS No current RHEV-M for RHEL KVM 6.x
RHEV-M 3.0 will have RHEL 6.2 hooks for VM-FEX configuration assistance
MacVTap has 3 distinct modes Bridge mode for normal end points connecting together within host (Adapter FEX has value here)
802.1Qbg “VEPA” for traffic hair-pinning on next upstream device
802.1Qbh “private-mode” for traffic always passing to controlling bridge (UCS FI)
VM-FEX uses private-mode
Virtual Machine interface management via editing of VM domain XML file
Trunking to guest TBD
UCS VM-FEX for KVM
RHEL 6.1 Administration
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 72
Create profiles for system
Can Create Clusters of KVM hosts
At FCS, Single Default Cluster only as there is no mechanism of a Cluster ID
Publish to client Cluster
Client can now be under the Cluster of KVM devices
UCS VM-FEX Configuration for KVM
Port Profiles in the VM Tab
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 73
KVM Domain XML Configuration
Security and Segmentation of with VM-FEX
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 75
Nexus 1000V
VSM
Virtual
Center
Virtual Network
Manager
(VNM)
VMW ESX
Server 2
VM
#3
VM
#4
Virtual
Service
Node #3
Service Data Path (SDP)
Virtual
Service
Node #2
VMW ESX
Server 1
VM
#2
VM
#1
Service Data Path (SDP)
Virtual
Service
Node #1
VMW ESX
Server N
VM
#5
VM
#6
Service Data Path (SDP)
Nexus 1000V VEMs
VM
#8
VM
#7
Virtual Security Gateway
VC: Server policies
VSM: Networking policies
VNM: Security policies
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 76
Firewall VSN Policy Model
VM
#1
VM
#8
VM
#7 VM
#6
VM
#4
VM
#3
VM
#2 VM
#5
DB Zone
Tenant_A
Nexus 1000 DVS
Zone 0 APP Zone Web Zone
Firewall
VSN
Internet/Intranet
Internet Zone Branch Zone
10.10.*.* +
10.12.10.*
Zone support
Multi-tenancy support
Context based (supports VM attributes as well as network constructs)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 77
VM
#7
Zone 1 Zone 2
Firewall
VSN
VEM
Flow Lookup Service
Lookup Service Data Path (SDP) VEM
Flow Lookup Service
Lookup Service Data Path (SDP)
VM
#3
VM
#11
Zone 3
VM
#1
VM
#2 VM
#6
Zone 3
VM
#12
VM Mobility
VM traffic Flow continues to be handled by appropriate Firewall VSN
Traffic State is re-built from Firewall VSN
VSN Mobility
Overlay tunnel ensures firewall VSN is reachable in the new host.
Traffic state moves with firewall VSN VM
Firewall VSN and VM Mobility Support
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 78
Security Zones and Unified Compute
3 4
1 3 4 5 6 7 8
Chassis IO Module A
1 2
Server Ports
3 4
1 3 4 5 6 7 8
1 2
Server Ports
VN Tag @ 10Gbe
2 2
Chassis IO Module B
Internal Connections 2 3 4 5 6
0 1
2 3 4 5 6 UCS 6x00 Physical Ports
Chassis IOM Ports
UCS 6x00 Physical Ports
Chassis IOM Ports
VIC CPU
Virtual Interface Control Logic
Virtual Interface Control Logic
vCenter
Controlled
interfaces on VMs
with forwarding
rules enforced on
dynamic adapters
and signaled on
Private Interfaces
1 1
8 7
Ethernet Uplink Ports
2 1
Fiber Channel Uplink Ports
8 7
Ethernet Uplink Ports
2 1
Fiber Channel Uplink Ports
6 5 6 5
0
Mgmt Uplink
0
Mgmt Uplink
CIMC KVM etc.
Cisco Adapter
UCS Fabric Interconnect B (port profiles) UCS Fabric Interconnect A (port profiles)
UCS Interaction with VMware and VM-FEX
Operations Model with VM-FEX (Stats and SPAN)
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 80
VM-FEX UCSM Visibility VM Visibility on VM Tab
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 81
VM-FEX UCSM Visibility Direct VM SPAN Sessions
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 82
www.YouTube.com/ciscodatacenter
Playlist UCS Technical
Videos
http://www.youtube.com/ciscodatacenter#p/c/
F04A2C6AA04DF055
Overview Cisco UCS
Advantage
http://www.youtube.com/watch?v=IW4zHXIjpPU
UCS Advantage Videos on YouTube
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 83
Category Title URL
UCS server Service Profiles and Templates http://www.youtube.com/watch?v=JW-YtVN75R0
UCS server Organizations and Roles http://www.youtube.com/watch?v=tb-L0zv3If
UCS server Extended Memory Technology http://www.youtube.com/watch?v=kS3ehPRcVDo
UCS server Server Pre-Provisioning http://www.youtube.com/watch?v=o7BuEE3hNPE
UCS server BIOS Policies http://www.youtube.com/watch?v=Pr6EptC9JXQ
UCS server RAID Policies http://www.youtube.com/watch?v=Vcs56wjUWuI
UCS server Firmware Policies http://www.youtube.com/watch?v=vjj8Xz0NqI4
UCS server Server Pools and Qualification Policies http://www.youtube.com/watch?v=KTw7M3T-VOw
UCS server Maintenance Policies http://www.youtube.com/watch?v=QQTlm98NgTI
UCS server High Availability During Upgrades http://www.youtube.com/watch?v=57HXMGn88HA
UCS server Monitoring with BMC BPPM http://www.youtube.com/watch?v=mdoEZf7tM5E
UCS server Microsoft Hyper-V on UCS http://www.youtube.com/watch?v=G3x_YOYK-Fo
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 84
Category Title URL
UCS I/O Adapter Templates http://www.youtube.com/watch?v=KpVEn3DhfOM
UCS I/O Network Interface Virtualization http://www.youtube.com/watch?v=njjbCEblxVc
UCS I/O Adapter Fabric Failover http://www.youtube.com/watch?v=tlu8RSq6T_M
UCS I/O Extend the Network to the Virtual Machine http://www.youtube.com/watch?v=Ylizxq18yxE
UCS I/O Traffic Analysis of All Servers http://www.youtube.com/watch?v=PHTdXy_8Zdg
UCS I/O Ethernet Switching Modes http://www.youtube.com/watch?v=roX8MRN66UM
UCS I/O Fibre Channel and Switch Modes http://www.youtube.com/watch?v=VSetsgOYYCo
UCS I/O FC Port Channels and Trunking http://www.youtube.com/watch?v=PpzKPguRTXc
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 85
Category Title URL
UCS
Infrastructure
Lights-Out Management http://www.youtube.com/watch?v=QEO1d_1vTxs
UCS
Infrastructure
Easy VM-FEX Deployment http://www.youtube.com/watch?v=0aAuj80cNvg
UCS
Infrastructure
Server Power Grouping http://www.youtube.com/watch?v=EgoFe33YoD8
UCS
Infrastructure
Blade and Rack-Mount Management http://www.youtube.com/watch?v=aOsx4YMiOho
UCS
Infrastructure
Manager Platform Emulator http://www.youtube.com/watch?v=ZNNrs2e0wvk
UCS
Infrastructure
Cisco Developer Network and Sandbox http://www.youtube.com/watch?v=Syhl6SAiwew
© 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential Presentation_ID 86
Thank you.