BRKDCT-2868

© 2009 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKDCT-2868 1

Network Integration of Server Virtualization with LAN and StorageNetwork Implications & Best Practices

BRKDCT-2868

Bjørn R. Martinussen

2© 2009 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKDCT-2868

Housekeeping

� We value your feedback- don't forget to complete your online session evaluations after each session & complete the Overall Conference Evaluation which will be available online from Thursday

� Visit the World of Solutions

� Please remember this is a 'non-smoking' venue!

� Please switch off your mobile phones

� Please make use of the recycling bins provided

� Please remember to wear your badge at all times including the Party


Session Objectives

At the end of the session, the participants should be able to:

� Objective 1: Understand key concepts of server virtualization architectures as they relate to the network.

� Objective 2: Explain the impact of server virtualization on DC network design (Ethernet & Fiber Channel)

� Objective 3: Design Cisco DC networks to support server virtualization environments


Server Virtualization

Network Implications of Server Virtualization & Best Practices


Virtualization

CPU

Mofied Stripped Down OS with

Hypervisor

Guest OS

App

VM

CPU

Host OS

VM

Hypervisor

VMware Microsoft

CPU

Modified OS

AppVM

Mofied Stripped Down OS with

Hypervisor

XEN aka Paravirtualization

Guest OS

App

Guest OS

App

Guest OS

App

Modified OS

App




VMware Architecture in a Nutshell

ESX Server Host

VirtualMachines

…

ProductionNetwork

MgmtNetwork

VM KernelNetwork

OS OS OS

ConsoleOS

App. App. App.

VM Virtualization Layer

Physical Hardware

CPU

memory


VMware HA Clustering

CPUmem

ory

ESX Host 2

Hypervisor

CPUmem

ory

ESX Host 1

Hypervisor

Guest OS

App1

Guest OS

App2

CPUmem

ory

ESX Host 3

Hypervisor

Guest OS

App3

Guest OS

App4

Guest OS

App5Guest OS

App1

Guest OS

App2


Application-level HA clustering(Provided by MSCS, Veritas etc…)

CPUmem

ory

ESX Host 2

Hypervisor

CPUmem

ory

ESX Host 1

Hypervisor

Guest OS

App1

Guest OS

App2

CPUmem

ory

ESX Host 3

Hypervisor

Guest OS

App3

Guest OS

App4

Guest OS

App5

Guest OS

App1

Guest OS

App2


HA + DRS

� HA takes care of Powering on VMs on available ESX hosts in the least possible time (regular migration, not VMotion based)

� DRS takes care of migrating the VMs over time to the most appropriate ESX host based on resource allocation (VMotion migration)


Agenda

� VMware LAN Networking

vSwitch Basics

NIC Teaming

vSwitch vs LAN Switch

Cisco/VMware DC DESIGNS

� Vmware SAN Designs

� VMware Virtual Networking


VMware Networking ComponentsPer ESX-server configuration

VMNICS = uplinksvSwitchVMs

vmnic0

vmnic1

vNIC

vNIC

Virtual Ports

VM_LUN_0007

VM_LUN_0005

vSwitch0


vNIC MAC Address

� VM’s MAC address automatically generated

� Mechanisms to avoid MAC collision

� VM’s MAC address doesn’t change with migration

� VM’s MAC addresses can be made static by modifying the configuration files

� ethernetN.address = 00:50:56:XX:YY:ZZ

� /vmfs/volumes/46b9d79a-2de6e23e-929d-001b78bb5a2c/VM_LUN_0005/VM_LUN_0005.vmx

� ethernet0.addressType = "vpx"

� ethernet0.generatedAddress = "00:50:56:b0:5f:24„

� ethernet0.addressType = „static“

� ethernet0.address = "00:50:56:00:00:06„


vSwitch Forwarding Characteristics

� Forwarding based on MAC address (No Learning): If traffic doesn’t match a VM MAC is sent out to vmnic

� VM-to-VM traffic stays local

� vSwitches TAG traffic with 802.1q VLAN ID

� vSwitches are 802.1q Capable

� vSwitches can create Etherchannels





VM �� Port-Group ��vSwitch


VLAN’s - External Switch Tagging - EST

PhysicalSwitches

VLAN tagging and stripping is done by the physical switch

No ESX configuration required as the server is not tagging

The number of VLAN’ssupported is limited to the number of physical NIC’s in the server

VM1 VM2 ServiceConsole

VMkernel

VMkernelNIC VSwitch A VSwitch B ESX

Server

Virtual NIC’s

VLAN 100 VLAN 200

Physical NIC’s


VLAN’s - Virtual Switch Tagging - VST

PhysicalSwitches

The vSwitch tags outgooing frames with the VLAN Id

The vSwitch strips any dot1Q tags before delivering to the VM

Physical NIC’s and switch port operate as a trunk

Number of VLAN’s are limited to the number of vNIC’s

No VTP or DTP. All static config. Prune VLAN’s so ESX doesn’t process broadcasts


VMkernel

VMkernelNIC VSwitch A ESX

Server

Virtual NIC’s

VLAN 100 VLAN 200

Physical NIC’sdot1Q


VLAN’s - Virtual Guest Tagging - VGT

PhysicalSwitches

Portgroup VLAN Id set to 4095

Tagging and stripping of VLAN id’s happens in the guest VM – requires an 802.1Q driver

Guest can send/receive any tagged VLAN frame

Number of VLAN’s per guest are not limited to the number of VNIC’s

VMware does not ship with the driver:

Windows E1000

Linux dot1q module


VMkernel

VMkernelNIC VSwitch A ESX

Server

Virtual NIC’s

VLAN 100 VLAN 200

Physical NIC’sdot1Q

dot1QVM applied


Agenda


vSwitch Basics

NIC Teaming




Meaning of NIC Teaming in VMware (1)

ESX Server Host

vSwitch Uplinks

vmnic0 vmnic1 vmnic2 vmnic3

vNIC vNICvNIC

vNIC

vNIC

ESX server NIC cards

NIC Teaming NIC Teaming

THIS IS NOT NIC Teaming


Meaning of NIC Teaming in VMware (2)T

his

is N

OT

Tea

min

g

Teaming is Configured at The vmnic Level


vSwitch0

VM1

vmnic0 vmnic1

Service ConsoleVM2

Port-Group 1VLAN 2

Port-Group 2VLAN 1

802.1qVlan 1,2

802.1qVlan 1,2

ESX Server

Design Example 2 NICs, VLAN 1 and 2, Active/Standby


Active/Standby per-Port-Group

VM5

VMNIC0

VM7 VM4 VM6

VMNIC1

.5 .7 .4 .6

CBS-rightCBS-left

Port-Group2Port-Group1

ESX Server

vSwitch0


Port-Group Overrides vSwitch Global Configuration


Active/Active

vmnic0 vmnic1

ESX server NIC cards

vSwitch

ESX server

VM1 VM2 VM3 VM4 VM5

Port-Group


Active/ActiveIP-Based Load Balancing

� Works with Channel-Group mode ON

� LACP is not supported (see below):

9w0d: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/14, changed state to up

9w0d: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/13, changed state to up

9w0d: %EC-5-L3DONTBNDL2: Gi1/0/14 suspended: LACP currently not enabled on the remote port.

9w0d: %EC-5-L3DONTBNDL2: Gi1/0/13 suspended: LACP currently not enabled on the remote port.

vmnic0 vmnic1

vSwitch

ESX server

VM1 VM2 VM3 VM4

Port-Group

Port-channeling


Agenda


vSwitch Basics

NIC Teaming




All Links Active, No Spanning-TreeIs There a Loop?

VM5

NIC1 NIC2

VM7 VM4 VM6

vSwitch1

NIC3 NIC4

.5 .7 .4 .6

CBS-rightCBS-left

Port-Group2Port-Group1

ESX Server


Broadcast/Multicast/Unknown UnicastForwarding in Active/Active (1)

vSwitch0

VM1

vmnic0 vmnic1

VM2

Port-Group 1VLAN 2

802.1qVlan 1,2

802.1qVlan 1,2

ESX Server


Broadcast/Multicast/Unknown UnicastForwarding in Active/Active (2)

vSwitch

VM1

NIC1 NIC2

VM2

ESX Host

802.1qVlan 1,2

802.1qVlan 1,2

VM3


Can the vSwitch Pass Traffic Through?

vSwitch

VM1

NIC1 NIC2

VM2

E.g. HSRP?


Is This Design Possible?

vSwitch

VM5 VM7

802.1q

802.1q

.5 .7

ESX server1

VMNIC1 VMNIC2

12

Catalyst1 Catalyst2


vSwitch Security

� Promiscuous mode Reject prevents a port from capturing traffic whose address is not the VM’saddress

� MAC Address Change, prevents the VM from modifying the vNICaddress

� Forget Transmits prevents the VM from sending out traffic with a different MAC (e.g NLB)



� Similarly to a LAN Switch:

Forwarding based on MAC address

VM-to-VM traffic stays local

Vswitches TAG traffic with 802.1q VLAN ID

vSwitches are 802.1q Capable

vSwitches can create Etherchannels

Preemption Configuration (similar to Flexlinks, but no delay preemption)

� Differently from a LAN Switch

No Learning

No Spanning-Tree protocol

No Dynamic trunk negotiation (DTP)

No 802.3ad LACP

2 Etherchannel backing up each other is not possible

No SPAN/mirroring capabilities: Traffic capturing is not the equivalent of SPAN

Port Security limited


Agenda


vSwitch Basics

NIC Teaming




vSwitch and NIC Teaming Best Practices

� Q: Should I use multiple vSwitchesor multiple Port-Groups to isolate traffic?

� A: We didn’t see any advantage in using multiple vSwitches, multiple Port-Groups with different VLANsgive you enough flexibility to isolate servers

� Q: Should I use EST or VST?

� A: Always use VST, i.e. assign the VLAN from the vSwitch

� Q: Can I use native VLAN for VMs?

� A: Yes you can, but to make it simple don’t . If you do, do not TAG VMs with the native VLAN

� Q: Which NIC Teaming configuration should I use?

� A: Active/Active, Virtual Port-ID based

� Q: Do I have to attach all NICs in the team to the same switch or to different switches ?

� A: with Active/Active Virtual Port-ID based, it doesn’t matter

� Q: Should I use Beaconing?

� A: No

� Q: Should I use Rolling Failover (i.e. no preemption)

� A: No, default is good, just enable trunkfast on the Cisco switch


Cisco Switchport Configuration

� Make it a Trunk

� Enable Trunkfast

� Can the Native VLAN be used for VMs?

� Yes, but IF you do, you have 2 options

Configure VLAN ID = 0 for the VMsthat are going to use the native VLAN (preferred)

Configure “vlan dot1q tag native” on the 6k (not recommended)

� Do not enable Port Security (see next slide)

� Make sure that “teamed” NICs are in the same Layer 2 domain

� Provide a Redundant Layer 2 path

� interface GigabitEthernetX/X

� description <<** VM Port **>>

� no ip address

� switchport

� switchport trunk encapsulation dot1q

� switchport trunk native vlan <id>

� switchport trunk allowed vlan xx,yy-zz

� switchport mode trunk

� switchport nonegotiate

� no cdp enable

� spanning-tree portfast trunk

� !

Typically: SC, VMKernel, VM Production


Configuration with 2 NICSC, VMKernel, Production Share NICs

Trunks

VM1

VMNIC1 VMNIC2

VM2

802.1q: Production VLANs,Service Console, VM Kernel 802.1q

ServiceConsole VM Kernel

ESX Server

vSwitch 0

Port-Group2

Port-Group3

Port-Group1

HBA1 HBA2

NIC teamingActive/Active

VST

Global Active/Active

Active/StandbyVmnic1/vmnic2

Active/StandbyVmnic2/vmnic1


Configuration with 2 NICsDedicated NIC to SC, VMKernel, Separate NIC for Production

Trunks

VM1

VMNIC1 VMNIC2

VM2

802.1q: Production VLANs,Service Console, VM Kernel 802.1q

ServiceConsole VM Kernel

ESX Server

vSwitch 0

Port-Group2

Port-Group3

Port-Group1

HBA1 HBA2

NIC teamingActive/Active

VST

Global Active/StandbyVmnic1 /vmnic2

Active/StandbyVmnic2 /vmnic1

Active/StandbyVmnic2 /vmnic1


Network Attachment (1)

802.1q802.1q:

Production,SC, VMKernel

ESX server1 ESX server 2

VMNIC1 VMNIC2

12 3

4

Catalyst1 Catalyst2

VMNIC1 VMNIC2

No Blocked Port,No Loop

All NICs are usedTraffic distributed

On all links

802.1q:Production,

SC, VMKernel

rootSecondary

root

TrunkfastBPDU guard

vSwitch vSwitch

Rapid PVST+



802.1q802.1q:

Production,SC, VMKernel


VMNIC1 VMNIC2

12 3

4

VMNIC1 VMNIC2

All NICs are usedTraffic distributed

On all links

Typical Spanning-TreeV-Shape Topology

802.1q:Production, SC, VMKernelroot

Secondary root

TrunkfastBPDU guard

vSwitchvSwitch

Rapid PVST+















vSwitch

802.1q:Production


1 2 7

Catalyst1 Catalyst2

vSwitch

No Blocked Port,No Loop

802.1q:Production,

SC, VMKernel

rootSecondary

root

TrunkfastBPDU guard

802.1q:SC and VMKernel

3

4 56

8

Rapid PVST+





Secondary root

TrunkfastBPDU guard

vSwitch

802.1q:Production


1 27

Catalyst1 Catalyst2

vSwitch


34 5

6 8

Rapid PVST+


How About?



Secondary root

TrunkfastBPDU guard

vSwitch

802.1q:Production


1 27

Catalyst1 Catalyst2

vSwitch


34 5

6 8


4 NICs with Etherchannel

802.1q:Production


12

73

4 5

6 8

“Clustered” switches

802.1q:SC, VMKernel

vSwitch vSwitch


VMotion Migration Requirements


VMKernel Network can be routed

ESX Server Host

VirtualMachines

…

ProductionNetwork

MgmtNetwork

VM KernelNetwork

VM KernelNetwork


VMotion L2 Design

VM4 VM5ESX Host 2 VM6

vSwitch0

vmnic0 vmnic1

vSwitch1 vSwitch2

vmnic2 vmnic3

vmkernel Serviceconsole

Rack10Rack1

ESX Host 1

vSwitch0

vmnic0

vSwitch2

vmnic2

vmkernel


HA clustering (1)

� EMC/Legato AAM based

� HA Agent runs in every host

� Heartbeats Unicast UDP port ~8042 (4 UDP ports opened)

� Hearbeats run on the Service Console ONLY

� When a Failure Occurs, the ESX Host pings the gateway (on the SERVICE CONSOLE ONLY) to verify Network Connectivity

� If ESX Host is isolated, it shuts down the VMs thus releaseinglocks on the SAN

� Recommendations:Have 2 Service Console on redundant paths

Avoid losing SAN access (e.g. via iSCSI)

Make sure you know before hand if DRS is activated too!

� Caveats:Losing Production VLAN connectivity only, ISOLATES VMs(there’s no equivalent of uplink tracking on the vswitch)

� Solution:NIC TEAMING


HA clustering (2)

COS 10.0.2.0

ESX2 Server Host

vmnic0

10.0.200.0

ESX1 Server Host

vmnic0

Prod 10.0.100.0

VM1 VM2

VM1 VM2

iSCSI access/VMkernel


Agenda


� VMware SAN Designs

Storage Fundamentals

Storage Protocols



Multiple ESX Servers—Shared Storage


Storage

Servers

ESXServer

ESXServer

ESXServer

ESXServer

Virtual Machines

A.vmdk

VMFS VMFS VMFSVMFS

VMFS

� Stores the entire virtual machine state in a central location

� Supports heterogeneous storage arrays

� Adds more storage to a VMFS volume dynamically

� Allows multiple ESX Servers to access the same virtual machine storage concurrently

� Enable virtualization-based distributed infrastructure services such as VMotion, DRS, HA

VMFS Is High Performance Cluster File System for Virtual Machines


Three Layers of the Storage Stack

Virtual Machine

ESX Server

Storage Array

Virtualdisks(VMDK)

DatastoresVMFS Vols(LUNs)

Physical disks


ESX Server View of SAN

� FibreChannel disk arrays appear as SCSI targets (devices) which may have one or more LUNs

� On boot, ESX Server scans for all LUNs by sending inquiry command to each possible target/LUN number

� Rescan command causes ESX Server to scan again, looking for added or removed targets/LUNs

� ESX Server can send normal SCSI commands to any LUN, just like a local disk


ESX Server View of SAN (Cont.)

� Built-in locking mechanism to ensure multiple hosts can access same disk on SAN safely

VMFS-2 and VMFS-3 are distributed file systems, do appropriate on-disk locking to allow many ESX Server servers to access same VMFS

� Storage is a resource that must be monitored and managed to ensure performance of VM’s

Leverage 3rd-party systems and storage management tools

Use VirtualCenter to monitor storage performance from virtual infrastructure point of view


Choices in Protocol

� FC, iSCSI or NAS?

Best practice to leverage the existing infrastructure

Not to introduce too many changes all at once

Virtual environments can leverage all types

You can choose what fits best and even mix them

Common industry perceptions and trade offs still apply in the virtual world

What works well for one does not work for all


Which Protocol to Choose?

� Leverage the existing infrastructure when possible

� Consider customer expertise and ability to learn

� Consider the costs (Dollars and Performance)

� What does the environment need in terms of throughput

Size for aggregate throughput before capacity

� What functionality is really needed for Virtual Machines

Vmotion, HA, DRS (works on both NAS and SAN)

VMware Consolidated Backup (VCB)

ESX boot from disk

Future scalability

DR requirements


FC SAN—Considerations

� Leverage multiple paths for high availability

� Manually distribute I/O intensive VMs on separate paths

� Block access provides optimal performance for large high transactional throughput work loads

� Considered the industrial strength backbone for most large enterprise environments

� Requires expertise in storage management team

� Expensive price per port connectivity

� Increasing to 10 Gb throughput (Soon)


iSCSI—Considerations

� Uses standard NAS infrastructureBest Practice to

Have dedicated LAN/VLAN to isolate from other network traffic

Use GbE or faster network

Use multiple NICs or iSCSI HBAs

Use iSCSI HBA for performance environments

Use SW initiator for cost sensitive environments

� Supports all VI 3 featuresVmotion, DRS, HA

ESX boot from HW initiator only

VCB is in experimental support today – full support shortly


NFS—Considerations

� Has more protocol overhead but less FS overhead than VMFS as the NAS FS lives on the NAS Head

� Simple to define in ESX by providing

Configure NFS server hostname or IP

NFS share

ESX Local datastore name

� No tuning required for ESX as most are already defined

No options for rsize or wsize

Version is v3,

Protocol is TCP

� Max mount points = 8 by default

Can be increase to hard limit of 32

� Supports almost all VI3 features except VCB


Summary of Features Supported

Protocol Vmotion,

DRS & HA

VCB ESX boot

from disk

FC SAN

Yes Yes Yes

iSCSI SAN

HW init Yes Soon Yes

iSCSI SAN

SW init Yes Soon No

NFS

Yes No No


Choosing Disk Technologies

� Traditional performance factors

Capacity / Price

Disk types (SCSI, FC, SATA/SAS)

Access Time; IOPS; Sustained Transfer Rate

Drive RPM to reduce rotational latency

Seek time

Reliability (MTBF)

� VM performance gated ultimately by IOPS density and storage space

� IOPS Density -> Number of read IOPS/GB

Higher = better


The Choices One Needs to Consider

� FS vs. RawVMFS vs. RDM (when to use)

� NFS vs. Block

NAS vs. SAN (why use each)

� iSCSI vs. FCWhat is the trade off?

� Boot from SANSome times needed for diskless servers

� Recommended Size of LUNit depends on application needs…

� File system vs. LUN snapshots (host or array vs. Vmware VMFS snapshots) – which to pick?

� Scalability (factors to consider) # hosts, dynamic adding of capacity, practical vs. physical limits


Trade Offs to Consider

� Ease of provisioning

� Ease of on-going management

� Performance optimization

� Scalability – Head room to grow

� Function of 3rd Party services

Remote Mirroring

Backups

Enterprise Systems Management

� Skill level of administration team

� How many shared vs. isolated storage resources


Isolate vs. Consolidate Storage Resources

� RDMs map a single LUN to one VM

� One can also dedicate a single VMFS Volume to one VM

� When comparing VMFS to RDMs both the above configurations are what should be compared

� The bigger question is how many VM can share a single VMFS Volume without contention causing pain

� The answer is that it depends on many variables

Number of VMs and their workload type

Number of ESX servers those VM are spread across

Number of concurrent request to the same disk sector/platter


Isolate vs. Consolidate

� Increased utilization� Easier provisioning� Less management

� Poor utilization� Islands of allocations � More management


Where Have You Heard This Before

� Remember the DAS � SAN migration

� Convergence of LAN and NAS

� All the same concerns have been raised before

What if the work load of some cause problems for all?

How will we know who is taking the lions share of resource?

What if it does not work out?

The Earth Is Flat!

If Man Were Meant to fly He Would Have Wings

Our Biggest Obstacle Is Conventional Wisdom!


VMFS vs. RDM—RDM Advantages

� Virtual machine partitions are stored in the native guest OS file system format, facilitating “layered applications” that need this level of access

� As there is only one virtual machine on a LUN, you have much finer grain characterization of the LUN,and no I/O or SCSI reservation lock contention. The LUN can be designed for optimal performance

� With “Virtual Compatibility” mode, virtual machines have many of the features of being on a VMFS, such as file locking to allow multiple access, and snapshots


VMFS vs. RDM—RDM Advantages

� With “Physical Compatibility” mode, it gives a virtual machine the capability of sending almost all “low-level” SCSI commands to the target device, including command and control to a storage controller, such as through SAN Management agents in the virtual machine.

� Dynamic Name Resolution: Stores unique information about LUN regardless of changes to physical address changes due to hardware or path changes


VMFS vs. RDM—RDM Disadvantages

� Not available for block or RAID devices that do not report a SCSI serial number

� No snapshots in “Physical Compatibility” mode, only available in “Virtual Compatibility” mode

� Can be very inefficient, in that, unlike VMFS, you can only have one VM access a RDM


RDMs and Replication

� RDMs mapped RAW LUNs can be replicated to the Remote Site

� RDMs reference the RAW LUNs via

the LUN number

LUN ID

� VMFS3 Volumes on Remote site will have unusable RDM configuration if either properties change

� Remove the old RDMs and recreate them

Must correlate RDM entries to correct RAW LUNs

Use the same RDM file name as old one to avoid editing the vmx file


Storage—Type of Access

� RAW

� RAW may give better performance

� RAW means more LUNs

More provisioning time

� Advanced features still work

� VMFS

� Leverage templates and quick provisioning

� Fewer LUNs means you don’t have to watch Heap

� Scales better with Consolidated Backup

� Preferred Method


Storage—How Big Can I Go?

� One Big Volume or Individual?

Will you be doing replication?

More granular slices will help

High performance applications?

Individual volumes could help

With Virtual Infrastructure 3

VMDK, swap, config files, log files, and snapshots all live on VMFS


What Is iSCSI?

� A SCSI transport protocol, enabling access to storage devices over standard TCP/IP networks

Maps SCSI block-oriented storage over TCP/IP

Similar to mapping SCSI over Fibre Channel

� “Initiators”, such as an iSCSI HBA in an ESX Server, send SCSI commands to “targets”, located in iSCSIstorage systems

Block storage

IP


VMware iSCSI Overview

� VMware added iSCSI as a supported option in VI3

Block-level I/O over TCP/IP using SCSI-3 protocol

Supporting both Hardware and Software Initiators

GigE NiCs MUST be used for SW Initiators (no 100Mb NICs)

Support iSCSI HBAs (HW init) and NICs for SW only today

Check the HCL for supported HW Initiators and SW NICs

� What we do not support in ESX 3.0.1

10 gigE

Jumbo Frames

Multi Connect Session (MCS)

TCP-Offload Engine (TOE) Cards


VMware ESX Storage Options

� 80%+ of install base uses FC storage

� iSCSI is popular in SMB market

� DAS is not popular because it prohibits VMotion

SCSIFC

VM VM

FC

iSCSI/NFS

VM VM

DAS

VM VM

FC


FC

Storage Array(LUN Mapping and Masking)MDS9000

Zone FC Name Server

pWWN-P

Single Login on a Single Point-to-Point Connection

Virtual Servers Share a Physical HBA� A zone includes the physical hba and

the storage array

� Access control is demanded to storage array “LUN masking and mapping”, it is based on the physical HBA pWWN and it is the same for all VMs

� The hypervisor is in charge of the mapping, errors may be disastrous

HW

Hyp

ervi

sor

Virt

ual

Ser

vers

pWWN-P

Mapping

FC


NPIV Usage Examples

‘Intelligent Pass-thru’Virtual Machine Aggregation

FC FC FC FC

NP_Port

F_PortF_Port

FC FC FC FC

FC

NPIV enabled HBA

Switch becomes an HBA concentrator


Raw Device Mapping

� RDM allows direct read/write access to disk

� Block mapping is still maintained within a VMFS file

� Rarely used but important for clustering (MSCS supported)

� Used with NPIV environments

FC

VM1 VM2

FC

RDM

VMFS

Mapping

FC


Storage Multi-Pathing

� No storage load balancing, strictly failover

� Two modes of operation dictate behavior (Fixed and Most Recent)

� Fixed ModeAllows definition of preferred paths

If preferred path fails a secondary path is used

If preferred path reappears it will fail back

� Most Recently UsedIf current path fails a secondary path is used

If previous path reappears the current path is still used

� Supports both Active/Active and Active/Passive arrays

� Auto detects multiple paths

FC

VM VM

FC


Q and A


Recommended Reading


Agenda


� VMware SAN Designs



Server Virtualization

Cisco Nexus 1000v Virtual Switch


VI 3.5 Network Configuration






Nexus 1000V ‘Virtual Chassis’ Model

� One Virtual Supervisor Module managing multiple Virtual Ethernet Modules

Dual Supervisors to support HA environments

� A single Nexus 1000V can span multiple ESX Clusters

SVS-CP# show moduleMod Ports Module-Type Model Status--- ----- -------------------------------- ------------------ ------------1 1 Supervisor Module Cisco Nexus 1000V active *2 1 Supervisor Module Cisco Nexus 1000V standby3 48 Virtual Ethernet Module ok4 48 Virtual Ethernet Module ok


Upstream-4948-1#show cdp neighborCapability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge

S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone

Device ID Local Intrfce Holdtme Capability Platform Port ID

N1KV-Rack10 Gig 1/5 136 S Nexus 1000V Eth2/2N1KV-Rack10 Gig 1/10 136 S Nexus 1000V Eth3/5N1KV-Rack10 Gig 1/12 136 S Nexus 1000V Eth21/2

� A single switch from control plane and management plane perspective

Protocols such as CDP operates as a single switch

XML API and SNMP management appears as a single ‘vi rtual chassis’

Single Chassis Management









Deploying the Cisco Nexus 1000V

1. VMW Virtual Center & Cisco Nexus 1000V relationship established

2. Network Admin configures Nexus 1000V to support new ESX hosts

3. Server Admin plugs new ESX host into network & adds host to Cisco DVS with Virtual Center

4. Repeat step 3 to add another host & extend DVS configuration

Collaborative Deployment ModelCollaborative Deployment Model

Virtual CenterVirtual Center

VMW ESXVMW ESXVMW ESX

Server 1Server 1

Nexus 1000V -VEMNexus 1000V Nexus 1000V --VEMVEM

Nexus 1000V

VSM

Nexus 1000VNexus 1000V

VSMVSM


Server NServer N


4. 4.

Nexus 1000V DVSNexus 1000V DVSNexus 1000V DVS


Introduction to Port Profiles

� Port Profiles are a collection ‘interface’ commands

switchport mode access

switchport access vlan 57

no shutdown

� Applied at the interface level using to either physical or virtual interfaces

� Dynamic configuration

Port Profile changes are propagated immediately to all ports using that profile

� Interfaces can be configured manually in conjunction with a profile


Port Profiles Propagation

� Port profiles are pushed via the Virtual Center API

� Upon connection/reconnection with Virtual Center the VSM re-verifies the correct port profile configuration exists within Virtual Center

� Port profile ‘state’ and ‘type’ must be set for propagation to occur

N1K-CP(config-port-prof) state enable

N1K-CP(config-port-prof) vmware port-group (optional name)



ServerServer

Policy Based VM ConnectivityWhat can a policy do? What can a policy do?

Virtual CenterVirtual Center Nexus 1000Nexus 1000

Nexus 1000 DVSNexus 1000 DVSNexus 1000 DVS

VM #1VM VM #1#1

VM #4VM VM #4#4

VM #3VM VM #3#3

VM #2VM VM #2#2

Policy definition supports:Policy definition supports:•• VLAN, PVLAN settingsVLAN, PVLAN settings

•• ACL, Port Security, ACL ACL, Port Security, ACL RedirectRedirect

•• NetFlowNetFlow CollectionCollection

•• Rate LimitingRate Limiting

•• QoSQoS Marking (COS/DSCP) Marking (COS/DSCP)

•• Remote Port Mirror (ERSPAN)Remote Port Mirror (ERSPAN)





VMware Administrator View

� Consistent Workflow: Continue to select Port Groups when configuring a VM in VMware Virtual Infrastructure Client






Server 2Server 2

Nexus 1000 -VEMNexus 1000 Nexus 1000 --VEMVEM


Server 1Server 1



Nexus 1000V

VSM

Nexus 1000VNexus 1000V

VSMVSM

Nexus 1000V DVSNexus 1000V DVSNexus 1000V DVS

VM #5VM VM #5#5

VM #8VM VM #8#8

VM #7VM VM #7#7

VM #6VM VM #6#6

1. Virtual Center kicks off a Vmotion(manual/DRS) & notifies Nexus 1000V

2. During VM replication, Nexus 1000V copies VM port state to new host

3. Once VMotioncompletes, port on new ESX host is brought up & VM’s MAC address is announced to the network

VM #4VM VM #4#4

VM #3VM VM #3#3

VM #2VM VM #2#2

VM #1VM VM #1#1

Network UpdateNetwork Update••ARP for VM1 sent to ARP for VM1 sent to networknetwork••Flows to VM1 MAC Flows to VM1 MAC redirected to Server 2redirected to Server 2

3. 3.

Mobility of Security & Network Properties




1. Works with all types of servers (rack optimized, blade servers, etc.)

2. Works with any type of upstream switch (Blade, Top or Rack, Modular)

3. Works at any speed (1G or 10G)

4. Nexus 1000V VSM can be deployed as a VM or a physical appliance

Blade Servers

Rack OptimizedServers

Nexus 1000V

VSM

Nexus 1000V Deployment Scenarios





VSM VSM VSM

VSMVSMVSM

VSM Virtual Appliance� ESX Virtual Appliance� Special dependence on CPVA

server

� Supports up to 64 VEMs

VMW ESX

Server 3

VM #9

VM #12

VM #11

VM #10

VEMVMW ESX

Server 2

VM #5

VM #8

VM #7

VM #6

VEMVMW ESX

Server 1

VM #1

VM #4

VM #3

VM #2

VEM

VSM Physical Appliance� Cisco branded x86 server� Runs multiple instances of the

VSM virtual appliance

� Each VSM managed independently

Virtual Supervisor Options


� Each Virtual Ethernet Module behaves like an independent switch

No address learning/synchronization across VEMs

No concept of Crossbar/Fabric between the VEMs

Virtual Supervisor is NOT in the data path

No concept of forwarding from an ingress linecard to an egress linecard (another server)

No Etherchannel across VEMsNexus 1000V

VSM

VMW ESX

VEMVMW ESX

VEMVMW ESX

VEM

Distributed Switching


Switching Interface Types� Physical Ethernet Ports

NIC cards on each server

Appears as ‘Eth’ interface on a specific module in NX-OS

Example – ‘Eth10/7’

Static assignment as long as the module ID does not change

Up to 32 per host

� Virtual Ethernet Ports

Virtual Machine facing ports

Appears as ‘Veth’ within NX-OS.

Not assigned to a specific module to simplify VMotion


Tracing Virtual Ethernet Ports

show interface VEthernet

Vethernet2 is up

Hardware is Virtual, address is 0050.5675.26c5Owner is VMware VMkernel, adapter is vmk0Active on module 8, host tc-esx05.cisco.comVMware DVS port 16777215Port-Profile is VmotionPort mode is access

Rx444385 Input Packets 444384 Unicast Packets0 Multicast Packets 1 Broadcast Packets572675241 Bytes

Tx687655 Output Packets 687654 Unicast Packets0 Multicast Packets 1 Broadcast Packets 1 Flood Packets592295257 Bytes0 Input Packet Drops 0 Output Packet Drops


Manageability and Scalability Details

� RBAC

� Wireshark

� ERSPAN

� LLDP, CDP

� EEM

� Rollback

� Cisco Nexus 1000V Virtual Supervisor Module: Virtual appliance in VMDK or ISO image, supports up to 64 VMware ESX or ESXi

� Cisco Nexus 1000V Virtual Ethernet Module: maximum 256 ports


Server Virtualization Key Takeaways

What you should takeaway from this session:� Ability to explain the key concepts of server

virtualization and know the key players in the market.

� Ability to explain to customers key network design criteria which must be considered when deploying server virtualization

� Ability to recommend network and storage best practices associated with deploying server virtualization technologies.


Meet The Expert

To make the most of your time at Cisco Networkers 2009, schedule a Face-to-Face Meeting with a top Cisco Expert.

Designed to provide a "big picture" perspective as well as "in-depth" technology discussions, these face-to-face meetings will provide fascinating dialogue and a wealth of valuable insights and ideas.

Visit the Meeting Centre reception desk located in the Meeting Centre in World of Solutions


Whitepapers and blogs

� http://www.cisco.com/go/nexus1000v/

� http://www.cisco.com/en/US/products/ps9670/prod_white_papers_list.html

� “The Role of 10 Gigabit Ethernet in Virtualized Server Environments”

� http://blogs.vmware.com/networking/

� http://www.cisco.com/go/datacenter/ look for “VMware Infrastructure 3 in a Cisco Network Environment”

� http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/vmware/vmware.html


Date post:	04-Dec-2015
Category:	Documents
Upload:	kds20850
View:	7 times
Download:	2 times

BRKDCT-2868

Documents