Table of Contents - VMware · N-VDS DPDK Data Plane Development Kit (DPDK), is a set of libraries...

Table of ContentsLab Overview - HOL-1986-02 Configuring NSX-T for High Performance Virtual NetworkFunctions on vCloud NFV Platform .................................................................................... 2

Lab Guidance .......................................................................................................... 3Introduction............................................................................................................. 9

Module 1- Configuring N-VDS and SR-IOV (30 minutes).................................................. 17Introduction........................................................................................................... 18Hands-on Labs Interactive Simulation:SR-IOV....................................................... 19Manual Hardware SR-IOV Configuration ................................................................ 20Hands-on Labs Interactive Simulation:SR-IOV and N-VDS and how to migrate .....30Manual SR-IOV and N-VDS and how to migrate..................................................... 31Conclusion............................................................................................................. 59

Module 2- Saturating the link and performance scalability (30 minutes)........................60Introduction........................................................................................................... 61Hands-on Labs Interactive Simulation:Testing Performance of n-VDS NIC ............63Manual Testing Performance of n-VDS NIC ............................................................ 64Conclusion............................................................................................................. 80

HOL-1986-02-EMT

Page 1HOL-1986-02-EMT

Lab Overview -HOL-1986-02 Configuring

NSX-T for HighPerformance Virtual

Network Functions onvCloud NFV Platform

HOL-1986-02-EMT


Lab GuidanceNote: It may take more than 90 minutes to complete this lab. You shouldexpect to only finish 2-3 of the modules during your time. The modules areindependent of each other so you can start at the beginning of any moduleand proceed from there. You can use the Table of Contents to access anymodule of your choosing.

The Table of Contents can be accessed in the upper right-hand corner of theLab Manual.

This HOL will demonstrate how vSwitch can be configured for data plane workloads toachieve high throughput, low latency atnd very low packet loss in the 0.001% range allessential for Telco, 5G and IoT network deployments. The lab will highlight the featuresof a new virtual Switch architecture.

Lab Module List:

• Module 1 - Getting Started (15 minutes) (Basic) Learn about the Technologiesused in this lab

• Module 2 - Configuring N-VDS and SR-IOV(30 minutes) (Advanced) How tosetup N-VDS and SR-IOV from the hardware on up

• Module 3 - Testing and Validating N-VDS configuration (30 minutes)(Advanced) How to configure N-VDS and migrate from SR-IOV to N-VDS

• Module 4 - Saturating the link and Performance Scalability(30 minutes)(Advanced) How to test the speed and link as well as setup the VM forperformance.

Lab Captains:

• All Modules - Neil Moore, Staff Group Consulting Architect, USA

A special thanks to:

Abe AtsemetMike CraigenCraig HouseholderDharma Rajan

This lab manual can be downloaded from the Hands-on Labs Document site found here:

http://docs.hol.vmware.com

HOL-1986-02-EMT


public/77180/c/259390

public/77180/c/246086

http://docs.hol.vmware.com

This lab may be available in other languages. To set your language preference and havea localized manual deployed with your lab, you may utilize this document to help guideyou through the process:

http://docs.hol.vmware.com/announcements/nee-default-language.pdf

Location of the Main Console

1. The area in the RED box contains the Main Console. The Lab Manual is on the tabto the Right of the Main Console.

2. A particular lab may have additional consoles found on separate tabs in the upperleft. You will be directed to open another specific console if needed.

3. Your lab starts with 90 minutes on the timer. The lab can not be saved. All yourwork must be done during the lab session. But you can click the EXTEND toincrease your time. If you are at a VMware event, you can extend your lab timetwice, for up to 30 minutes. Each click gives you an additional 15 minutes.Outside of VMware events, you can extend your lab time up to 9 hours and 30

minutes. Each click gives you an additional hour.

Alternate Methods of Keyboard Data Entry

During this module, you will input text into the Main Console. Besides directly typing itin, there are two very helpful methods of entering data which make it easier to entercomplex data.

HOL-1986-02-EMT


http://docs.hol.vmware.com/announcements/nee-default-language.pdf

Click and Drag Lab Manual Content Into Console ActiveWindow

You can also click and drag text and Command Line Interface (CLI) commands directlyfrom the Lab Manual into the active window in the Main Console.

Accessing the Online International Keyboard

You can also use the Online International Keyboard found in the Main Console.

1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar.

<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><ahref="http://www.youtube.com/watch?v=xS07n6GzGuo" target="_blank">Try watching this video on www.youtube.com</a>, or enableJavaScript if it is disabled in your browser.</div></div>

HOL-1986-02-EMT


Click once in active console window

In this example, you will use the Online Keyboard to enter the "@" sign used in emailaddresses. The "@" sign is Shift-2 on US keyboard layouts.

1. Click once in the active console window.2. Click on the Shift key.

Click on the @ key

1. Click on the "@ key".

Notice the @ sign entered in the active console window.

HOL-1986-02-EMT


Activation Prompt or Watermark

When you first start your lab, you may notice a watermark on the desktop indicatingthat Windows is not activated.

One of the major benefits of virtualization is that virtual machines can be moved andrun on any platform. The Hands-on Labs utilizes this benefit and we are able to run thelabs out of multiple datacenters. However, these datacenters may not have identicalprocessors, which triggers a Microsoft activation check through the Internet.

Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoftlicensing requirements. The lab that you are using is a self-contained pod and does nothave full access to the Internet, which is required for Windows to verify the activation.Without full access to the Internet, this automated process fails and you see this

watermark.

This cosmetic issue has no effect on your lab.

Look at the lower right portion of the screen

HOL-1986-02-EMT


Please check to see that your lab is finished all the startup routines and is ready for youto start. If you see anything other than "Ready", please wait a few minutes. If after 5minutes you lab has not changed to "Ready", please ask for assistance.

HOL-1986-02-EMT


IntroductionIn this Module you will gain a high level overview learn about:

In this Module you will learn about:

• VMware vCloud NFV Platform 3.0 release highlights• VMware vCloud NFV OpenStack Edition 5.0• VMware NSX-T 2.2• SR-IOV• N-VDS

Getting Started

Adam works at Rainpole Telecom and is new to NFV. He is interested in using VMwarevCloud NFV platform for virtualizing data intensive workloads that need highthroughput. Solutions of Adam’s interest include Radio Access Virtualization, 5G, IoT etc.Adam is interested to know about NSX-T features that enable VM that can be configuredfor high throughput, meeting the packet mix criteria as per RC 2544. In this moduleAdam will learn the basics of VMware vCloud NFV platform as shown in Figure-1 andabout features of NSX-VDS (N-VDS) a new distributed vSwitch that works using theprinciples of Intel Data Plane Development Kit (DPDK).

HOL-1986-02-EMT


VMware vCloud NFV OpenStack Edition 5.0

From a Virtual Infrastructure Manager (VIM) perspective VMware enables customers touse VMware vCloud Director (vCD) or VMware vCloud® NFV™ OpenStack Carrier Edition(VIO) as VIM. This flexibility and freedom of choice in VIM, provides the fastest path forcommunications service providers (CSPs) to deploy network functions virtualization(NFV) services on a common platform infrastructure using the VIM of choice. In this HOLwe will use VIO as VIM.

VMware recently announced VIO 5.0 (powered by OpenStack Queens) which deliversscale and availability for hybrid applications across VM and container-based workloads.Figure below shows the openStack projects in VIO 5.0 and VMware technologyintegration stack. For NFV operators, VIO 5.0 will help fast track a path towards Edgecomputing with VIO-in-a-box, Kubernetes support, secure multi-tenant isolation andaccelerated network performance.

NSX-T Data Center

What is NSX -T?

HOL-1986-02-EMT


NSX-T is a multi-hypervisor capable networking and security platform.NSX-T is highlyextensible platform that address more endpoint heterogeneity including containers,Kubernetes support, public clouds and cross cloud services with AWS, and otherhypervisors. The architecture and critical components of NSX-T are as shown below.

As a leading network virtualization platform NSX-T supports ability to run telco gradeVirtual Network Functions (VNF). VNFs can be signaling plane, data plane, managementplane types. To support the needs for data plane intensive VNF, NSX-T 2.2 releasesupports an accelerated virtual switch called NSX-VDS (N-VDS in short). The switchprovides an Enhanced Data-path mode. Enhanced, because N-VDS runs using DPDKprinciples and allows containers and VMs to achieve significant improvements inresponse time, reduced network latencies and breakthrough network performance. Thisimproved performance is achieved while maintaining the operational simplicity ofvirtualized NICs, and enabling the VNFs to make use of all the goodness they can get inusing virtualization functionalities like vMotion, snapshot, distributed resourcescheduling, fault tolerance etc. which one cannot achieve if the bypass the virtualizationlayer by using techniques like SR-IOV or Direct passthrough mechanism.

The N-VDS

N-VDS Design Principle

N-VDS is built using the principles of Intel Data Plane Development Kit (DPDK). IntelDPDK is a set of libraries and drivers that support accelerated software processing forgreater throughput and scalability. Intel DPDK enables higher levels of packet processingthroughput than what is achievable using the standard Linux kernel network stack. This

HOL-1986-02-EMT


optimized library gives application developers the ability to address challenging dataplane processing needs that are typically found in Telecom, networking workloads.

NSX Manager

NSX Manager provides the graphical user interface (GUI) and the REST APIs for creating,configuring, and monitoring NSX-T components, such as controllers, logical switches,and edge services gateways.

NSX Manager is the management plane for the NSX-T eco-system. NSX Managerprovides an aggregated system view and is the centralized network managementcomponent of NSX-T. It provides a method for monitoring and troubleshooting workloadsattached to virtual networks created by NSX-T.

HOL-1986-02-EMT


The NSX-T Management Plane Agent (MPA) is an NSX Manager component that lives oneach and every node (hypervisor). The MPA is in charge of persisting the desired state ofthe system and for communicating non-flow-controlling (NFC) messages such asconfiguration, statistics, status and real time data between transport nodes and themanagement plane.

N-VDS incorporates DPDK elements and not the full DPDK library. NIC vendors developN-VDS drivers, through the standard IO Certification (NIOC) program that VMware offers.N-VDS vSwitch exposes a different SDK for drivers and hence the need for different NICdrivers.

HOL-1986-02-EMT


N-VDS DPDK

Data Plane Development Kit (DPDK), is a set of libraries and drivers that supportaccelerated software processing for greater throughput and scalability.

Intel DPDK enables higher levels of packet processing throughput than what isachievable using the standard Linux kernel network stack.

This optimized library gives application developers the ability to address challengingdata plane processing needs that are typically found in Telecom, networking workloads.

SR/IOV

Single Root I/O Virtualization (SR-IOV) uses physical functions (PFs) and virtual functions(VFs) to manage global functions for the SR-IOV devices. PFs are full PCIe functions thatinclude the SR-IOV Extended Capability which is used to configure and manage the SR-IOV functionality. It is possible to configure or control PCIe devices using PFs, and the PFhas full ability to move data in and out of the device. VFs are lightweight PCIe functionsthat contain all the resources necessary for data movement but have a carefullyminimized set of configurations.

SR-IOV allows for a single physical device to be shared amongst multiple guests, withindividual virtual functions (VFs) on the physical device to assigned to VM in the pass-through mode. The number of virtual functions presented depends on the device. For

HOL-1986-02-EMT


SR-IOV-enabled PCIe devices to function, you must have the appropriate BIOS andhardware support, as well as SR-IOV support in the guest driver or hypervisor instance.

Lab Architecture

This HOL lab setup is as shown in figure above. There are two key clusters. One is acompute cluster where the VNFs can be deployed on the hosts. The other is amanagement cluster. The Management cluster in this lab hosts vCenter appliance, NSXcontroller VMs, NSX-edge, NSX-Manger virtual appliance. On the compute side a VNF isconfigured to use SR-IOV. Another VNF is configured to use N-VDS. To show that a VNF

HOL-1986-02-EMT


configured using SR-IOV and can be migrated to N-VDS, two separate clusters arecreated for convenience to avoid reboot when migrating a VNF from SR-IOV enableddevice to a N-VDS enabled device.

HOL-1986-02-EMT


Module 1- Configuring N-VDS and SR-IOV (30

minutes)

HOL-1986-02-EMT


IntroductionThis module introduces Adam to the new concepts of NSX VDS (N-VDS), a new virtualswitch used to improve the performance of data plane intensive workloads. As part ofconfiguring the lab, activities like vSphere, vCenter, NSX-T as infrastructure componentsand VIO-CE as VIM are pre-deployed. Activities like preparing a common cluster withvSphere Distributed Switch (VDS) and N-VDS in the same cluster, configuring portgroups, transport zones are pre-completed.

Adam has used SR-IOV in the past. The deployment experience showcased variouschallenges and limitation he had to run into. While Adam noticed improve performancein the workload he noticed he was gaining that with significant risk and cost due todependency on hardware, limitations of virtualization functionality that he can use andthe resulting cost was significant. He thus wishes to first configure a VNF using SR-IOV,then learns about how simple it is to configure N-VDS, migrate the SR-IOV configuredVNF to N-VDS and then manage it through VMware Integrated OpenStack (VIO). Bydoing this Adam will learn:

1. How to configure and enable SR-IOV in a network card and BIOS setup for SR-IOVand N-VDS configuration

2. How to Configure a VNF to use with SR-IOV3. How to configure a VNF to use with N-VDS4. How to migrate an SR-IOV configured VNF to N-VDS5. How to use Non-Uniform Memory Access (NUMA)

How to configure and enable SR-IOV in a network card and BIOS setup for SR-IOV and N-VDS configuration

Single Root I/O Virtualization (SR-IOV) uses physical functions (PFs) and virtual functions(VFs) to manage global functions for the SR-IOV devices. PFs are full PCIe functions thatinclude the SRIOV Extended Capability which is used to configure and manage the SR-IOV functionality. It is possible to configure or control PCIe devices using PFs, and the PFhas full ability to move data in and out of the device. VFs are lightweight PCIe functionsthat contain all the resources necessary for data movement, but have a carefullyminimized set of configurations.

SR-IOV allows for a single physical device to be shared amongst multiple guests, withindividual virtual functions (VFs) on the physical device to assigned to VM in the pass-through mode. The number of virtual functions presented depends on the device. ForSR-IOV-enabled PCIe devices to function, you must have the appropriate BIOS andhardware support, as well as SR-IOV support in the guest driver or hypervisor instance.

In this section, we walk you through how to setup SR-IOV on a Dell 740XL utilizing anIntel NIC(Network Interface Card)

HOL-1986-02-EMT


Hands-on Labs InteractiveSimulation:SR-IOVThis part of the lab is presented as a Hands-on Labs Interactive Simulation. This willallow you to experience steps which are too time-consuming or resource intensive to dolive in the lab environment. In this simulation, you can use the software interface as ifyou are interacting with a live environment.

1. Click here to open the interactive simulation. It will open in a new browserwindow or tab.

2. When finished, click the “Return to the lab” link to continue with this lab.

The lab continues to run in the background. If the lab goes into standby mode, you canresume it after completing the module.

HOL-1986-02-EMT


http://docs.hol.vmware.com/hol-isim/HOL-2019/hol-1986-02-sriovconfig.htm

Manual Hardware SR-IOV ConfigurationIntroduction

In this how-to, we walk you through how to setup SR-IOV on a Dell 740XL utililzing anIntel NIC(Network Interface card)

Dell SR-IOV Configuration Example

The SR-IOV settings are set in the BIOS, which can be set when the server is booting up.We normally use the iDRAC to select boot into bios on the next boot up. The server

default boot order is to start loading the Operating System, but we need to change thatin order to boot up into the BIOS first.

Double click on the Google Chrome icon that says "idrac".

HOL-1986-02-EMT


Login to iDrac

1. You will need to login to iDrac using username: vmware and password:VMware1! and launch the virtual console.

HOL-1986-02-EMT


BIOS Setup

2. After you launch the console, you need to click on "Boot" and select "BIOS Setup" toenter into BIOS setting on the next reboot.

3. Now you can click on "Power" and power on the system to get into the BIOS setup.

System Setup

Please take a look at the following settings to fully enable SR-IOV. It has two areas forconfiguration, one at the global level, then another at the interface level. In thisexample, we will walk you through where the settings are.

HOL-1986-02-EMT


4. After the system boots into the BIOS, click on System BIOS.

HOL-1986-02-EMT


5. Click or arrow down to Integrated Devices, this is where the global setting for SR-IOV is at. This can be enabled for dual SR-IOV and N-VDS environments.

6. Click on the radio button for SR-IOV Global enable. If SR-IOV is disabled in the BIOS,you will not be able to use SR-IOV in vSphere. This setting does not affect N-VDS.

HOL-1986-02-EMT


then click on the [BACK] Button

Device Settings

The next steps are to enable SR-IOV on the individual network card. On some server,enabling SR-IOV globally doesn't enable it on the NIC, so you need to make sure it isenabled on the device level.

HOL-1986-02-EMT


7. Click on Device Settings.

8. Then click on NIC in Slot 1 Port 1: Intel In this HOL, we are using the INTEL X710.

HOL-1986-02-EMT


Some NICS might not have the same feature sets displayed. and different models ofmachines will display different additional capabilities.

HOL-1986-02-EMT


Device Level Configuration

9. Click on "Device Level Configuration"

HOL-1986-02-EMT


10. If the selection for Virtual Mode is set to "None" (The default). change the Optionto SR-IOV

Repeat the change on any other ports you require for SR-IOV. In the case that thisinterface will only be used for VMWare nVDS, enabling SR-IOV is not needed. You willnot be doing this for any other ports.

Conclusion

These settings are set in the BIOS, which can be set when the machine boots up, if theyare not set, the hypervisor/OS can not access the additional functionality provided bythe NIC.

HOL-1986-02-EMT


Hands-on Labs InteractiveSimulation:SR-IOV and N-VDS and howto migrateThis part of the lab is presented as a Hands-on Labs Interactive Simulation. This willallow you to experience steps which are too time-consuming or resource intensive to dolive in the lab environment. In this simulation, you can use the software interface as ifyou are interacting with a live environment.




HOL-1986-02-EMT


http://docs.hol.vmware.com/hol-isim/HOL-2019/hol-1986-02-sriovmigration.htm

Manual SR-IOV and N-VDS and how tomigrateIn this section we will examine how to migrate between SR-IOV and N-VDS.

What is SR-IOV?

Single Root I/O Virtualization (SR-IOV) uses physical functions (PFs) and virtual functions(VFs) to manage global functions for the SR-IOV devices. PFs are full PCIe functions thatinclude the SR-IOV extended capability which is used to configure and manage the SR-IOV functionality. The PF has full ability to move data in and out of the device, and it isalso possible to configure or control PCIe devices using PFs. Virtual Functions (VFs) arelightweight PCIe functions that contain all the resources necessary for data movementbut have a carefully minimized set of configurations.

SR-IOV allows for a single physical device to be shared amongst multiple guests, withindividual virtual functions (VFs) on the physical device assigned to VM in the pass-through mode. The number of virtual functions presented depends on the device. ForSR-IOV-enabled PCIe devices to function, you must have the appropriate BIOS andhardware support, as well as SR-IOV support in the guest driver or hypervisor instance.

HOL-1986-02-EMT


How to enable SR-IOV in vSphere WebClient

1. Click on Configure.2. Expand networking and click on Physical Adapter.3. Select the physical adaptors you want to enable the SR-IOV option.4. Click on the Pencil icon to edit the setting.5. Change the SR-IOV status from “Disabled” to “Enabled”6. Click on “Ok” to save the settings.

Once SR-IOV is enabled, a reboot is required for the change to take effect.

HOL-1986-02-EMT


After the host reboots, you need to make sure SR-IOV status show Enabled for thatNIC.

Assign a Virtual Function to a virtual machine in thevSphere Web Client

Virtual Functions (VFs) are lightweight PCIe functions that contain all the resourcesnecessary for data movement but have a carefully minimized set of configurations.

Before you try adding the SR-IOV PCI device to the VM Configuration you need to makesure that all prerequisites are met.

Prerequisites:

• Verify that the Physical NIC supports SR-IOV and you are using the latest driver/firmware for the NIC.

• Verify the SR-IOV is enabled in the BIOS. Keep in mind with some servers youmust enable it in two different places, one on the global config and seconddirectly under the NIC settings in BIOS.

• Verify SR-IOV is enabled on the host. You can enable it in vSphere Web Client.

HOL-1986-02-EMT


When adding the SR-IOV passthrough to the virtual machine, first you need to makesure the VM is powered off.

Steps:

1. Right-click on the powered off VM and select Edit Settings.2. Select "Network" from the "New Device" drop-down menu and click on "Add".3. Expand the "New Network" field and locate the "Adapter Type".4. Select "SR-IOV passthrough" for the adapter type.5. Click on “OK”

HOL-1986-02-EMT


Challenges with using SR-IOV for NFV

Adam has now learned about configuring SR-IOV, and has a data plane workload that isusing it. He had configured it like the above and used it mainly to achieve highthroughput. While he noticed throughput improvements he also noticed majordeficiency in SR-IOV and the inability to use many virtualization functions.

SR-IOV support status FAQ (2038739)

To overcome these limitations and also gain performance with lower cost, Adam decidedto use N-VDS. He takes the next step to learn about configuring N-VDS.

HOL-1986-02-EMT


https://kb.vmware.com/s/article/2038739

The N-VDS

N-VDS is the new VMware enhanced networking stack that incorporates DPDK elementsand not the full DPDK library. N-VDS is available with vSphere 6.7 and NSX-T 2.2. N-VDSvSwitch exposes a different SDK/DPDK for drivers and hence the need for different NICdrivers. You also need to make sure the firmware you are using match's the N-VDSdrivers. Intel 710 NIC with drivers that support N-VDS is being used for this HOL.

N-VDS is integrated to VMware NSX-T infrastructure base. This allows the switch to notonly deliver high performance but also support vSphere rich features like HA, DRS,Snapshot, vMotion etc. which are not available when a VNF is configured using SR-IOV

or other virtualization layer by-pass technology.

• N-VDS is supported for a VLAN backed network.• 10G and 40G NICs are supported.• Management of the switch is supported through VIO

VMs as data plane intensive works loads are connected to N-VDS. Three VMs as controlplane workloads with VDS and validation of network connectivity between them will bepart of this module exercise for the user. As part of N-VDS deployment validatingnumber of logical cores (lcores) allocated, maximum cores supported, VLAN basedlogical switch connectivity from SR-IOV to N-VDS will be done.

HOL-1986-02-EMT


Verify pNIC Driver

Before Adam starts using N-VDS he needs to make sure the pNIC driver is using thecorrect N-VDS.

You can run esxcfg-nic -e to see N_VDS driver is enabled on that pNIC. Below is a typicalscreenshot

New Transport Zone - Standard Mode

The N-VDS allows for virtual-to-physical packet flow by binding logical router uplinks anddownlinks to physical NICs. When you create a transport zone, you have the option tocreate as Standard or Enhanced Datapath.

N-VDS Standard Mode:With the standard mode, you have two Traffic Type.

1. Overly Traffic2. VLAN Traffic

HOL-1986-02-EMT


New Transport Zone - Enhanced Mode

N-VDS Enhanced Mode.When you add a transport node to a transport zone, the N-VDS associated with thetransport zone is installed on the transport node. Each transport zone supports a singleN-VDS. An enhanced datapath N-VDS has the performance capabilities to support NFV(Network Functions Virtualization) workloads, supports both VLAN and overlay networks,and requires an ESXi host that supports enhanced datapath N-VDS.

Migrating from SR-IOV to N-VDS backed network adapter

After learning about N-VDS features and benefit, Adam wants to migrate his workloadVM (HOL-1986-02-SRIOV-00) that is currently using SR-IOV back network adapter to N-VDS. After migrating to N-VDS, he will have the performance of SR-IOV with the addedbenefit of vSphere features, such as vMotion and DRS.

First, we are going to power on the VM name HOL-1986-02-SRIOV-00 and do a ping testwhile it is on the SR-IOV network adapter.

1. On the left panel click on VM "HOL-1986-02-SRIOV-00" to select it.

HOL-1986-02-EMT


2. Under the Menu click on "Action" and then "Power" to "Power On" the VM.

Launch Web Console

After the VM is powered on we are going to login to the VM and test the VM networkconnection by pining a physical workload.

HOL-1986-02-EMT


We need to open the VM Web Console by clicking on "Launch Web Console". The VMConsole is going to open on a tab on your browser.

HOL-1986-02-EMT


1. Click inside the console windows to activate the keyboard.2. For the Username type root and for the password type VMware1!3. After you login please type "ip addr" in the command prompt, and then press

enter. This will show you the current network configuration o that VM.4. On the next command prompt type " ping -c 5 10.252.208.254" to prove the

VM is able to reach a physical workload.5. You close the browser tab by clicking on the red X. This will bring back to vCenter.

Power Off HOL-1986-02-SRIOV-00

We still have the HOL-1986-02-SRIOV-00 VM selected on the panel, so we are going topower the VM to get it ready for the next module.

HOL-1986-02-EMT


1. Click on action go to "Power" and select "Power off".2. On the next screen click on "Yes" .

Enable SR-IOV in settings

On our previous test the VM was using SR-IOV network adapter and on our next module,we are going to migrate it from SR-IOV to N-VDS. There is no live migration from SR-IOVto N-VDS. However, you can do a cold migration, and the process is straightforward. Onthis module, I’m going to walk you through the process step by step.

Since vMotion is not supported with SR-IOV, you have to remove the SR-IOV networkadapter from the VM before you start the migration.

See steps below on migration:

HOL-1986-02-EMT


1. Click on "Action" and select "Edit Settings".2. Remove the “SR-IOV network adapter 1” by clicking on the X.3. Click on “Ok” to save the setting. Now the VM is ready for migration.

Migrate SRIOV-00

In the next step we will migrate VM name: HOL-1986-02-SRIOV-00 from host w3-nfv-compute20 on Compute SR-IOV cluster to host w3-nfv-compute021 on the nVDS-Cluster.

HOL-1986-02-EMT


1. Click on "Action" and then select "Migrate..." to start the migration wizard.2. Select "Change both compute resource and storage" and click on "Next".3. Click on "HOL-1986-02-DC" to expand it and select "Compute-nVDS-Cluster".4. Leave the default setting and click on "NEXT" three time and then click on

"FINISH" to complete the vMotion.

Add Network Adapter

Before we power on the VM we need to add a new vmxnet3 network adapter, andconnect that adapter to the N-VDS backed logical switch.

HOL-1986-02-EMT


1. Click on "ACTION" and select "Edit Settings..".2. Click on "ADD NEW DEVICE" button and select "Network Adapter".

HOL-1986-02-EMT


1. On the "New Network" click on the drop down arrow next to"10.252.104.on22-vLan-108" and select "Browse...".

2. Select "nvds-Trunk" network and click on "OK".3. Click on “Ok” to save the setting.

Power On SRIOV-00 VM

We powered the VM off to migrate it to the Compute-nVDS-Cluster and now we need topower backup to test the network.

HOL-1986-02-EMT


• Click on ACTIONS >> Power >> Power ON.

Launch Console and login

Once the VM is powered on you will need to login to check the network connectivity.

HOL-1986-02-EMT


1. Click on "Launch Remote Console" .2. The VM Console will open on a new tab and click on the new tab.3. For the Username type root and for the password type VMware1!4. After you login type: ip addr in the command prompt and press enter. Review the

VM assigned IP address.

Ping

Now the VM is completely switched to the N-VDS network and now we need to makesure we have IP connectivity for that VM.

HOL-1986-02-EMT


• In the command prompt type "ping -c 5 10.252.208.254" and you will get replyfrom the physical workload.

This will complete migration of a VM from SR-IOV backed network to N-VDS backednetwork. Now that you are using n-VDS you will have the full virtualization feature onthis VM and as you can see the process was very simple.

What is NUMA?

Non-Uniform Memory Access (NUMA) is a computer memory design used inmultiprocessing, where the memory access time depends on the memory locationrelative to the processor. Under NUMA, a processor can access its own local memoryfaster than non-local. To get the best performance from VM network we recommendaligning your NUMA node and fine-tuning your VM advance settings. On this module,we are going to show you how to check your NIC NUMA, and adjust the VM settings.

When using vCloud NFV platform for virtualizing data intensive workloads that needhigh throughput. You’ll need to fine-tune your VM setting, by aligning the NUMA nodesand by changing some advanced configuration of the virtual machine. In this module, Iwill show you how to do that.

HOL-1986-02-EMT


How to find out which NUMA your Network Adapter card ispart of

The VM NUMA needs to aligned to the host physical network adapter NUMA Node.

Using putty, you can SSH to the host the VM is running on.

In command prompt type: vsish -e get /net/pNics/<vmnicX>/properties | grep NUMA.

You can see in this lab we are using vmnic4 and vmnic5, both of these NIC's are usingNUMA Node 0.

Set VM NUMA affinity

Now that we know the pNIC NUMA Node, we need to make sure the VM NUMA affinity is0, so it is the same as the physical NIC NUMA.

HOL-1986-02-EMT


To check the VM NUMA affinity, right click on the VM select “Edit Settings”.

Click on “VM Option” tab.

Click on “Advanced” to see the additional option.

HOL-1986-02-EMT


Click on “Edit Configuration”.

HOL-1986-02-EMT


Search for “aff” to see if the NUMA is configured for the VM.

If the result is empty you need to add it In the name box type “numa.nodeAffinity” andValue box type “ 0” then click on add.

HOL-1986-02-EMT


Click on “Ok” twice to save the setting.

Latency Sensitivity

The VM option called "Latency Sensitivity" will default to "Normal" setting this to Highcan yield significantly lower latencies and jitters, because of the following mechanismsthat take effect in ESXi.

To change the VM Latency settings you need to click on the "VM Options" tab.

HOL-1986-02-EMT


Find the "Latency Sensitivity" drop down on the bottom left corner, and select "High".

Click on "Ok" to save the setting.

HOL-1986-02-EMT


Enable multiple cores for high I/O workloads

• By default, ESXi uses one transmit thread regardless of the number of vNICsassociated with a VM. To achieve significant parallelism for high I/O workloadswhere the transmit thread becomes CPU bottle necked, you can configure a VM touse one transmit thread per vNIC, by adding the following setting to the VM’sconfiguration: ethernetX.ctxPerDev = "1"

• In the search box you can type "ctx" to see if the VM has already beenconfigured. If the search result is blank then you can add the string. Please note:this needs to be added individually for each vNic you have on that VM.

HOL-1986-02-EMT


Now you will confirm the string has been added correctly, by entering "ctx" in the searchbox. There should be a result reflected for each individual vNic.

HOL-1986-02-EMT


Set CPU Reservation

The CPU reservations should be also set to match the CPU type in the VM. For example,if the host has 2.4 GHz processors and the Virtual machine have 4 assigned. Thereservation would be 4x2.4 = 9.6 GHz, so the reservation should be set to 9600 MHz.The memory reservation should also be set to "Unlimited" to reserve all memory for

that VM and avoid any memory contention.

HOL-1986-02-EMT


ConclusionIn this module you learned about:

• Configuring and enabling a VNF using SR-IOV• The value of N-VDS• Configuring N-VDS for Data plane intensive workloads• Migration of VNF from SR-IOV to N-VDS• The value of NUMA and configuring for latency sensitivity

Congratulations! You've finished Module 1

In this lab, you learned what SR-IOV is, what N-VDS is, and how to migrate an existingworkload from one to the other!

If you are looking for additional information on NFV, try one of these:

• http://www.vmware.com/go/nfv• https://blogs.vmware.com/telco/• https://www.telecomtv.com/content/vmware-channel/• https://www.linkedin.com/showcase/vmware-nfv/

Then Proceed to any module below which interests you most.



How to End Lab

To end your lab click on the END button.

HOL-1986-02-EMT


http://www.vmware.com/go/nfv

https://blogs.vmware.com/telco/

https://www.telecomtv.com/content/vmware-channel/

https://www.linkedin.com/showcase/vmware-nfv/

Module 2- Saturating thelink and performance

scalability (30 minutes)

HOL-1986-02-EMT


IntroductionN-VDS (E) The enhanced mode is suggested for data plane intensive workloads

Some high transactional control plane VNFs

Northbound connectivity is VLAN backed

Edge cluster and services such as LB, FW, NAT are not required fromNSX-T

N-VDS (S) The standard mode is suitable for control and management planeworkloads

Requires Overlay and VLAN backed connectivity

Requires stateful and stateless edge services such as LB, FW, NAT

Hosts Preparation for Data-Intensive Workloads

Before a data-intensive workload can attach to an N-VDS (E) switch, the hosts needspecific a configuration.

NICs intended to be used by N-VDS (E) should be connected in equal amounts andbalanced on all NUMA nodes

N-VDS (E) switch on each host with dedicated physical NIC is created

The same number of cores on each NUMA node should be assigned to the N-VDS (E)switch

Install Enhance network Poll Mode driver for N-VDS (E) physical NICs which can bedownloaded from https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI67-INTEL-I40EN-ENS-1.0.4&productId=742

The host N-VDS (E) switch is setup similarly to the topology below. Vertical alignmentbetween the VNFC, N-VDS (E) and NIC CPU on same NUMA is established to ensureperformance. The CPU scheduler will take care of ensuring that VNFC vCPU map tophysical CPU cores on the same NUMA where N-VDS (E) physical cores are mapped.

In the following example, we will show you how to use Cisco's TREX, l3frwd and customscripts to evaluate the speed of your hardware. Your configuration of network card, nic

HOL-1986-02-EMT


https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI67-INTEL-I40EN-ENS-1.0.4&productId=742

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI67-INTEL-I40EN-ENS-1.0.4&productId=742

and cpu core allocations will impact your test. This test is an example, with furthertuning, we could move the packets per second even higher!

HOL-1986-02-EMT


Hands-on Labs InteractiveSimulation:Testing Performance of n-VDS NICThis part of the lab is presented as a Hands-on Labs Interactive Simulation. This willallow you to experience steps which are too time-consuming or resource intensive to dolive in the lab environment. In this simulation, you can use the software interface as ifyou are interacting with a live environment.




HOL-1986-02-EMT


http://docs.hol.vmware.com/hol-isim/HOL-2019/hol-1986-02-saturatelink.htm

Manual Testing Performance of n-VDSNICFor this iSim, we will be showing how to setup the environment to test the throughput ofthe N-VDS switch. Many factors go into the speed and throughput results. For example,CPU speed can play a large role in the attained results.

Another factor, Virtual Machine configuration, needs to be verified that it has been setupfor high throughput and low latency.

Let's first log into the vSphere Web Client and take a look at the Virtual MachineSettings:

HOL-1986-02-EMT


Then select (HTML5) for user interface to log into the vSphere Web Client

The username and password are already pre-entered. Please click on login.

We will be working with the VM HOL-1986-02_nVDS-Module4. this is an Ubuntu VM withDPDK libraries installed and will be running l3forwarder in a later step.

Lets select the Virtual machine we will be checking and modifying the settings to get itprepared for the test. Click on the HOL-1986-02_N-VDS-Module4

HOL-1986-02-EMT


Next, expand the VM hardware section. Then expand CPU. After expanding, we cansee that The reservations have been set and the shares have been set to high.

In this test. Adapters 2 through 5 are used. Network Adapter 1 is used formanagement of the VM and allows us the ssh (secure shell) into the VM to start the test

HOL-1986-02-EMT


Next we will click on the connected check box to enable the network adapters and bringthem into a connected state.

HOL-1986-02-EMT


We need to set additional options, please click on "VM Options"

Next we will need to set the latency sensitivity is set to high. With this the CPUreservations should be also be set to match the CPU type in the VM. For example, if thishost has 2.4Ghz processers, and this Virtual machine has 4 assigned. The reservationwould be 4x2.4 (9.6 Ghz)

HOL-1986-02-EMT


Please click on Advanced and change the Latency from normal to high

on the same screen, click on Edit Configuration (right above the Latency Sensitivity)

then click on the picture of the funnel to filter the results. We will look at the NUMAconfiguration and ctx parameters for the Ethernet interfaces. type in numa in the filter.

HOL-1986-02-EMT


In our test, we want to make sure that the Virtual Machine is using the Intel Networkcard in NUMA 0 of the computer architecture. numa.nodeAffinity set to "0" makes surethat the Virtual Machine will stay aligned.

HOL-1986-02-EMT


HOL-1986-02-EMT


By default ESXi uses one transmit thread regardless of the number of vNICs associatedwith a VM. In order to achieve significant parallelism for high I/O workloads where thetransmit thread becomes CPU bottle necked, you can configure a VM to use onetransmit thread per vNIC, by adding the following setting to the VM’s configuration

In this test, we have 4 NICs for the test. The value for ethernet0.ctxPerDev is left offbecause it is for management only.

In the filter we will type ctx to verify that they are set.

If one was not set, we would set the value to ethernetx.ctxPerDev (where X is theethernet number)

Verifying N-VDS on esxi

First lets log into the ESXi host and check the settings to make sure all is setup for thetest.

HOL-1986-02-EMT


Please click on on the nVDS-Host icon to log into the host where we will be running thetest.

SecureCRT is setup to auto log you in. On the Console please type:

esxcfg-module -g i40en_ens <Enter>

This will check to see if the N-VDS driver is installed. If it returns 1, this means it isinstalled.

then type:

clear <enter>

HOL-1986-02-EMT


to clear the screen for the next section.

In the next section, we will check to mkae sure N-VDS is enabled and the correct driver(i40en_ens) is enabled on vnic4

please type the following command to check!

esxcfg-nics -e | grep 710

HOL-1986-02-EMT


Lets now look at the numa alignment of the Intel X710 Network card. for this test, weare making sure all is aligned to NUMA 0, which is the same as the DPDK Virtualmachine

please type:

vsish -e cat /net/pNics/vmnic4/properties | grep NUMA <ENTER>

HOL-1986-02-EMT


note the "grep" is to shorten up the output. With a full listing, we would learn driverversion, mac addresses, speeds etc

To verify that the allocated lcores (vCPUs) are on the correct NUMA boundry.

please type:

esxcli network ens lcore list <enter>

From this output the Affinity is set to "0", the same as the numa.nodeAffinity attributewe set on the Virtual Machine earlier

then type clear<enter>

HOL-1986-02-EMT


Next let's take a look and see how many lcores are allocated to the DPDK Virtal Machinewe have setup for the test.

Please type in:

nsxdp-cli ens port list <enter>

HOL-1986-02-EMT


Wait a minute! Where is our Virtual Machine? We don't see any lcores (or the VM!) .Lets go back to the main window and log into VMware vSphere and take a look!

please click on the secureCRT icon to minimize

then select VMware vSphere

HOL-1986-02-EMT


Starting up the test

Now we will go through the steps to start up the test and where in your environment tolook for what your throughput is at any time.

The esxtop command followed by "N" will show the networking. This command willshow you your speed and packets per second. this command is the same for NVDS andnon-nVDS based switches.

HOL-1986-02-EMT


ConclusionThank you for going through this Hands On Lab!

In this lab, you learned the methods of how N-VDS allocated resources at the kernellevel and how to identify them. You also were able to see that Virtual Machines can bevMotioned from one host to another without having to be shut down and moved, thenhave the resources reallocated!

VMware vCloud NFV 3.0 (4:18)

Demonstration of VMware vCloud NFV 3.0 a fully integrated, modular, multi-tenantNetwork Functions Virtualization (NFV) platform. Leveraging NSX-Managed VirtualDistributed Switch performance optimized on servers powered by Intel Xeon ScalableProcessors.

Congratulations! You've finished Module 2 and havereached the end of the Lab!

If you are looking for additional information on NFV, try one of these:

• http://www.vmware.com/go/nfv• https://blogs.vmware.com/telco/• https://www.telecomtv.com/content/vmware-channel/• https://www.linkedin.com/showcase/vmware-nfv/

Then Proceed to any module below which interests you most.

<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><ahref="http://www.youtube.com/watch?v=9L0XeI4mEkQ" target="_blank">Try watching this video on www.youtube.com</a>, or enableJavaScript if it is disabled in your browser.</div></div>

HOL-1986-02-EMT


http://www.vmware.com/go/nfv

https://blogs.vmware.com/telco/

https://www.telecomtv.com/content/vmware-channel/

https://www.linkedin.com/showcase/vmware-nfv/



How to End Lab

To end your lab click on the END button.

HOL-1986-02-EMT


ConclusionThank you for participating in the VMware Hands-on Labs. Be sure to visithttp://hol.vmware.com/ to continue your lab experience online.

Lab SKU: HOL-1986-02-EMT

Version: 20200210-210244

HOL-1986-02-EMT


http://hol.vmware.com/

Date post:	17-Jun-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Table of Contents - VMware · N-VDS DPDK Data Plane Development Kit (DPDK), is a set of libraries...

Documents