White Paper
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 1 of 17
Virtualized SAP HANA on FlashStack Converged Infrastructure
Technical Considerations and Best Practices
August 2017
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 2 of 17
Contents Introduction
Purpose of this document Audience Document reference Test environment
Reference architecture Storage layer Server and network layers Virtual machine, OS and application layers
Sizing Compute Storage
For more information Certified SAP HANA Hardware Directory SAP HANA TDI documentation SAP Notes Other links
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 3 of 17
Introduction
Support for running SAP HANA virtualized on VMware vSphere has been available since the release of SAP HANA
Support Package Stack (SPS) 05. Support has been available starting with VMware vSphere 5.1 for nonproduction
scenarios. Most recently, support has been available with SAP HANA SPS 12 or later for production single–virtual
machine and multiple–virtual machine use cases with up to 4 terabytes (TB) of main memory. Customers now can
gain the benefits of virtualization for their SAP HANA environments and also achieve cost optimization through
more efficient resource utilization.
Both SAP HANA appliance and SAP HANA Tailored Datacenter Integration (TDI) delivery methods are supported
for SAP HANA deployments on VMware vSphere. Virtualized SAP HANA systems, whether delivered preinstalled
and configured by a certified vendor or implemented by the customer using existing hardware listed in the Certified
and Supported SAP HANA Hardware Directory, should be verified with the SAP HANA Hardware Configuration
Check Tool (HWCCT) to qualify for support from SAP. The main use of SAP HANA on VMware is within the TDI
framework. Increasing numbers of customers are taking advantage of the extended flexibility that comes with
running SAP HANA TDI.
IDC confirmed in a recent survey (IDC US42198216) that Cisco is the leading vendor for SAP HANA infrastructure.
In line with the current trend toward converged infrastructure to further simplify and support multitenant workload
deployments, Cisco and Pure Storage have partnered to deliver FlashStack converged infrastructure as a platform
for a variety of SAP workloads, including fully virtualized workloads. FlashStack uses best-in-class server and
network components integrated with the programmability of the Cisco Unified Computing System™
(Cisco UCS®)
backed by high-performance all-flash storage from Pure Storage. With Cisco UCS servers and Pure Storage
FlashArray//m series listed in the Certified and Supported SAP HANA Hardware Directory’s Certified Appliances
and Certified Enterprise Storage sections respectively, FlashStack provides an optimized platform for
implementations of SAP HANA TDI.
Purpose of this document
This document provides configuration guidelines for implementing virtualized SAP HANA on FlashStack converged
infrastructure. It describes the best practices to be followed for the storage, server, network, virtualization, OS, and
application layers for an optimal configuration.
Audience
This document is intended for IT architects, engineers, and administrators who are responsible for configuring and
deploying the SAP HANA platform in a VMware virtualization environment. It assumes that the reader has a basic
knowledge of FlashStack, Cisco UCS, VMware vSphere, Linux, and SAP HANA TDI scenarios.
Document reference
Please refer the Cisco® Validated Design document FlashStack for SAP HANA TDI published in February 2017 for
deeper understanding of FlashStack converged infrastructure design principles and configuration. This document
builds on this design and addresses the virtualization scenario, highlighting general best practices.
The best practices documented here are in line with VMware’s Best Practices Guide for SAP HANA and serves as
a reference for implementation on FlashStack converged infrastructure.
Test environment
Table 1 lists the hardware and software versions used in the test environment described in this document.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 4 of 17
Table 1. Test environment details
Layer Device Image
Compute Cisco UCS 6332-16UP Fabric Interconnect pair Release 3.1(2b)
Cisco UCS C460 M4 Server (3 nodes) Release 3.1(2b)
Cisco UCS Virtual Interface Card (VIC) 1385 (1per C460 M4 node) Release 3.1(2b)
Network Cisco Nexus® 9372PX-E Switch pair Release 7.0(3)I2(4)
Storage Cisco MDS 9148S 16G Multilayer Fabric Switch pair Release 7.3(0)DY(1)
Pure Storage FlashArray //m50 R2 Release 4.8.3
Software Cisco UCS Manager Release 3.1(2b)
VMware vSphere ESXi Release 6.5.0
VMware vCenter Release 6.5.0
Reference architecture
Figure 1 shows the various layers of architecture, from the storage layer to the application layer. It shows the
possible constituents of each layer for FlashStack converged infrastructure.
FlashArray //m series arrays make up the storage layer. Cisco MDS 9100 Series Multilayer Fabric Switches and
Cisco Nexus 9000 Series Switches at the network layer cater to Fibre Channel and Ethernet traffic respectively.
Cisco UCS C-Series Rack Servers or B-Series Blade Servers provide the compute. SAP HANA is installed in
virtual machines running either the SUSE Linux Enterprise Server (SLES) or Red Hat Enterprise Linux (RHEL) OS
hosted on the VMware ESXi virtualization platform.
Figure 1. Architecture overview: Layered representation
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 5 of 17
Virtualized SAP HANA can be deployed in both scale-up and scale-out system configurations.
Storage layer
The Pure Storage Purity Operating Environment automatically handles the tuning, encryption, solid-state disk
(SSD) wear leveling, and data resiliency (among other operations) at the array level. You only need to name the
volume (the target for the datastore), enter the size, and attach the volume to the appropriate ESXi host groups.
An important benefit of Pure Storage FlashArray//m is that it enables customers to design their SAP HANA
environment datastores with a logical approach rather than constraining them to the limitations of virtual machines
per datastore. However, it is a good practice to segregate the instance’s OS boot disks and SAP HANA disks
(data, log, and hanashared) in separate datastores, at a minimum. For example, Figure 2 shows a virtualized SAP
HANA scale-up system cisvhana1 with 512 GB of RAM configured with a 100-GB OS boot disk (including a
/usr/sap partition) along with the configuration file in one datastore (VMboot-Datastore). It also includes disks for
/hana/data (768 GB), /hana/log (256 GB), and /hana/shared (512 GB) in another datastore (vHANA-Datastore).
Figure 2. Datastore selection and disk formatting
A single datastore hosting SAP HANA disks (data, log, and shared) would also suffice to support multiple
virtualized SAP HANA instances for most use cases. Configuring separate datastores for data (requiring higher
read performance) and logs (requiring higher write performance), though desirable, may not be necessary because
the back-end all-flash array provides a uniform performance-class I/O subsystem, managed by the Purity
Operating Environment.
Server and network layers
This section describes best practices for the server and network layers.
Multipath configuration
VMware vSphere ESXi provides the platform for the virtualized SAP HANA system virtual machines. ESXi uses its
Native Multipathing Plugin (NMP) architecture to manage I/O multipathing to underlying SAN storage volumes.
FlashArray has an active-active front-end controller architecture. FlashArray volumes are preferably claimed, by
default, by the Storage Array Type Plugin (SATP) for Asymmetric Logical Unit Access (ALUA) devices and use
round-robin (RR) path selection policy (PSP). That is, they use VMW_PSP_RR with an I/O operations limit of 1 in
order to alternate logical paths after every I/O operation. Refer to VMware Docs for more information.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 6 of 17
If this setup is not available in a particular ESXi release, the same result can be achieved by running the following
command on each ESXi host prior to the presentation of FlashArray devices:
#esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -V "PURE" -M "FlashArray" – P
"VMW_PSP_RR" -O "iops=1"
Power management
A Cisco UCS service profile defines a single server and its storage and networking characteristics. It is a logical
representation of a server that encapsulates identity, configuration, and connectivity information. Service profiles
also include operational policy information such as BIOS parameters, boot paths, and boot order.
SAP HANA workloads are sensitive to latency. Because achieving the lowest possible latency and saving power on
the hosts are mutually exclusive, the recommended approach is to disable any form of power management to aid
applications when performance needs are more important. In this case, you should disable the C-state and
enhanced C-state (C1E) power management options.
Additionally, set the ESXi host’s hardware power management to the high-performance mode (Figure 3).
Figure 3. VMware ESXi host Power Management settings
Figure 4 shows optimum BIOS parameter settings for best performance in a virtualization scenario.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 7 of 17
Figure 4. Cisco UCS service profile BIOS policy Processor settings
Hyperthreading and non-uniform memory access management
Hyperthreading and non-uniform memory access (NUMA) management are especially important for configurations
with multiple virtual machines. SAP HANA workloads are sensitive to memory latency with relatively lower
processor utilization. For such workloads, the recommended approach is to use hyperthreading with fewer NUMA
nodes. You also should size the virtual machines to stay within a NUMA node or within the least number of NUMA
nodes, as best as possible, to optimize performance.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 8 of 17
To achieve this setup, implement the configuration parameter numa.PreferHT=1 on a per-host basis (Figure 5).
Alternatively, you can implement the parameter numa.vCPU.preferHT=TRUE on each virtual machine. Refer to
VMware KB 2003582 for more details.
Figure 5. VMware ESXi host: Advanced system settings for numa.PreferHT
Fibre Channel connectivity
The FlashArray//m series are block-only arrays. The ESXi hosts should be configured to boot from SAN. A
redundant pair of virtual host bus adapters (vHBAs)—one each for SAN fabric A and SAN fabric B defined in the
service profile—serve as the initiators (Figure 6). A 10-GB volume from the array can be presented to the host and
used as its ESXi boot disk (Figure 7).
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 9 of 17
Figure 6. Cisco UCS service profile template: Storage > vHBA settings
Figure 7. Pure Storage GUI: Properties for ESXi host boot from SAN disk
Network considerations
SAP categorizes the HANA system networks into three logical communication zones (Figure 8):
● Client zone: External communication networks (for example, for application server and client network
connectivity)
● Internal zone: Internal communication networks (for example, for internode communication in a scale-out
system and replication traffic in a distributed configuration)
● Storage zone: All storage-related networks (for example, for the Network File System (NFS)–based
/hana/shared network and backup network and for NFS-based data and log file system access when NFS is
used)
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 10 of 17
Figure 8. SAP HANA TDI network requirements (source: SAP SE)
Table 2 summarizes the various use cases that these networks serve and their relevance to a virtualized SAP
HANA solution on FlashStack.
Table 2. SAP HANA TDI and FlashStack network requirements mappings
Name Use case FlashStack virtualized SAP HANA solution relevance
Minimum bandwidth requirements
Client zone
Application server network Communication between SAP application server and database
Required 1 or 10 Gigabit Ethernet
Client network Communication between user or client application and database
Required 1 or 10 Gigabit Ethernet
Internal zone
Internode network Node-to-node communication within a scale-out configuration
Required for scale-out configurations only
10 Gigabit Ethernet
System replication network SAP HANA system replication Optional To be determined with customer
Storage zone
Backup network Data backup Optional 10 Gigabit Ethernet or 8-Gbps Fibre Channel
Storage network Communication between nodes and storage
● Optional for scale-up systems
● Required for NFS-based /hana/shared file system access in scale-out scenario
10 Gigabit Ethernet
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 11 of 17
Infrastructure-related networks
Administration and VMware High Availability (HA) network
SAP HANA system administration and VMware HA management
Required 1 Gigabit Ethernet
VMware vMotion network Resource utilization balancing for a vSphere cluster
Required when using multiple ESXi hosts in a VMware Distributed Resource Scheduler (DRS)–enabled cluster
10 Gigabit Ethernet
To determine the number of virtual network information cards (vNICs) to define when creating the service profiles
to support both bandwidth and failover requirements, map the use-case information using the matrix in Table 2.
Note the following:
● Traffic management for the various networks can be performed at the virtual machine level using VLANs
while ensuring consistency with the definitions on the data center uplink switch.
● For network bandwidth management when using a single virtual switch (vSwitch) to consolidate all network
traffic or when using a separate vSwitch for each zone to segregate traffic based on the required bandwidth,
be sure that the optimum number of load-balanced uplinks are assigned.
● For the internal zone, storage zone, and vMotion network, the recommended approach is to configure
jumbo frames end-to-end.
Virtual machine, OS and application layers
This section presents best practices for the virtual machine, OS, and application layers.
Virtual machine configuration
The recommended approach is to configure the cores per socket to match the actual hardware configuration. For
example, configure 22 cores per socket for Intel® Xeon
® processor E7-8880 v4, 24 cores per socket for Intel Xeon
processor E7-8890 v4, and 28 cores per socket for Intel Xeon Platinum 8180 and 8176 processors (Figure 9).
Figure 9. Setting cores per socket
Be sure that the vCPU hot plug is not selected because it disables the virtual NUMA (vNUMA) optimization feature
(Figure 10).
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 12 of 17
Figure 10. CPU Hot Plug setting
The recommended approach is to reserve CPU and especially the memory resources for production virtual
machines. Select the “Reserve all guest memory (All locked)” option (Figure 11).
Figure 11. Virtual machine memory reservation setting
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 13 of 17
Use paravirtual adapter VMXNET 3 type vNICs to configure the network interfaces for all the networks defined
(Figure 12).
Figure 12. vNICs adapter type selection
Use a dedicated Small Computer System Interface (SCSI) controller of type VMware Paravirtual for each data, log,
and hanashared disk to implement parallel I/O processing by separating the disk I/O streams. Also use eager-
zeroed thick provisioning for SAP HANA data and log disks (Figure 13).
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 14 of 17
Figure 13. SCSI controller type and disk provisioning type selection
Operating system
To configure optimal settings for running SAP HANA on SLES or RHEL, follow the respective SAP notes that
provide the SAP HANA database recommended OS settings guidelines as you would do for bare-metal
implementations. See the “For more information” section of this document for all relevant SAP Notes.
ESXi has its own I/O scheduling mechanisms that not only makes guest OS IO scheduling redundant but also adds
to unnecessary overhead. The recommended approach is to disable I/O scheduling at the virtual machine level.
Append elevator=noop to the kernel boot arguments.
Application layer
Both scale-up and scale-out configurations are possible in the virtualized environment, as they are in bare-metal
scenarios. Note the following points when implementing these configurations on FlashStack:
● In a scale-up system configuration, /hana/shared can also be a virtual machine disk (VMDK) created from
the same datastore as the one that hosts the data and log disks of the SAP HANA instance because it is
local to that instance. It does not necessarily be a NFS mount.
● In a scale-out configuration, /hana/shared must be accessible by all nodes of the scale-out cluster
concurrently, and at all times needing it to be a NFS mount. FlashArray //m series arrays are block-only
arrays. In a bare-metal scenario, this setup is achieved by configuring the scale-out nodes’ OS in a high-
availability cluster with /hana/shared configured as an Oracle Cluster File System (OCFS2) cluster file
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 15 of 17
system resource. (Refer to the Cisco Validated Design for FlashStack for SAP HANA TDI for more
information.) However, a VMware-supported scale-out solution requires the SAP HANA shared file system
to be NFS only. Other solutions—such as OCFS and IBM General Parallel File System (GPFS)—are not
supported by VMware. To address this requirement, you can configure a dedicated virtual machine as an
NFS server and export a locally mounted disk as /hana/shared to the cluster nodes. The high availability of
the NFS server virtual machine can be ensured by activating the vSphere HA and DRS features and, in
extreme cases, enabling VMware Fault Tolerance (FT), taking into account proper sizing considerations.
Sizing
This section provides sizing inputs in terms of the virtual machine’s memory to virtual CPU (vCPU) ratio and the
number of virtualized SAP HANA instances and nodes supported by FlashArray //m50 R2.
Compute
The CPU generation platform and SAP’s guidelines regarding the memory range supported for a particular socket
count form the basis for sizing virtualized SAP HANA instances. The vCPU and virtual RAM (vRAM) maximums for
a specific vSphere ESXi version also influence the sizing. Additionally, some memory overhead should be
considered because ESXi needs some for its own kernel and for hosting virtual machine memory pages. Refer to
VMware Docs for more information.
With Intel Xeon Platinum platform, SAP allows memory configurations of 128 GB to 3 TB on four-way server
systems for SAP Business Warehouse on HANA (BWoH). It allows up to 6 TB for SAP Suite on HANA (SoH)
workloads. The matrix in Table 3 provides a reference for technically feasible virtual machine configurations with
the current Intel Xeon Platinum platform (supported only on Cisco UCS M5 servers) and previous Intel Xeon
processor generations.
Table 3. Virtual machine sizing: vCPU and vRAM with processor platform matrix
vCPU (with hyperthreading on) vRAM
Intel Xeon E7 v4 8880 Intel Xeon E7 v4 8890 Intel Xeon Platinum 8180 and 8176
BWoH SoH BWoH SoH BWoH SoH
≤SPS10 ≥SPS11 ≤SPS10 ≥SPS11 ≤SPS11 ≥SPS12
22 22 22 24 24 24 24 28 28 32
22 22 22 24 24 24 24 28 28 64
22 22 22 24 24 24 24 28 28 128
22 22 22 24 24 24 24 28 28 256
44 44 22 48 48 24 24 28 28 384
66 44 44 72 48 48 24 28 28 512
88 66 44 96 72 48 48 56 28 768
110 88 66 120 96 72 48 84 28 1024
110 88 120 96 72 112 56 1536
110 120 96 84 2048
120 112 3072
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 16 of 17
Storage
In the test environment with FlashArray //m50 R2, tests were run with a single datastore for SAP HANA data and
log files to ascertain the number of virtualized SAP HANA instances that the array could support, satisfying the
SAP HWCCT tool TDI key performance indicators (KPIs).
Table 4 summarizes the results when tests were run in the lab environment with one datastore for both HANA
persistence partition constituents. The recommended approach is to perform case-specific HWCCT-based tests to
verify that the system meets the TDI KPIs.
Table 4. Array sizing test results with fsperf parameters used
FlashArray //m series array Number of datastores for SAP HANA data and log disks
Number of virtualized SAP HANA instances supported
//m50 R2 01: One datastore hosting both data and log disks of all the virtualized SAP HANA instances.
Up to 12
For more information about the HWCCT based fsperf parameters, refer to SAP Note 1943937.
© 2017 Cisco and/or its affiliates. All rights reserved. This document is Cisco Public. Page 17 of 17
For more information
Certified SAP HANA Hardware Directory
● Certified SAP HANA Hardware Directory: Enterprise Storage
SAP HANA TDI documentation
● SAP HANA TDI: Overview
● SAP HANA TDI: FAQ
● SAP HANA TDI: Storage Requirements
● SAP HANA TDI: Network Requirements
SAP Notes
● SAP Note 2161991: VMware vSphere configuration guidelines
● SAP Note 1788665: SAP HANA Support for virtualized / partitioned (multi-tenant) environments
● SAP Note 2393917: SAP HANA on VMware vSphere 6.5 in production
● SAP Note 1943937: Hardware Configuration Check Tool - Central Note
● SAP Note 2235581: SAP HANA: Supported Operating Systems
Other links
● SAP Community Network: SAP HANA TDI on Cisco UCS and VMware vSphere
● SAP Document Archive link: https://archive.sap.com/documents/docs/DOC-60312
● Cisco UCS: Design Zone for SAP Applications (technical documentation)
● Cisco UCS: Data Center Solutions for SAP (customer references)
● FlashStack converged infrastructure: FlashStack Virtual Server Infrastructure Design Guide for VMware
vSphere 6.0 U2
● FlashStack converged infrastructure: Cisco Validated Design for FlashStack VDI use case
● VMware blog: SAP on VMware updates
Printed in USA C11-739549-00 08/17