WHITE PAPER

SPLUNK ENTERPRISE AND VMAX ALL FLASH – PERFORMANCE ASSESSMENT TESTS AND BEST PRACTICES VMAX® Engineering White Paper

ABSTRACT This white paper provides details on the performance assessment tests and best practices for deploying Splunk Enterprise with Dell EMC VMAX All Flash storage array.

December 2016

2

Copyright © 2016 EMC Corporation. All rights reserved. Published in the USA.

Published December 2016

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, SnapVX, SRDF, Vblock, Data Domain, CloudArray, V-Brick, ProtectPoint, VMAX All Flash, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

EMC is now part of the Dell group of companies.

Splunk Enterprise and VMAX All Flash

Part Number H15604.1

3

TABLE OF CONTENTS

EXECUTIVE SUMMARY ...........................................................................................................4 AUDIENCE ........................................................................................................................................ 4

PRODUCT OVERVIEW .............................................................................................................5 VMAX Storage Array ......................................................................................................................... 5

Splunk Enterprise .............................................................................................................................. 6

Splunk core architecture ............................................................................................................................ 6

Splunk buckets and storage tiers ............................................................................................................... 6

Splunk clustering........................................................................................................................................ 7

DEPLOYMENT CONSIDERATIONS .........................................................................................8 Splunk “HA” and “DR” ....................................................................................................................... 8

Splunk and data reduction ................................................................................................................. 8

Splunk and LVM striping ................................................................................................................... 8

PERFORMANCE ASSESSMENT OF POWEREDGE R730 AND VMAX STORAGE FOR SPLUNK ENTERPRISE .............................................................................................................9

Overview ........................................................................................................................................... 9

Physical configuration ....................................................................................................................... 9

Logical configuration and host software .......................................................................................... 10

Virtualization best practices ............................................................................................................. 11

Indexer setup................................................................................................................................... 12

Storage setup using LVM striping ............................................................................................................ 12

Host limits ................................................................................................................................................ 13

Performance Assessment tests ....................................................................................................... 14

Test stages .............................................................................................................................................. 14

Test results ...................................................................................................................................... 15

Test summary .................................................................................................................................. 15

SUMMARY .............................................................................................................................. 16

REFERENCES ........................................................................................................................ 16 Dell EMC Documentation ................................................................................................................ 16

VMware Documentation .................................................................................................................. 16

Splunk Enterprise Documentation ................................................................................................... 16

4

EXECUTIVE SUMMARY Access to and analysis of machine data is one of the fastest growing and complex areas of big data. It is also one of the most valuable sources of data, containing a definitive record of events that can reveal information about user transactions, customer behavior, machine behavior, security threats, fraudulent activity, and more.

Making use of this data, however, presents real challenges for ingestion, storage, transformation, and analytics. Many traditional monitoring solutions designed to do both data management and analysis are not engineered to handle the high volume, high velocity, and highly diverse sources of machine data.

Splunk Enterprise is the industry-leading platform for machine data. It gives real-time insight and understanding into what is happening, and provides end-to-end visibility across your IT infrastructure to enable informed, data-driven decisions.

VMAX® All Flash offers a combination of ease of use, scalability, high performance, and a robust set of data services, making it an excellent storage choice for Splunk Enterprise deployments. The advantages of VMAX All Flash include performance optimization for raw data loads, indexing, dense (bandwidth-focused) and sparse (IOPS-focused) searches.

When considering Splunk deployments using Direct Attached Storage (DAS) vs. SAN-based storage arrays such as VMAX, there are a few key differentiators, such as storage availability, capacity management, avoiding imbalances in storage utilization (bottlenecks), non disruptive upgrade (NDU), data protection, security, and monitoring.

With VMAX All Flash, resources are shared, including flash drives, CPU cores, and storage cache. This prevents bottlenecks and increases resource utilization. SAN management is easy using Unisphere, CLI, or REST APIs, data is protected with T10-DIF, and can be encrypted with D@RE1. In addition, thin provisioning allows consumption of only storage that is actually being used. With SnapVX™ local snapshots can be created or restored in seconds (regardless of data capacity), and SRDF® provides consistent remote replications to any distance.

VMAX All-Flash storage and Dell PowerEdge R730 servers configured according to the best practices published in this guide are an excellent choice for customers looking for flexible, easy to use, and performant infrastructure to support a Splunk Enterprise deployment.

AUDIENCE This white paper is intended for system administrators, storage administrators, and system architects who are responsible for implementing Splunk Enterprise in environments with VMAX All Flash storage systems. Readers should have some familiarity with Splunk Enterprise and VMAX storage arrays.

1 D@RE refers to Data at Rest Encryption optional feature of VMAX which is utilized via ASIC and doesn’t incur any performance overhead for host I/Os.

5

Product Overview

VMAX STORAGE ARRAY The VMAX family of storage arrays is built on the strategy of simple, intelligent, modular storage. It incorporates a Dynamic Virtual Matrix interface that connects and shares resources across all VMAX engines, allowing the storage array to seamlessly grow from an entry-level configuration into the world’s largest storage array. It provides the highest levels of performance, scalability, and availability featuring advanced hardware and software capabilities.

The new VMAX All Flash family introduced in 2016 is designed to take advantage of the latest, most cost-efficient 3D NAND flash drive technology. It features multi-dimensional scale, large write-cache buffering, back-end write aggregation, high bandwidth, and low latency.

Figure 1 VMAX All Flash storage arrays 450F/850F (left), and 250F (right)

VMAX All Flash offers a combination of ease of use, scalability, high performance, and a robust set of data services that makes it an ideal choice for Splunk Enterprise deployments:

Ease of use: VMAX uses virtual provisioning to create new storage devices in seconds. All VMAX devices are thin, consuming only the storage capacity that is actually written to, which increases storage efficiency without compromising performance. VMAX devices are grouped into Storage Groups and managed as a unit, including device masking to hosts, performance monitoring, local and remote replications, compression, Host I/O limits, and more. In addition, VMAX management can be done using Unisphere for VMAX, Solutions Enabler CLI, or REST APIs.

High performance: VMAX All Flash is designed for high performance and low latency. It scales from one to eight engines (V-Bricks). Each engine consists of dual directors, each with 2-socket Intel CPUs, front-end and back-end connectivity, Infiniband internal fabric, and a large mirrored and persistent cache. All writes are acknowledged to the host as soon as they registered with VMAX cache2 and only later, perhaps after multiple updates, are written to flash. As a result, writes are extremely fast.

Reads also benefit from the VMAX large cache. When a read is requested for data that is not already in cache, it is sent directly from the back-end (flash) to the front-end (host) and only later staged in the cache for possible future access. VMAX also excels in servicing high bandwidth sequential workloads leveraging pre-fetch algorithms, optimized writes, and fast front-end and back-end interfaces.

Data services: VMAX All Flash natively protects all data with T10-DIF from the moment it enters the array until it leaves (including replications). With SnapVX™ and SRDF®, VMAX offers many topologies for consistent local and remote replications. VMAX offers optional Data at Rest Encryption (D@RE), integrations with Data Domain® such as ProtectPoint™, or cloud gateways with CloudArray®. Other data services, including Quality of Service (QoS)3, Compression, “Call-Home” support feature, none-disruptive upgrades (NDU), none-disruptive migrations (NDM), and more. In virtual environments VMAX also offers support for VAAI primitives such as write-same, xcopy, and others.

2 VMAX All Flash cache is large (from 512GB-16TB, based on configuration), mirrored, and persistent because of the vault module that protects the cache content in case of power failure, and restores it when the system comes back up. 3 Two separate features support VMAX QoS. The first relates to Host I/O limits that allow placing IOPS and/or bandwidth limits on “noisy neighbors” applications (set of devices) such as test/dev environments. The second relates to slowing down the copy rate for local or remote replications.

6

While outside the scope of this paper, VMAX can also be purchased as part of a Converged Infrastructure (CI) called Vblock™. More information on this can be found at: https://store.emc.com/us/Product-Family/VBlock-Products/Dell-EMC-Vblock-System-740/p/VCE-Vblock-System-740

SPLUNK ENTERPRISE Splunk Enterprise is a software platform that enables search, analysis, and visualization of the machine-generated data gathered from different sources in your IT infrastructure, including applications, networking devices, host and server logs, mobile devices, and more. It provides real-time insight, understanding, and end-to-end visibility across your IT infrastructure to enable informed, data-driven decisions.

SPLUNK CORE ARCHITECTURE

Figure 2 provides a graphic high-level overview of the Splunk system architecture. A Splunk Enterprise instance can perform the role of a forwarder to queue and transmit the collected data to the Splunk indexers. It can also perform the role of a search head, an indexer, or both in the case of small deployments. Once the daily ingest rate or search load exceeds the sizing recommendations for a combined instance environment, Splunk Enterprise scales horizontally by adding additional indexers and search heads. For more information, see: Splunk Capacity Planning Manual.

This white paper focuses on the indexing tier, where the bulk of the work is done, including parsing, transforming, indexing, and retrieving data when a search is issued from the search head.

Figure 2 Splunk architecture overview

SPLUNK BUCKETS AND STORAGE TIERS

When data is received by a Splunk Enterprise indexer, the indexer parses the raw data based on the timestamp of the events and writes them to index files. Splunk implements a form of storage tiering using hot/warm and cold buckets of data to optimize performance for newly indexed data and provide an option to keep older data for longer periods on higher capacity storage.

Newly indexed data lands in a hot bucket, where it is actively read and written by Splunk. When the number of hot buckets is reached, or when the size of the data in the hot buckets exceeds the specified threshold, the hot bucket is rolled to a warm bucket. Warm buckets reside on the same tier of storage as hot buckets. The only difference is that warm buckets are read-only. In DAS deployments this means that they can be backed up with traditional, file-based backup tools unlike hot buckets. In SAN deployments, storage consistent snapshots can include both hot and warm buckets. It is important that the storage identified for hot/warm data is your fastest storage tier because it has the biggest impact on the performance of your Splunk Enterprise deployment.

When the number of warm buckets or volume size is exceeded, data is rolled into a cold buckets, which often reside on another tier of storage. Although beyond the scope of this paper, Dell EMC has worked with Splunk on validating Dell EMC Isilon® as a storage platform for cold buckets. Isilon is an excellent choice for Splunk cold data, with its massive scalability and competitive cost of ownership. Finally, cold buckets can roll to frozen buckets for compliance or archival purposes. Frozen data cannot be searched directly

7

and must be manually “thawed” in order to be searched. For that reason, using a scale-out solution like Isilon to achieve a longer retention period is desirable to keep your valuable data searchable in Splunk for longer . Figure 3 shows the Splunk bucket concept and lifecycle.

Figure 3 Splunk buckets lifecycle

SPLUNK CLUSTERING

A full discussion about Splunk high-availability (HA) and disaster recovery (DR) is beyond the scope of this paper; however, some concepts must be understood in order to consider the right approach to scale, availability, and replications.

Indexer high availability is achieved by using Indexer clustering.

The Index cluster contains:

• Master node which manages the cluster activities

• Peer nodes, which are the Indexers that search, index, and maintain multiple copies of the data

• Search head, which coordinates searches across the peers

Splunk defines two critical parameters: a Replication Factor (RF) and a Search Factor (SF). The RF determines how many copies of the raw data the clustered Indexers should keep. As a result, the cluster can tolerate a failure of (RF-1) peer nodes. In other words, if the RF=3, 2-node failure can be tolerated without losing access to data. The SF determines how many searchable copies of the data the clustered Indexers should keep. A searchable copy of the data is a copy that was fully indexed by an Indexer. In other words, a copy of the raw data may be available based on RF, but may not be searchable due to smaller SF. In that case, although the data is available it can’t be searched until the index is built—an operation that may take some time.

To summarize, Splunk cluster provides high availability (HA) and data protection by adding Indexers and keeping multiple copies of the data based on the replication factor (RF). Note that when a peer node is placed in a remote site it provides also disaster recovery (DR).

For more information about Splunk clusters see: http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Aboutclusters

8

DEPLOYMENT CONSIDERATIONS

SPLUNK “HA” AND “DR” As discussed earlier, Splunk cluster provides high availability (HA) and remote replications (as part of disaster recovery, DR). Since Splunk hot/warm buckets are placed on high-performance storage, the higher the RF, the more storage capacity is consumed for the sake of availability. It is worth noting that based on how Splunk replicates data in a clustered indexer environment, the replicated data is rarely an identical copy of the original indexed data when looked at from a block or file perspective. For this reason, storage systems offering deduplication provide little storage savings benefit for Splunk Enterprise clustered indexer environments.

VMAX All Flash provides both increased storage availability (such as redundant components, hot disk sparing, online upgrades, and more) as well as high performance. When using Splunk Enterprise with VMAX storage customers can safely go with a replication and search factor of RF=2/SF=2 for clustered deployments rather than the default of RF=3/SF=2. Keeping an RF/SF of 2 is sufficient to make sure that a node failure will not interfere with the ability to execute searches, though higher RF/SF should be considered if the criticality of the applications demand it.

Normally, a Splunk cluster remote Indexer node can provide a disaster protection without downtime. Alternatively, SRDF can provide both Sync and Async replications at the storage level which are consistent across all the devices in the storage group, and therefore easy to manage as a unit. Traditional SRDF replicaions require a manul restart of the application at the remote site if the local site becomes unavailable. An alternative is SRDF/Metro, which keeps both local and remove site active/active and in sync. With SRDF/Metro the host can be connected via multipathing to both sites increasing its availability.

In addition, VMAX SnapVX can provide local (or remote, when combined with SRDF or a remote Splunk cluster Indexer node) copies of the data. SnapVX snapshots are pointer-based, take seconds to create, and only consume as much storage as the modified data on the source or target. Snapshots can be used as gold copies that can be restored in seconds, or accessed from other hosts for test, development, and other purposes that a copy of the Splunk data can provide.

SPLUNK AND DATA REDUCTION Splunk is very efficient in how it stores data and uses host-based compression to reduce the size of the raw data written to disk. For syslog-like data, the data written to disk is approximately half the size of the raw data. Splunk also creates an index which accounts for approximately 30% of the total data written to disk during the indexing process. The index file is left uncompressed and therefore is a source for additional storage savings for storage platforms like VMAX that provide compression at the storage layer. VMAX All Flash storage compression can be enabled or disabled at a storage group level (a group of devices).

VMAX also offers thin provisioning. With thin provisioning, as Splunk Indexers allocate hot/warm buckets, only the actual capacity that is utilized is consumed on the VMAX storage. As more capacity is needed VMAX can add flash drives to the Storage Resource Pool (SRP), without interrupting host workload. It should be noted, however, that although thin provisioning allows the host devices to be sized for future growth, the larger the devices, the more storage metadata (and therefore storage cache) they consume. Therefore, devices should be reasonably sized, and the storage system should be monitored to make sure cache and SRP capacities meet growth requirements.

VMAX All Flash storage compression can be disabled for Splunk hot/warm devices, because further compressing compressed data provides limited benefits.

SPLUNK AND LVM STRIPING Splunk hot/warm buckets use a filesystem structure. A filesystem is created on a single host LUN or a volume when host Logical Volume Manager (LVM) is used. The best way to allow for high I/O concurrency across multiple storage devices is to use host LVM striping.

Linux LVM uses a concept of a volume group (VG) that contains the VMAX All Flash devices. Then a logical volume (LV) is created and striped across the VG. Finally, a filesystem, such as XFS, can be created on the LV and mounted for Splunk Enterprise hot/warm buckets.

As demonstrated in the tests described in this paper, it is highly recommended to use a high performance file system, such as XFS, created on top of an LVM striped volume with multiple VMAX devices.

9

Note: The number of devices needed to achieve high I/O concurrency and performance using LVM striping is not fixed and depends on the actual I/O profile and needs. Each host device provides an I/O queue and the multipathing software spreads the queued I/Os across the available paths. In most cases, 8 VMAX devices per Indexer, and 4 host initiators with multipathing software can satisfy application high performance and high I/O concurrency of a 2-socket server. However, with larger SMP servers more devices and connectivity may be required. Host monitoring tools such as iostat can show host I/O queues and Unisphere for VMAX can help with monitoring storage performance to identify and alleviate unnecessary performance bottlenecks.

PERFORMANCE ASSESSMENT OF POWEREDGE R730 AND VMAX STORAGE FOR SPLUNK ENTERPRISE

OVERVIEW The hardware and software configuration described below was used to assess the performance of Splunk Enterprise on Dell PowerEdge R730 servers with VMAX All-Flash storage.

PHYSICAL CONFIGURATION The physical test environment is shown in Figure 4. It consisted of a single V-Brick™ VMAX 850FX, three Dell R730 servers (forwarders and indexers), one Cisco UCS C240M3 server (search head), network and SAN switches.

Each of the servers utilizes two dual-port host bus adapters (HBAs) for a total of four initiators per ESXi server connected to the SAN switches. The servers use two networks; a 1GbE public network for user connectivity and management, and a 10GbE private network for inter-node communication for the case of high-availability clusters, though in these tests Splunk HA cluster wasn’t used.

Figure 4 Test environment - physical configuration In this deployment, ESXi 6.0 hypervisor was installed on the four physical servers. Splunk Enterprise can be configured in physical or virtual environments. For these tests it was configured in a virtual environment to satisfy the workload requirement of 3 x Forwarders, 3 x Indexers (in 1:1 relationship), and a Search Head. In an actual deployment the choice of virtual or physical is left to the solution architect.

Table 1 describes the hardware components used for the performance assessment tests.

Table 1 Performance assessment tests hardware components Device Quantity Configuration Description

VMAX 850FX 1 • 1 V-Brick (32 x SSD in RAID5) VMAX All Flash single engine (V-Brick)

ESXi Hosts (Indexer/Forwarder) 3 • Dell R730

• 12 core x 2 Intel Xeon E5-2690 v3 @ 2.60 GHz

• 128 GB Memory

• 10GbE network ports

VMware ESXi host.

Each ESXi hosted two VMs – a Forwarder and an Indexer. See Table 2 for more details.

10

• 2 x dual port 16Gb HBAs

ESXi Host (Search Head) 1 • Cisco C240

• 10 core x2 Intel Xeon E5-2680 v2 @ 2.80 GHz

• 96 GB Memory

• 10GbE network ports

• 2 x dual port 16Gb HBAs

VMware ESXi host

The ESXi hosted a single VM – Search Head

LOGICAL CONFIGURATION AND HOST SOFTWARE The environment sizing was created using the Splunk recommended guidelines found here: Splunk Reference Hardware. In this deployment we installed and configured the following components for a distributed Splunk environment:

• One Splunk Search Head

• Three Splunk Universal Forwarders

• Three Splunk Indexers

The indexers and forwarders have a 1 to 1 ratio for the sake of the performance assessment tests. For an actual Splunk deployment, follow Splunk best practices to determine the appropriate ratio that fits your needs.

Figure 5 outlines the Splunk distributed environment.

Figure 5 Splunk distributed environment

The host and software components on the virtual machines are described in Table 2.

The Indexer VMs were installed with RedHat Enterprise Linux 7.2, configured with 24 virtual cores and 96 GB RAM each. Twelve VMAX devices of 256 GB were presented to each VM as raw device mapping (RDM). On the host we used Linux LVM to stripe the devices with 1 MB stripe depth and created a single volume, where an XFS filesystem was created and used for Splunk hot/warm data buckets.

The Forwarder VMs were installed in a similar way to the Indexers. They were configured with only 16 GB RAM as there was no need for more, in accordance with Splunk virtualization best practices of not overcommitting virtual memory. The reason 24 virtual cores were given to both Indexers and Forwarders (against best practices of not overcommitting vCores) was that the test workload separated the two test phases. During the setup phase the Forwarders were busy and during the test phase they were inactive, and only the Indexers were busy. In a non-test environment, a Forwarder and Indexer won’t share a physical server, nor will virtual cores be overcommitted.

11

The Search Head was configured in a similar way, but with a less powerful server: a UCS C240M3with 20 virtual cores and 24GB RAM.

Table 2 Performance assessment test hardware and software components Device Quantity Configuration Description

Indexer VMs 3 • RHEL 7.2

• 24 virtual cores

• 96GB RAM

• 12 x 256GB storage devices

• XFS filesystem

• Splunk Enterprise 6.4

Splunk Indexer VM

Forwarder VMs 3 • RHEL 7.2

• 24 virtual cores

• 16 GB RAM

• 12 x 256GB storage devices

• XFS filesystem

• Splunk Universal Forwarder 6.4

Splunk Forwarder VM

Search Head VM 1 • RHEL 7.2 • 20 CPU • 24 GB Memory

Splunk Search Head VM

Figure 6 shows the logical setup of the test environment.

Figure 6 Logical layout used for deployment

VIRTUALIZATION BEST PRACTICES In this deployment, ESXi hypervisor was installed on 4 physical hosts. The following best practices from Dell EMC, VMware and Splunk were implemented to provide optimal performance for all Splunk virtual machines running on the ESXi hosts:

• VMware Native Multi-Pathing (NMP) was used. This enables all device multipathing to be performed at the ESXi level.

o The default I/O operations limit parameter was changed from 1000 to 1 for the round robin path selection policy.

• Raw device mapping (RDM) for VMAX device allocation was used.

o In a virtual deployment there is a choice between using VMFS or RDM. Physical RDM, or pRDM, allows 1:1 device mapping between storage and VM which in turn allows easier use of storage replications such as snapshots. VMFS allows easier management at the hypervisor level.

12

• Dynamic resource scheduling (DRS) and high availability (HA) were disabled to prevent Indexers from being overloaded onto single host.

• Guest IO scheduler on all VM’s was disabled.

• Network Time Protocol (NTP) synchronization between all Hosts and VMs was established.

For detailed information concerning these best practices, refer to VMware: Performance Best Practices for VMware vSphere 6.0, Dell EMC: Using EMC VMAX Storage in VMware vSphere Environments and Splunk: Deploying Splunk Enterprise Inside Virtual Environments.

INDEXER SETUP

STORAGE SETUP USING LVM STRIPING

On each Indexer VM, Linux LVM was used to create a single volume group (VG) that contained the 12 VMAX All Flash devices. Then a logical volume (LV) was created striped across the VG. Finally, an XFS filesystem was created on the LV and mounted for Splunk Enterprise Hot/Warm buckets.

The following commands are a sample script for creating the storage setup on the Indexer. Note that the Inq binary is used to identify the appropriate devices based on their size. The Inq binary can be found at: ftp://ftp.emc.com/pub/symm3000/inquiry/. Since it is a stand-alone binary and does not require the full Solutions Enabler package it is easy to use for quick device listing. If Solutions Enabler is installed on the VM then the sympd command can be used instead.

#!/bin/bash set -x export dev_count=12 export stripe_depth=1m # Create a list of the devices with 256GB to work on #################################################### # sympd list -gb | grep " 256.0" | awk '{print $1}' > 256GB_devs.txt /download/inq -no_dots | grep 262145280 | awk '{print $1}' > 256GB_devs.txt # Read all devices to an array ############################## mapfile -t devsArray < ./256GB_devs.txt # run pvcreate and vgcreate ########################### INDEX=1 while [ "$INDEX" -le $dev_count ] do dev=${devsArray[$INDEX-1]} # get the next device from the array echo adding device: $dev pvcreate -f $dev # pvcreate if [ "$INDEX" -eq 1 ]; then # only the first device creates the volume vgcreate splunkvg --physicalextentsize 32MiB --force --yes $dev else # the other devices extend the volume vgextend splunkvg $dev fi INDEX=$((INDEX+1)) # next device done # create the volume ################### lvcreate --name splunklv --stripes $dev_count --stripesize $stripe_depth --size 3000G splunkvg # display the volume #################### lvdisplay /dev/splunkvg/splunklv # FS Create time mkfs.xfs -n size=64k -f /dev/splunkvg/splunklv

13

mkdir /mnt/splunk if [ ! `cat /etc/fstab | grep splunklv | wc -l` ]; then echo "Adding entry to fstab" echo "/dev/splunkvg/splunklv /mnt/splunk xfs noatime,nodiratime,nobarrier,nodiscard,nodev 0 0" >> /etc/fstab fi mount /mnt/splunk chown -R splunk.splunk /mnt/splunk

HOST LIMITS

The following Linux tuning parameters were adjusted on all Splunk Indexers to support better performance of the Splunk Infrastructure:

• Increased shell limits (for Splunk users) on all Indexer VM’s. These values are found in the /etc/security/limits.conf file.

o Number of open files (nofile)

Soft = 8192

Hard = 65536

o Number of user processes (nproc)

Soft = 16384

Hard = 16384

14

PERFORMANCE ASSESSMENT TESTS

Test stages

For this white paper, Dell EMC performed a series of tests to assess the performance of VMAX storage with varied synthetic indexing and search workloads for Splunk Enterprise. This test scenario demonstrates the high performance of VMAX All Flash in a Splunk Enterprise environment.

Table 3 describes the test stages

Table 3 Test Stages Stage Test stage name Details Test duration

1 Index stage 1 TB of data from each of the 3 forwarders transformed into events and processed by the Indexers

8 hours

2 Static search stage Search test to run over data in the static index without an indexing workload

8 hours

3 Streaming search stage Streaming search stage to introduce the indexing load to the system and at the same time run the search test

9 hours

For both static and streaming search, the test methodology introduces various search types with a different concurrency, as Table 4 explains.

Table 4 Search Types Test Stage Name Search Type Search Type Details Concurrency

Static and streaming search stage

Dense search This search returns a large set of matching records for a given set of data in a given period of time.

Note: Dense searches are primarily CPU bound.

4, 8, 12,...,64

Rare search This search returns a very small number of matching records for a given set of data in a given period of time.

Note: Rare searches are primarily I/O bound

4, 8, 12,...,64

Note: For more information regarding dense and rare searches and why they are CPU-bound and I/O-bound, see: How search types affect Splunk Enterprise performance.

15

TEST RESULTS Figure 7 illustrates the peak (maximum) search performance as seen from the array during the full Splunk test run. During the IOPS bound test scenarios, the array reached 118,000 IOPS and during the bandwidth bound test scenarios the array reached 2.6 GB/sec read bandwidth.

Figure 7 peak search performance

TEST SUMMARY By using a single-engine VMAX All-Flash with 32 x SSD flash drives, and 3 x Dell R730 Indexer servers, and by following the best practices described in this paper, we were able to achieve 2.6 GB/sec bandwidth, and 118K IOPS. These are not VMAX All-Flash performance limits and they only represent the test environment and configuration used for the performance assessment tests. It is important to size VMAX correctly based on the specific workload and data services requirements. Dell EMC VMAX Sizer is the easy to use tool that Dell EMC personnel use to determine the appropriate VMAX configuration, based on the user’s requirements.

All the test results indicated that the configuration used for this white paper exceed the performance of Splunk’s documented reference hardware. For more information, refer to Reference Hardware in the Splunk Capacity Planning Manual.

16

Summary Deep insight into new or previously ignored data sources has resulted in increased competitive advantages for corporations, as they can improve productivity, profitability, customer experience, and customer retention. Splunk is the leading platform in this space, enabling collection and analysis of data and real-time insights into data sources. As customers take advantage of these capabilities and increase the volume of their analyzed data, supporting the performance, reliability, and security of the underlying infrastructure becomes critical. VMAX All-Flash storage and Dell PowerEdge R730 servers configured according to the best practices published in this guide are an excellent choice for customers looking for flexible, easy to use, and performant infrastructure to support a Splunk Enterprise deployment.

References

Dell EMC Documentation • VMAX All Flash Data Sheet.

• Using EMC VMAX Storage in VMware vSphere Environments

VMware Documentation • Performance Best Practices for VMware vSphere 6.0

Splunk Enterprise Documentation • Welcome to Splunk Enterprise

• Splunk Enterprise Distributed Deployment Manual

• Managing Indexers and Clusters of Indexers

• Splunk Capacity Planning Manual

• Splunk Admin Manual