Cisco HyperFlexDesign, Performance and...3. Data is then three way written (RF3) to SSD / HDD on 3...

Cisco HyperFlex Design, Performance and Benchmarking for All-Flash and NVMe Clusters

Silvo Lipovše, sistemski inženjer

19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia 3

Agenda

BRK

What’s New

All-Flash System Design

Benchmarking Tools

Performance Testing

Conclusion

What’s New in HyperFlex

4


What’s New With Cisco HyperFlex 2.6?

All-Flash Now Supports NVMe Caching Disks Support for Self Encrypting Disks (SEDs)

Cisco Intersight Cloud-Based Management

HyperFlex Connect Native HTML5 GUI with Role Based Access Control

Native Snapshot Based Replication Expanded Compute-Only Node Support

M5 Generation Server Models

Mixed Clusters of M4 and M5 Servers, Higher Capacities and

Larger Scale


New Disk OptionsNon-Volatile Memory Express (NVMe)

Replaces SAS write caching disk

More efficient streamlined protocol

Uses PCI-Express instead of legacy serial bus

Multi-queue, more commands per queue, high parallelism

Higher performance, lower latency

Self Encrypting Disk (SED)

Used for write caching disk and capacity disks

Disk hardware encrypts all data written to the device

Can use UCSM derived local encryption keys, or keys from a centralized KMS via KMIP

Secure erase, re-key


Cisco HyperFlex All Flash Configurations HXAF240c Cluster

Capacity-Optimized 3–16 Node Cluster

Per-NodeCaching Device: 1 x 800 GB SSD, 400 GB

NVMe, or 800 GB SEDCapacity Disks: up to 23 x 960 GB, 3.8 TB

SSD, or 800 GB SED Boot/Housekeeping: M.2 SSD + 120GB or

240GB SSD

HXAF220c Cluster

Smallest Footprint 3–16 Node Cluster

Per-NodeCaching Device: 1 x 800 GB SSD, 400 GB

NVMe, or 800 GB SEDCapacity Disks: 6 x 960 GB, 3.8 TB SSD,

or 800 GB SED Boot/Housekeeping: M.2 SSD + 120GB or

240GB SSD

Expanded Clusters

Compute-heavy Hybrid(Compute Bound Apps/VDI)

Up to 16 Compute NodesBlade or Rack

B200 M3/M4, B260 M4, B420 M4, B460 M4, C220 M3/M4, C240 M3/M4, and C460 M4

Local Disk, SD Card or SAN Boot

3-16 Node HXAF220 or HXAF240 Cluster

Standard UCS 2 & 4 Socket Servers

+


HyperFlex Native Data Protection and SecurityProtect against Site

Failure

Data-At-Rest Encryption thru Self-Encrypting Drives

Enterprise Key Management Support

Data-At-Rest Loss Prevention

Native Replication of Snapshots across Sites

Disaster Recovery Protection

Regulatory Compliance

Certifications starting with FIPS, Common Criteria

Compliance for Data Privacy (GDPR, HIPAA, PCI-DSS etc.)

Secure Platform

Vulnerability Assessments and Hardening

Role-based access control and Auditing


Cisco Intersight Cloud-Based Management


Native HyperFlex Connect HTML5 GUI

All-Flash System Design

11


Building on the Right FoundationCisco HX Data Platform

Built From the Ground Up for

Hyperconvergence

Distributed Log-Structured File

System Designed for Scale-out,

Distributed Storage

Advanced Data Services (Snapshots, Clones, Replication)

and Data Optimization (Inline Dedupe,

Compression) Without Trade-offs

Better Flash Endurance and

Disk Performance

Computing, Storage,

Networking, and Hypervisor Integration

No Reliance on Legacy

Filesystems or

Technology

Distributed File system

Local Disks

UniqueArchitecture

Local Disks Local Disks Local Disks


Software Modules Inside a Server

Controller VM Has Direct Control of Drives

VAAI Plugin Offloads Snapshots

and Clone OperationsIO Visor Module Presents

NFS to ESX and Stripes IO

DATASTORE/VOLUME

HYPERVISOR

HDD

HDD

SDD

SDD

SSD SSD SSD SSD

CONTROLLERVMVAAI

stHyp

VM VM VM VM VM

IO Visor

stHypervisorSvc Coordinates Native

Recovery Snapshots and Replication

19 - 21 March 2018 | Cisco Connect | Rovinj, Croatia

Read Cache

Log FS

L1

Read

Distributed Objects

All Flash

DRAM

Read Cache

14

Optimizing for All Flash: Read Path

Log FS

L2

L1

Read

Distributed Objects

Hybrid L1 (DRAM) Cache:• Metadata

and data • Helps with

small bursts

L2 (SSD) Cache:• De-duplicated

Read Cache• Use aggregate

performance of all Read Cache SSDs

SSD

DRAM

No L2 (SSD) Cache:• No flash on flash

advantage• Use aggregate

performance of all SSDs

SCVM RAM:• Unified capacity disk

index cache• Improvements by

avoiding additional index reads


Technical Improvements in HX 2.5+ Updated Linux kernel in the controller VMs

– Gains of 20-30% in read performance

Optimized write cache destaging process– More consistent performance with less variability

Metadata Optimization– Less space consumption, fewer updates required, faster index lookups

Cleaner Improvements– Lower performance impact, less variability

Write Log Enhancements– In-line compression with no latency penalty, makes effective write log size larger

15


DATASTORE DATASTORE

16

Data Distribution and Non-Disruptive Operations

Stripes blocks of virtual disk files across servers using a hashing algorithm

Replicate one or two additional copies to other servers

Handle entire server or disk failures

Restore back to original number of copies Rebalance VMs and data post replacement Rolling “one-click” software upgrades

CONTROLLERHYPERVISOR

VM VM VM


VM VM VM


VM VM VM


VM VM VM

File.vmdk

D1 E1A1 B1 C1B2 A2 A3C2 C3 D2D3 E2E3 D1E1 B3 B3

EDCBA


Life of a Write IO: Write LogCONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

AA’’ A’

ACKA

1. Application writes data (A) to OS2. VM writes to VMDK on datastore3. Write intercepted by IOvisor and sent to primary controller for that VMDK segment and offset4. Primary data is compressed and committed to WL SSD5. Simultaneously, data replicated to two other nodes (RF3) and committed to their WL SSD6. Acknowledgement sent to VM7. Data is successfully committed and protected

B B’ B’’CC’’C’

D D’D’’

Write Log Capacity Used

Size: 8GB or 32GB Hybrid / 60GB AF


Life of a Write IO: Destage

AA’’ A’

1. When the active write log of a node is full, the active log becomes the secondary log and destages2. Primary data copies are deduplicated3. Data is then three way written (RF3) to SSD / HDD on 3 separate nodes4. After all data is successfully committed, data is purged from the secondary log on the caching SSD5. In hybrid systems, recent writes are added to the read cache area of the SSD during destage

B B’ B’’CC’’C’

D D’D’’

Write Log Capacity Used

CONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

A’A’’AA

BBB’B’’

CC’’C’ D’’ D D’


CONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

CONTROLLERVM

IOvisor

VMVM

ESXiDS

19

Life of a Read IO

ESXi

Read A

1. Application reads data from OS filesystem2. VM reads from VMDK on datastore3. Read intercepted by IOvisor and sent to primary controller for that VMDK segment and offset

a) First check active write log b) Next check passive write log (the duplicated write log of the other nodes)c) Look in L1 (DRAM) cached) Look in L2 (SSD) cache [Not in AF models]e) Retrieve from SSD / HDD – copied to L2 [Not in AF models] and L1, decompress & returned to IOvisor

4. Return data to VM

Active WL

Passive WL

L1

L2

SSD (AF) / HDD (Hybrid)


Dynamic Data Distribution• HX Data Platform stripes data across all nodes simultaneously,

leveraging cache across all SSDs for fast writes, and HDDs for capacity, via the low-latency UCS internal network

• Balanced space utilization, no data migration required following a VM migration, failure tolerance, and no hotspots

Systems Built on Conventional File Systems Write Locally, Then Replicate, Creating Performance Hotspots

CONTROLLERHYPERVISORHYPERVISOR CONTROLLERHYPERVISOR CONTROLLERHYPERVISOR

VM VMVM VM VMVM VM VMVM

HX Data Platform

VM VMVM

CONTROLLERCONTROLLER


Scale Compute

HX Data Platform

Add NodesScale Capacity Within Nodes

HX Data Platform

CONTROLLERHYPERVISORCONTROLLERHYPERVISOR CONTROLLERHYPERVISOR CONTROLLERHYPERVISOR

IOVisorIOVisor

IOVisorIOVisor

IOVisorIOVisor

IOVisorIOVisor

VM VM VM VM

Non-HyperFlex Hosts Can Connect to

Storage with IOVisor

VM VMVM VM VMVM VM VMVM VM VMVM

IOVisor

VM VM

IOVisor

VM VM

21

Independent Scaling of Compute and Capacity


HyperFlex Capacity Considerations SSD reliability is significantly better than HDD; from 2-10 times lower annual

replacement rate in studies by major online providers.

HyperFlex installer no longer defaults to RF3 for an all-flash system, forcing the end user to make a decision.

Consider RF2 for all-flash systems that do not require triple data copies or multiple component redundancy.

RF2 shows 15-40% reduced write latencies in lab testing.

RF2 can provide clusters up to 591 TiB maximum capacity.

Keep overall cluster capacity consumed below 70% for best performance.

Benchmarking Tools

23


Iometer Intel project now developed as an open source project

No active development in 3+ years

Primarily a Windows GUI tool, but ports exist for Linux and Mac OS X

Easy configuration of an access profile

Not a primary tool for our team any longer

Issues:– Not very easy to set up server/client multi node tests– Difficult to script runs on Windows– No configurable dedupable or compressible data patterns– Default patterns are highly dedupable which can give false results

24


FIO Written by Jens Axobe, the current Linux kernel block layer maintainer

Compiled binary install for most Linux distributions

Configurations to test are written in text based job files

Highly customizable

FIO can be configured to run at startup, then a master machine can start tests remotely

Issues:– FIO must be installed on all testing machines– If FIO is not already running, you cannot start a multi-node test– Output can quickly get cluttered and hard to read

25


Vdbench Written by Henk Vandenbergh at Oracle

Java application that can run on almost all platforms with a JRE

Configurations to test are written in text based job files

Highly customizable

Easy server/client job execution via SSH or RSH

Easy to read output in HTML files

Has become the standard tool for the Cisco HX BU TME team

Issues:– Thread and JVM count can make calculating the OIO per disk difficult to calculate

26


Vdbench Output


Cisco HX Bench Cisco developed tool to automate many facets of benchmark testing

Download at https://hyperflexsizer.cloudapps.cisco.com (CCO login required)

Deploys a master test controller VM from an OVA file on your HX cluster

HTML GUI makes deployment of VMs and test configuration easy

Uses vdbench for the testing

Use built in test scenarios or build your own

Automatically deploys lightweight Ubuntu VMs as the test load generators

Test results and charts are customizable and even easier to view than the default HTML output from vdbench

28

https://hyperflexsizer.cloudapps.cisco.com/


Cisco HX Bench User Interface


Others HCIBench

– VMware automation wrapper that executes vdbench on Photon VMs– Results are identical to vdbench, somewhat easier to use but not as flexible

HammerDB– Used internally for database workload qualifications

SQLIO, SLOB, Jetstress, etc.– Application specific tests not used for general performance characterization

Mixed Workload Tools– Examples: Weathervane, VMmark

30


Vdbench Example Job FilesVdbench

dedupunit=8kdedupratio=1.25compratio=1.4hd=default,user=root,shell=ssh,jvms=1hd=vdbench1,system=192.168.100.11hd=vdbench2,system=192.168.100.12hd=vdbench3,system=192.168.100.13hd=vdbench4,system=192.168.100.14

sd=default,openflags=o_direct,threads=16sd=sd1,host=vdbench*,lun=/dev/sdb

wd=wd1,sd=sd*,xfersize=8k,rdpct=70,seekpct=100

rd=rd1,wd=wd1,iorate=max,elapsed=5m,interval=5


Vdbench Example Job ExecutionEnsure each load generator VM has the vdbench java executable in the same location, and copy SSH keys so SSH logins happen without a password.

Since the vdbench job file specifies the clients as well, just run the test:

vdbench –f 7030_4k.vdb –o /var/www/html/

• Easy to loop multiple runs in sequence:for i in {1..3} ; do vdbench –f 7030_4k.vdb –o /var/www/html/ && sleep 3600 ; done

• Simple scheduling of runs with Linux at:# at 1am tomorrow

at> vdbench -f 7030_4k.vdb -o /var/www/html/output/7030_4k.tod

at> <EOT> (Press CTRL+d to end the job creation)

Performance Test Results

35


HX Hybrid to All-Flash Performance Comparison

0,0

1,0

2,0

3,0

4,0

5,0

6,0

7,0

0.4GB 100GB 800GB 2TB

Perf

orm

ance

Mu

ltip

le

Effective Working Set Size per NodeTests done on 4 x HX240

Performance With Varying Working Set

All Flash Performance Multiple

HX240 hybrid read cache 1.2 TB per node

• AF offers performance improvements and lower latency even at small working sets

• AF offers huge performance and latency improvements for working sets that do not fit in the hybrid model read cache


Base Configurations for Vdbench Tests

HX All-Flash HX All-Flash + 40GbE HX All-Flash + 40GbE and NVMe HX All-Flash M5

Version HX 2.0.1a HX 2.0.1a HX 2.5.1a HX 2.6.1b

HW Configuration

8 x HXAF240 M4, each with:• WL: 800G SSD• Data: 10 x 960G• VIC 1227• E5-2690v4• 512 GB RAM

8 x HXAF240 M4, each with:• WL: 800G SSD• Data: 10 x 960G• VIC 1387• E5-2690v4• 512 GB RAM

8 x HXAF240 M4, each with:• WL: 400G NVMe• Data: 10 x 960G• VIC 1387• E5-2690v4• 512 GB RAM

8 x HXAF240 M5, each with:• WL: 400G NVMe• Data: 10 x 960G• VIC 1387• Xeon 6154• 384 GB RAM

Benchmark Setup

• Vdbench 5.04.06• 8 Linux VMs per host (64 total)• Each VM has 1 x 200GB virtual disk, HX filesystem ~60% full (12.4 TB working set)• Priming: 64K writes 100% sequential• Tests run 3 times and averaged


Benchmark 1 Performance Results Using Vdbench Tests8K 100% Sequential Write Workload

• Target Write Latency < 10ms• 16 OIO Per Disk• 27% Latency Reduction and

IOPs Gain with 40 GbE• 6% Latency Reduction and

IOPs Gain on 2.5 + NVMe• 8% Latency Reduction and

IOPs Gain with HX 2.6 on M596.426

122.514130.015

140.397

10,618,35 7,87 7,30

0,002,004,006,008,0010,0012,0014,0016,0018,0020,00

0

20.000

40.000

60.000

80.000

100.000

120.000

140.000

160.000

2.0 2.0 + 40GbE 2.5 + 40GbE 2.6 M5

LatencyIOPs

Maximum Sequential WriteIOPs Latency


Benchmark 2 Performance Results Using Vdbench Tests8K 100% Random Read Workload

• Target Read Latency < 5ms• 32 OIO Per Disk• 7% Latency Reduction and

IOPs Gain with 40 GbE• 23% Latency Reduction and

IOPs Gain on 2.5 + NVMe• 3% Latency Reduction and

IOPs Gain with HX 2.6 on M5

389.526415.449

511.992 529.241

5,26 4,934,00 3,87

0,002,004,006,008,0010,0012,0014,0016,0018,0020,00

0

100.000

200.000

300.000

400.000

500.000

600.000

2.0 2.0 + 40GbE 2.5 + 40GbE 2.6 M5

LatencyIOPs

Maximum Random ReadIOPs Latency


Benchmark 3Performance Results Using Vdbench Tests100% Random 70/30% R/W with Varying Block Sizes

4K 8K 16K 64K 256K

IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency IOPs R Latency W Latency

2.0 150,673 2.12 6.36 130,511 2.40 7.47 92,130 3.49 10.35 34,965 10.75 23.70 10,079 45.60 62.80

2.0 + 40 GbE 152,434 2.22 6.00 142,721 2.36 6.47 111,905 3.13 8.03 51,146 6.96 17.16 19,712 11.68 59.28

2.5 + 40 GbE 161,131 2.24 5.37 160,290 2.09 5.75 136,433 2.45 6.78 56,339 6.30 15.74 20,036 9.17 63.61

2.6 M5 180,037 1.72 5.49 173,193 1.91 5.39 140,842 2.36 6.61 61,227 5.45 15.18 20,732 8.96 61.37

• Target Read Latency < 5ms and Write Latency < 10ms for 4-16K blocks• 8 OIO Per Disk• ~6% Latency Reduction and IOPs Gain with 40 GbE at 4-8K, 50-100% Gains for Larger Blocks• 10-22% Latency Reductions and IOPs Gains with HyperFlex 2.5 and NVMe Caching Disks• 3-11% Latency Reductions and IOPs Gains with HyperFlex 2.6 on M5 Generation Servers


Benchmark 4Performance Results Using Vdbench Tests8K 100% Random 70/30% R/W IOPs Curve

• Curve test measures IOPs stability by calculating the standard deviation of IOPs as a percentage of the final average result, at various performance levels, starting with a 100% unthrottled test, then again at a percentage of that maximum

• Maximum speed, unthrottled tests will always have more variability due to more aggressive background tasks• 20% through 60% values are typically very low because the system is not being pushed very hard• 65% improvement in IOPs stability with 40 GbE at 80% of maximum HX 2.5 shows a further 78% improvement• HX 2.5 also shows a 54% improvement of IOPs stability in the 100% unthrottled test• HX 2.6 on M5 generation servers shows improvements in all tests, including 28% at 100%, and 45% at 80%

100% 20% 40% 60% 80%

IOPs IOPs STDV

Avg Latency IOPs IOPs

STDVAvg

Latency IOPs IOPs STDV

AvgLatency IOPs IOPs

STDVAvg


AvgLatency

2.0 138,190 19.0% 3.71 27,666 0.5% 1.07 55,299 0.5% 1.57 82,960 0.5% 2.20 110,594 6.6% 3.14

2.0 + 40 GbE 151,568 18.6% 3.38 30,331 0.5% 0.99 60,621 0.5% 1.37 90,957 0.5% 2.08 121,250 2.3% 2.63

2.5 + 40 GbE 169,983 8.6% 3.01 34,031 0.4% 0.86 68,058 0.5% 1.17 102,023 0.5% 1.66 136,018 0.5% 2.28

2.6 M5 173,193 6.2% 2.95 34,699 0.2% 0.80 69,331 0.2% 1.21 103,953 0.2% 1.68 138,580 0.3% 2.38



Benchmark 5Performance Results Using Vdbench TestsI/O Blender 100% Random 70/30% R/W IOPs Curve

100% 20% 40% 60% 80%

IOPs IOPs STDV

Avg Latency IOPs IOPs

STDVAvg


AvgLatency IOPs IOPs

STDVAvg


AvgLatency

2.0 104,624 25.6% 4.89 20,967 0.5% 1.31 41,898 0.5% 1.72 62,825 0.5% 2.53 83,756 4.4% 3.89

2.0 + 40 GbE 114,354 22.2% 4.47 22,902 0.5% 1.04 45,792 0.5% 1.47 68,664 0.5% 2.47 91,435 3.1% 2.97

2.5 + 40 GbE 137,078 13.6% 3.73 27,466 0.5% 0.96 54,897 0.5% 1.45 82,292 0.5% 2.13 109,727 0.5% 2.77

2.6 M5 137,416 12.1% 3.72 27,496 0.2% 0.80 54,990 0.3% 1.49 82,456 0.3% 1.96 109,955 0.4% 2.89

• Test measures IOPs stability by calculating the standard deviation of IOPs as a percentage of the final average result, at various performance levels, starting with a 100% unthrottled test, then again at a percentage of that maximum

• Block Size Mix: 4K 40%, 8K 40%,16K 10%, 64K 10%• 20% through 60% values are typically very low because the system is not being pushed very hard• 30% improvement in IOPs stability with 40 GbE at 80% of maximum, HX 2.5 shows a further 84% improvement• HX 2.5 also shows a 39% improvement of IOPs stability in the 100% unthrottled test• HX 2.6 on M5 generation servers shows improvements in all tests, including 11% at 100%, and 20% at 80%


0

5000

10000

15000

20000

25000

30000

1 7

13

19

25

31

37

43

49

55

61

67

73

79

85

91

97

10

3

10

9

11

5

12

1

12

7

13

3

13

9

14

5

15

1

15

7

16

3

16

9

17

5

18

1

Intervals

Vendor Y

VM1 VM2 VM3 VM4 VM5 VM6


48

Per VM Test Results VariationIOPS in a 70/30 Workload – 1 Hour Duration

0

5000

10000

15000

20000

25000

30000

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

106

113

120

127

134

141

148

155

162

169

176

Intervals

HX




HyperFlex Performance Summary

HyperFlex shows large gains in performance by transitioning to 40 GbE and 3rd Gen FIs and VICsHyperFlex 2.5 + 40 GbE offers significant gains in IOPs, latency and stability for all test workloadsHyperFlex 2.6 on M5 shows further improvements in IOPs, latency and more consistent performanceHX All Flash, unlike competition, shows highly consistent performance across all VMs

BRK50

Date post:	24-May-2020
Category:	Documents
Upload:	others
View:	0 times
Download:	0 times

Cisco HyperFlexDesign, Performance and...3. Data is then three way written (RF3) to SSD / HDD on 3...

Documents