Building Your Own Robust and Powerful Software Defined Storage with VMware vSAN Tips on Choosing Hardware for vSAN Deployment
2
Agenda
1 Overview of VSAN
2 VSAN VCG at a Glance
3 VSAN Hardware Guidance (Ready Nodes, Post Deployment Experience)
4 Ready Node Configurator 2.0 - Demo
5 VSAN Certification Under the Hood
6 Q & A
Overview of VSAN
4
Powered by VMware VSAN and vSphere
Runs on any standard x86 server
Pools HDD/SSD into a shared datastore
Delivers enterprise-grade scale and performance
Managed through per-VM storage policies
Deeply integrated with the VMware stack
vSphere VSAN
VSAN Datastore
5
Supporting a Broad Variety of Use Cases
Virtual San
Business Critical Apps Virtual Desktops (VDI)
DR / DA
Test/Dev DMZ
ROBO Management
Staging
6
VSAN is the Market Leader
VSAN Customers 5,000 Y-O-Y
Growth >200%
Countries have VSAN Presence
109
Industry Verticals use VSAN
100%
Penetration in Fortune 1000 (US)
40%
5 of top 7 Healthcare companies in US
20 of top 25 Finance Companies in US
20 of top 25 High Tech Companies in US
12 of top 15 Retail companies in US
vSAN VCG at a Glance
8
8
164 Ready
Nodes
65 Unique Server
Platforms
15 Server
Partners
VSAN Offers the Widest Diversity of Server Choices
9
9
164 Ready
Nodes
65 Unique Server
Platforms
15 Server
Partners
VSAN Offers the Most Flexibility of Configuration Choices
✔ Widest Diversity of Platform Choices Most Flexibility ✔
10
10
164 Ready
Nodes
65 Unique Server
Platforms
15 Server
Partners
We are Continuously Refreshing and Keeping Up to Date….
70
Ready Nodes
AUGUST 2015
11
11
We are Continuously Refreshing and Keeping Up to Date….
163 NVMe
August 2016
? NVMe
August 2015
12
12
We are Continuously Refreshing and Keeping Up to Date….
163 NVMe
August 2016
0 NVMe
August 2015
13
We support the latest NVMe Devices
13
P3500, 3600, 3700
PM1725
SN150
14
14
164 Ready Nodes
65 Unique Server Platforms
15 Server Partners
163/ 220 NVMe / PCIe devices
122 IO controllers
Broadest Deployment Options
Deployment Choices
Broadest Deployment Choices
VxRail Ready Node Build Your Own
15
15
164 Ready Nodes
65 Unique Server Platforms
15 Server Partners
163 / 220 NVMe / PCIe devices
122 IO controllers
Broadest Deployment Options
Factory Install Broadest Deployment Choices
VxRail Ready Node Build Your Own
Software Flexibility
Ready to Run Ready Nodes
16
16
164 Ready Nodes
65 Unique Server Platforms
15 Server Partners
163 / 220 NVMe / PCIe devices
122 IO controllers
Broadest Deployment Options
Agility Broadest Deployment Choices
VxRail Ready Node Build Your Own
Out of Box Experience
Ready to Run Ready Nodes + Auto CONFIG
17
17
164 Ready Nodes
65 Unique Server Platforms
15 Server Partners
163 / 220 NVMe / PCIe devices
122 IO controllers
Broadest Deployment Options
Choice Flexibility Broadest Deployment Choices
VxRail Ready Node Build Your Own
Out of Box Experience
Ready to Run Ready Nodes + Auto CONFIG
Real Time Validation
Ready Node Configurator 2.0
Hardware Guidance
19
HCI is Pushing the Server IOs like Never Before!
19
CPU
DRAM
DRAM
PCIe Bus
Compute / Memory IO Subsystem
VSAN CERT
20
1. NVMe and PCIe Devices on the PCIe bus (Extension of Memory)
20
CPU
DRAM
DRAM NVMe PCIe
PCIe Bus
Compute / Memory IO Subsystem 10’s µs 10s ns – 1µs
21
2. Network Adaptors – Inter Cluster Communication
21
CPU
DRAM
DRAM
NVMe PCIe
PCIe Bus
Compute / Memory
IO Subsystem
NIC 10GbE
10’s ns – 1µs
10’s µs
22
3. IO Controllers to Propagate the IOs to the Drives
22
CPU
DRAM
DRAM
NVMe PCIe
PCIe Bus
IO Controller
SATA / SAS
Compute / Memory
IO Subsystem
NIC 100µs – 1ms
>10s ms
1 ns – 1µs
10’s µs
23
Design Guidance
23
1. All Components must be on the HCL/VCG
2. Turn off caching for HDDs ; turn off local drive cache
24
HCI is pushing the IO Subsystem like Never Before!
24
IO Controllers
NIC
NVMe / PCIe devices
Drives
1 2
3 4
VSAN CERT
VSAN CERT
VSAN CERT
IO CERT
25
VSAN Hardware Guidance Focus on 3 Components
25
HBA / IO Controllers NVMe / PCIe devices
Drives
1 2
3
26
Considerations for NVMe and PCIe devices
26
PCIe NVMe
PCIe Bus
Choosing the right form factor - HHHL, 2.5” and M.2
Better performance with NVMe as VSAN cache layer for higher IOPs
1
2
27
Considerations for NVMe and PCIe devices (and SSDs)
27
PCIe NVMe
PCIe Bus
Choose the correct Endurance and Performance Rating
Endurance: TB written in 5 years
Performance: Writes per second
3
*
http://www.vmware.com/resources/compatibility/vsan_profile.html
*
28
Considerations for NVMe and PCIe devices
28
PCIe NVMe
PCIe Bus
Choose the correct Endurance and Performance Rating
Endurance: TB written in 5 years
Performance: Writes per second
3
4
http://www.vmware.com/resources/compatibility/vsan_profile.html
Search for VSAN Quick Reference
Guide
29
3. IO Controllers to Propagate the IOs to the Drives
29
CPU
DRAM DRAM
NVMe PCIe
PCIe Bus
IO Controller
SATA / SAS
Compute / Memory
IO Subsystem
NIC
30
IO Controller Modes of Operation
30
Firmware Commands
RAID Acceleration Block
PCIe Interface
PCIe Host Interface
Link 0 Link 1
SAS / SATA Interface
(1) Passthrough
(2) RAID
31
VSAN prefers Passthrough Over RAID
Firmware Commands RAID Block
PCIe Interface
PCIe Host Interface
Link 0 Link 1
VSAN prefers passthrough mode over RAID mode
For best performance use controllers with Queue Depth of 512 or higher
1
2 SAS SATA Interface
32
1 Most Optimized VSAN Experience
2 Simplicity of Initial set up and Ongoing Operations
3 Consistency of Experience
Why Passthrough is Preferred
Passthrough mode is also called HBA mode
33
Do Not Share non VSAN and VSAN Drives on Same Controller
vSphere Virtual SAN
Boot Devices and Log Devices, should not be behind the same controller backing VSAN drives
VMFS (other no VSAN file systems) have different IO completion expectation
1
2
34
Make sure you use the latest Driver Firmware Listed ?
1. Running the correct combination of driver and firmware is critical for any IO Device
2. Don’t use a later version of driver and firmware if it is not on the VSAN VCG …no matter what anyone tells you!
35
Boot Options
SD / USB 4 – 8 GB
SATADOM Min: 16GB
SSD / HDD
§ Direct to Syslog server or use network dump Collector
§ Persist or use network dump collector
§ Endurance 384 TBW for
VSAN 6.2 and beyond
§ Persist or use network dump collector
§ Separate Controller compared to VSAN
Drives Connect to the PHYs and Arranged into Disk Groups
CONFIDENTIAL 36
Firmware Commands
RAID Acceleration Block
PCIe Interface
PCIe Host Interface
Link 0 Link 1 Disk Group:1
Disk Group:2
PHYs
37
Max Disk Group is 5 per HOST
• Cluster: 2-64 hosts • Host: 1-5 disk groups • Disk Group
– 1 SSD / NVMe / PCIe for cache
– 1-7 SSD or HDD for capacity
Host 1 Host 2 Host 3 Host 64 …
Disk Group
1
Disk Group
2
SSD or NVMe or
PCIe 1
SSD or
HDD1
SSD or
HDD2
SSD or
HDD7
Disk Group
5 …
…
Cache
Capacity
10GbE
1
7
38
Firmware Commands
RAID Acceleration Block
PCIe Interface
PCIe Host Interface
Link 0 Link 1
NVMe Caching Tier with SSD Capacity Tier
NVMe / PCIe
PCIe Interface
Disk Group
39
Disk Groups for VSAN
40 Max Drives per Host
3 - 4 Disk Groups
Balanced Disk Groups
More Drives / Disk Group
40
Read / Write Patterns for All Flash vs. Hybrid
SSD or NVMe or
PCIe 1
…
Cache
Capacity
All Flash Hybrid
CACHE R/W Split 100% Writes 70% Reads ,
30% Writes
Reads Capacity Tier Both Tiers
Writes Cache Tier Both Tiers
NVMe or
SSD
HDD HDD HDD
NVMe or
SSD NVMe
or SSD
Much lower endurance Capacity Device
Ready Nodes
42
Firmware Processing Unit
RAID Processing Unit
PCIe Interface
PCIe Host Interface
Link 0 Link 1
Ready Nodes Abstracts the Complexities
Ready Nodes abstract the complexities and provide a simple consumption model
BYO doesn’t factor in additional server hardware and Firmware Expanded Backplane
Chassis FW
Drives SAS Expander
Boot Options
43
Controller + Expander Certifications only available on Ready Nodes
Expanded Backplane
Chassis FW Drives SAS Expander Boot Options
Post Deployment Experience
45
Errors Do Happen…Don’t Worry!
Health Check
Burn In Tests
Flag incompatible Firmware, Drivers
Network, Performance, Capacity
46
Errors Do Happen…Don’t Worry! – Future Release
Health Check
Burn In Tests
Update Controller’s Firmware and Driver
Auto Config
Ready Node Configurator
48
DEMO
vSAN Certification Under the Hood
50
Principle 1 Identify the different types of I/O: Application IO
APPLICATION I/O
HOUSEKEEPING I/O
51
Principle 1 Identify the different types of I/O : Rebuild / Resync IO
APPLICATION I/O
RESYNC / REBUILD I/O
52
Principle 1 Identify the different types of I/O
APPLICATION I/O
ESX MANAGEMENT I/O RESYNC / REBUILD
I/O
HOUSEKEEPING I/O
53
Principle 2 Account for Worst Case Scenario
APPLICATION I/O
RESYCN / REBUILD I/O
HOUSEKEEPING I/O
✔
✔
✔
Worst case load on the system
54
Principle 3 Initiate Controller Task Management Activities during IO fulfillment
Virtual SAN
IO Controller
IO controller may initiate its own Error Handling Operations
This is over an above the regular IO Operations
55
Certification Suite Tests
Performance
Different combinations of
Read / Write profiles*
Steady State Analysis
Sustained period of time with
Heavy I/O
Worst Case Behavior
Planned and Unplanned tests
Drive or Disk Group
Failure
Stress Hot Plug
Sequential / Random
Both drive and entire disk group failure
Constantly Evolving and Enhancing our Certification Suite
Formulating Rigorous Equivalency model to reduce certification efforts
*100r0w, 70r30w, 0r100w, 50r50w
Q & A