Delivering Predictable Storage Performance with Storage Quality of Service

Patrick Lang

CDP-B323

Private Clouds

Many VMsWorkloadsTenants

Compute servers

Shared storage clusters Scale-out File Server Cluster

Hyper-V Cluster

Virtual Machines

SMB3 Storage Network Fabric

Capacity Demand

OLTP

Web

Storage Pool

Capacity Demand

Capacity Demand

OLTP

Web

Storage Pool

OLTP

Web

Overprovisioning vs Oversubscription

Cost vs risk.

Overprovisioning• More

capacity• Higher

cost• Lower

average utilization

Oversubscription• Just

enough capacity

• Lower cost

• Max utilization

Capacity Demand

OLTP

Web

Web

OLTP

Storage Pool

Storage Pool

Capacity Demand

OLTP

Web

Web

OLTP

Storage Pool

Goals

Monitor storage performance & identify trouble signs

Use the right strategy to deploy Storage QoS

Deliver predictable performance in your private cloud

Storage Performance Monitoring

Performance MonitoringView from virtual machineWhat the customer sees

View from the hostWhat the admin sees

VM: Performance Counters

\PhysicalDisk(*)\Avg. Disk Queue Length\PhysicalDisk(*)\Current Disk Queue Length\PhysicalDisk(*)\Avg. Disk sec/Read

Host: Performance Counters

\Hyper-V Virtual Storage Device(*)\Queue Length\Hyper-V Virtual Storage Device(*)\Latency\Hyper-V Virtual Storage Device(*)\Normalized Throughput

Performance Counters in PerspectiveQueue DepthAverage is good indicator of impactDesktop workloads have more variability than server

Aim for <2 for good responsivenessServer may be sustained

Aim for <16

Average LatencyUsers likely to notice sluggish performance at > 25ms

Normalized ThroughputTrack overall cost for meteringUse to set Storage QoS policies

Normalized IOPsInput/Output Operations per Second, adjusted for size

1k = 1 normalized IOP8k = 1 normalized IOP10k= 2 normalized IOPs64k = 8 normalized IOPs

Available in performance counters & VM metrics

Used by Storage QoS

VM MetricsIntroduced in Windows Server 2012 R2

Provides accounting forCPUMemoryNetworkDisk

Preserved across VM state changesStart/Stop/RebootLiveMigration

Tracking Usage with MetricsEnabled using PowerShellEnable-VMResourceMonitoringMeasure-VMReset-VMResourceMonitoringDisable-VMResourceMonitoring

Includes Per-Disk Usage(get-vm TestVm1 | measure-vm).HardDiskMetrics

VirtualHardDisk : Microsoft.HyperV.PowerShell.HardDiskDriveAverageNormalizedIOPS : 5569AverageLatency : 44760DataRead : 1100DataWritten : 565NormalizedIOCount : 213045 (New in Technical Preview)

Monitoring Recap

Performance Counters

• VM or Host• Identify impacted VMs

VM Metrics

• Host• Measure & plan

Storage QoSin Windows Server 2012 R2

Scale-out File Server Cluster

Hyper-V Cluster

Virtual Machines

RateLimiter

s

RateLimiter

s

RateLimiter

s

RateLimiter

s

SMB3 Storage Network Fabric

Storage QoS – Windows Server 2012 R2

FeaturesMetrics - Per-VM & VHDMinimum & Maximum normalized IOPs per VHD

BenefitsMitigate impact of noisy neighbors with limitsMeet performance minimums on local non-shared storageNo dependency on storage features

Applying Storage QoS in Windows Server 2012 R2Reactive approachEnable performance counters or metrics on all machinesSort by normalized IOPsDecide if usage appropriate, or apply a limit

Proactive approachApply limit to all individual VM’s VHDsRaise as needed based on customer demandConsider tiers based on planned usage as VMs deployed

VDI: 100 IOPs limitGeneral: 500 IOPs limitHigh Performance: > 500 IOPs limit

What’s Next in Storage QoSWindows Server Technical Preview

Automating Storage Management

Easier to monitorMetrics from all VMs available on Scale-Out File ServerEnabled by default

Easier to manageImproved fairness by defaultPolicies centralized on Scale-Out File ServerFlexible - Per-VM, Per-Tenant, …

Effective on shared storageMeet storage service level agreements even with changing demands

Joint work with Microsoft ResearchPart of Predictable Data Centers effort

http://research.microsoft.com/datacenters/

Mission statement: “Provide customers with predictable performance in shared cloud environments.”

In-depth technical material: Storage Developer Conference (SDC’14) presentation:

http://www.snia.org/sites/default/files/TomTalpey_Storage_Quality_Service.pdfAcademic publication at SOSP’13:

http://research.microsoft.com/apps/pubs/default.aspx?id=198941

Continuous Performance Refinement Measure

current capacity

Use max-min fair algorithm

to meet policies

Adjust rate limits at the

source

Respond to changing demand

Scale-out File Server Cluster

Hyper-V Cluster

Virtual Machines

I/OSched

I/OSched

I/OSchedPolicy

Manager

RateLimiter

s

RateLimiter

s

RateLimiter

s

RateLimiter

s

SMB3 Storage Network Fabric

Storage QoS Evolved

Building BlocksCentralized policy manager on Scale-Out File Server Cluster

I/O Scheduler distributed across storage nodes

Profiler & rate limiter on Hyper-V compute nodes

Storage QoS MonitoringEnabled by default on Scale-Out File ServerTechnical Preview – see guide for upgrade instructions

Identifies and tracks usage by all VMs

Available DataFlow – VM Name & Id, HostCounters - *IOPs, *LatencyPolicy – PolicyId, MinimumIOPs, MaximumIOPs

Storage QoS PoliciesDefined on Scale-Out File Server

Applied to Hyper-V virtual disk

The rest is automaticScale-out File Server Cluster

Hyper-V Cluster

Virtual Machines

I/OSched

I/OSched

I/OSchedPolicy

Manager

SMB3 Storage Network Fabric

Policy 1Policy

2

Storage QoS PoliciesProperty Example ValueName DesktopVM

PolicyID 8d730190-518f-4087-9362-3971255acf36

MinimumIOPs 100

MaximumIOPs 200

TypeSingle InstanceMulti-Instance

MultiInstance

Types of Policies

Single-InstanceResource distributed among VMsIdeal for representing a clustered workload, application, or tenant

Multi-InstanceAll VMs perform the sameIdeal for creating per-VM performance tiers

IOPs0

20

40

60

80

100

120

140

160

180

200

100

100

VM1 VM2 VM1 VM20

20

40

60

80

100

120

140

160

180

200

Maxim

um

IOPs

= 2

00

Maxim

um

IOPs

= 2

00

Sizing Policies

Min < Max Min = Max

Min Only Max Only

Maximum OnlyNo guaranteed minimum performanceEqual weightReduces variability compared with no policy at allCan reserve capacity for later

Max=500

Minimum = MaximumMost consistent performance for VMsUse to offer performance tiers to customersNo bursting

Performance weighted if over-capacity

100

Minimum < MaximumMeets minimum performance SLAUse to offer performance tiers to customers

Variable performance above minimumAllows for burstingExcess capacity fairly distributed according to demand

Maximum enforced

Max=200Min=100

Minimum OnlyMeets minimum performance SLACan ensure important workloads deliver good performance

Highly variable above minimumExcess capacity fairly distributed according to demand

Highest overall capacity utilizedBest for burstingIndividual VM performance will degrade as load increasedMin=100

Single Instance for Tenant1 Policy createdOverall MinimumIOPs, MaximumIOPs

Applied to each VM for tenant

Storage QoS adjusts based on demandCan see dynamic Min/Max distributed among VMs within policy

Creating Child PoliciesSingle-instance policies can be subdividedProvide specific allocations to tenant VMs

Tenant1MinimumIOPs = 1000MaximumIOPs = 1000

Tenant1_ImportantVmMinimumIOPs = 500

MaximumIOPs = 1000

Evaluating Windows Server Technical PreviewStorage QoSStep by step deployment guide

http://go.microsoft.com/fwlink/?LinkId=517877* Be sure to check upgrade steps

Hyper-V Management Scripts

https://gallery.technet.microsoft.com/scriptcenter/Storage-QoS-script-for-a0d26330

Storage Survival GuideIncludes links to new features, step by step guides, scripts, and more

http://blogs.technet.com/b/josebda/archive/2014/10/03/windows-server-technical-preview-storage-survival-guide.aspx

Scale-out File Server Cluster

Hyper-V Cluster

Virtual Machines

I/OSched

I/OSched

I/OSchedPolicy

Manager

RateLimiter

s

RateLimiter

s

RateLimiter

s

RateLimiter

s

SMB3 Storage Network Fabric

Control and monitor storage performance

Flexible and customizabl

e

Policy per VHD, VM, Service or Tenant

Define Minimum & Maximum IOPs

Fair distribution within policy

Simple out of box behavior

Enabled by default for Scale Out File Server

Automatic metrics (normalized IOPs & latency) per VM & VHD

Management

PowerShell built-in for Hyper-V and SOFS

System Center VMM and Ops Manager

Storage QoS – Windows Server Technical Preview

CDP-B349: Storage Management in a Hybrid Cloud Environment with Windows Server and

System Center (Thursday Oct 30 3.15 PM)

Be ReadyMonitor storage performance & identify trouble signs

Use the right strategy to deploy Storage QoS

Deliver predictable performance in your private cloud

Storage Sessions in chronological order

CDP-B323: Delivering Predictable Storage Performance with Storage Quality of Service in Windows Server vNext (Wednesday Oct 29 8:30 AM)

CDP-B339: Leveraging SAN Replication for Enterprise-Grade Disaster Recovery with Microsoft Azure Site Recovery and System Center 2012 R2 (Wednesday Oct 29 3.15 PM)

CDP-B352: Stretching Failover Clusters and Using Storage Replica for Disaster Recovery in Windows Server vNext (Wednesday Oct 29 5 pm)

CDP-B349: Storage Management in a Hybrid Cloud Environment with Windows Server and System Center (Thursday Oct 30 3.15 PM)

CDP-B325: Designing Scale-Out File Server with DAS Storage in Windows (Friday Oct 31 8.30 am)

Related Content

Find Me Later At the Hyper-V & Virtualization Booth

Come visit us in the Microsoft Solutions Experience (MSE)!Look for the Cloud and Datacenter Platform area TechExpo Hall 7

For more informationWindows Server Technical Previewhttp://technet.microsoft.com/library/dn765472.aspx

Windows Server

Microsoft Azure

Microsoft Azurehttp://azure.microsoft.com/en-us/

System Center

System Center Technical Previewhttp://technet.microsoft.com/en-us/library/hh546785.aspx

Azure Pack Azure Packhttp://www.microsoft.com/en-us/server-cloud/products/windows-azure-pack

Resources

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources for IT Professionals

http://microsoft.com/technet

Sessions on Demand

http://channel9.msdn.com/Events/TechEd

Developer Network

http://developer.microsoft.com

Please Complete An Evaluation FormYour input is important!TechEd Schedule Builder CommNet station or PC

TechEd Mobile appPhone or Tablet

QR code

Evaluate this session

© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.