1

The Four Horsemen ofStorage System Performance

Stephen Foskettstephen@fosketts.net@SFoskettBlog.Fosketts.net

© Foskett Services

© Foskett Services 2

Stephen Foskettis the organizer of Tech Field Day,

proprietor of Gestalt IT,strangely interested in storage,

baseball believer,all-around nerd, car nut,

Microsoft MVP and VMware vExpert,former first-chair bass clarinet player and punk rock frontman,

obsessive about lightbulbs, lover of a good Manhattan,watch blogger, Apple blogger, vegetarian blogger,

dad to three kids with anagram names,grammar obsessive, avid reader,

King of the Andals and the First Men,humanist, frequent traveler,

and (apparently) lover of his own voice

© Foskett Services 3

© Foskett Services 4

The Rule of Spindles

© Foskett Services 5

The Nature of Disks

• Disks are mechanical – heat, vibration, rotation, seek

• Read/write heads can only access a single spot on the disk at once

• Sequential throughput is much higher than random

© Foskett Services 6

Combining Spindles

• Spread data across drives to overcome disk performance limits

• RAID was invented for this

© Foskett Services 7

The Rule of Spindles

• Adding more spindles is usually faster than adding faster spindles

• Disks just can’t get much faster

• Slower disks are becoming the norm

© Foskett Services 8

Never Enough Cache

© Foskett Services 9

Overcoming the Limits of Spindles

• Solid state storage is much faster than disks – RAM, flash, etc

• Most modern storage systems are tiered, with RAM, flash, and disk

• Solid-state is more expensive, but flash is getting cheaper all the time

© Foskett Services 10

Five Uses for Disk Buffers•Read cache - frequently-requested data is read

from memory rather than disk• I/O-matching - slower disks and faster interfaces

work together•Read-around (ahead or behind) pre-fetch cache•Read-after-write - saving recently-written data to

serve later read requests•Command queue – writes are reordered

© Foskett Services 11

Write-Through and Write-Back Cache

© Foskett Services 12

I/O As a Chain of Bottlenecks

© Foskett Services 13

The Chain of Command

• Storage isn’t just disks and arrays; all that data has to go somewhere

• Most I/O travels through five or more busses or channels between CPU and disk drive

© Foskett Services 14

The Bottle Neck

• How long will it take to fill or empty a disk drive or array?

• Which is the slowest link?• Can we bring storage closer to

compute?

© Foskett Services 15

A Chain of Bottlenecks

© Foskett Services 16

A Lack of Intelligence

© Foskett Services 17

The Stack of Lies

• We have lots of compute power, but very little communication through the I/O stack

• Each layer simplifies for the next

• Disks “know” nothing about data (and neither do most arrays)

© Foskett Services 18

De-Multiplex and Communicate

• Generally, more communication through the stack gives a better result overall

• Removing the I/O blender will help

© Foskett Services 19

Building Better Storage

© Foskett Services 20

Defeating the Four Horsemen

1. Understand the nature of disks

2. Tier storage3. Attack bottlenecks4. Get integrated

Thank You!

Stephen Foskettstephen@fosketts.net

@SFoskett

blog.fosketts.netTechFieldDay.com

21