Introduction to Solid State Drives

transcript

Introduction to Solid State Drive

Technology

Saturday, November 16, 13

Class Overview

• The Evolution of Storage Technology

• Spinning Disks

• Storage Metrics

• Solid State Technology

better understanding of spinning disks, understand high & low level flash, issues with SSDs

The Evolution of Storage Technology

Pre-History

Density/Time Speed/Time

Spinning Disks

The Parts of a Hard Drive

PlattersThe Parts of a Hard Drive

Actuator Arms and Heads

Controllerand Interface

Voltron Force Assemble!

Disk Interface

• Removable (USB/CF)

• SATA1 / II / III

• Nearline SAS

• SAS

• Fibre Channel

• PCI-e

Spinning Disk Removable Media

• Advantages:

• Nigh Universal

• Disadvantages:

• Slower

• Fragile

• Easily lost.

• Abstraction Layer

Disk Interface

SATA I / II / III

• Speeds: 1.5 / 3 / 6Gb/s

• Requires AHCI for things like NCQ

• Subset of SAS

• Shares IDE command set

Disk Interface

AHCI - Advanced Host Controller Interface (IDE is ok for TRIM)NCQ on SSD ensure SSD has things to do while host is latent(Intel can queue 32 requests) - logo from SATA-IO (intnl org)

SATA 3.1

• Approved July 2011

• Universal Storage Module

• mSATA

• QTRIM

Disk Interface

(This time, it’s personal)

QTRIM - queued TRIM commands, USM is a mobile drive standard

SAS / Nearline SAS

• SAS

• Enhanced CRC checking

• 512/520/528 bit blocks

• Low density, high reliability

• Nearline SAS

• ...not so much

Serially Attached SCSI - 16 bits of CRC

Disk Geometry

PlattersDisk

Geometry

TracksDisk

Geometry

Cylinders Disk

Geometry

Cylinders Disk

Geometry

SectorsDisk

Geometry

Logical Block Addressing

• First introduced as an abstraction layer

• Replaced CHS addressing

• Address Space is Linear (block 0 - n)

• Size of address space depends on the standard at time of manufacture.

DiskGeometry

Currently at 48-bit LBA -

Variables Affecting Spinning Disk IO Rate

Platter Rotational Speed

Seek Speed

Data Density

Controller CacheVariables Affecting Spinning

Disk Speed

Size / battery backed / hybrid drives

Spinning Disk Damage Vectors

Movement

• Movement vertical or parallel to platter

• Measured in G forces

• Head Crashes

• Spinning Down

• Head uses “Landing Strip”

• Repeated platter contact causes damage to the read/write head

Used to manually park the heads | Putting your computer to sleep can cause the head to park | nanocoating on the bumpy landing strip

Protection against movement

• “Active Drive Protection”: Free-fall sensor

• Has a lift arm to lift the head away from the platter

• Some protection systems are in the drive, some are in the controller

• Don’t mix the two

Apple: Sudden motion sensor, Lenovo/IBM: Hard Drive Active Protection System

Next slide: “You know, vibrations are movement...”

You know, vibrations are movement...

Yes, vibrations are important, too

Shelf Life

• Oil / Lubricants in bearings

• Temperature fluctuations

• Magnetic “events” (bit rot)

• Outgassing / vapor removal

Long-term “archival quality” drives with long-life lubricant Long term “cold storage” arrays which periodically spin up drives every few weeks to clean & scrub the data

Spinning Disks in RAID

• Redundant Array of Inexpensive Disks

• Common RAID levels:

• 0,1,5,6,10

• Software / Hardware

Important Considerations

• Redundancy

• Capacity

• Speed

• Robust

SpinningDisks

In RAID

Speed: Dedicated hardware? Single point of failure? Parity Calculation? How long to rebuild a drive? NUMBER OF SPINDLES!! Redundancy: How many drive failures? URE errors? Capacity: Parity stripe or mirrors? (harder better faster stronger)

Advantages

• Linear Speed

• Price (Per Gigabyte)

• Well-Understood

SpinningDisks

In RAID

Disadvantages

• Random Speed

• Price (per IOPS)

• Failure Rate

• Rebuild Speed

SpinningDisks

In RAID

Storage Metrics

• What are they?

• What aren’t they?

The Simplified Equation

IOPS = 1/(((R+W)/2)/1000) + (L/1000)

= Average Read Time= Average Write Time= Average Latency

Rule of Thumb Assumptions

RPM IOPS5400 50-807200 80-10010k 130-15015k 180-200

Determining IOPS

• Per Drive

• Manufacturer’s Stated Numbers

• Rule of Thumb

• Per RAID Array

• Write penalty

IO Profiling

• Active Tools

• Bonnie++

• dd

• Intel NAS Toolkit

• Passive Tools

• io(stat/meter/top), atop

• Resource Monitor / Process Explorer

http://www.intel.com/products/server/storage/NAS_Perf_Toolkit.htm

Solid State Drive Technology

NOR Flash

• Reads and writes are atomic single-bit

• Expensive

• Small specific use cases

Won’t talk about NOR much.

NAND Flash

• Reads are based on “read blocks” (4k)

• Writes are based on “erasure blocks”

• Cheap (and getting cheaper)

• Broad use cases

Read / Write Profiles

• Logical addresses abstracted from LBA

• No seek time

• Reads are generally very fast

• Writes are typically slower

Random and Linear IO have identical access timeNext slide: The magic

The Magic

InsulatingBarrier

Pure Silicon

Doped silicon capable of holding an electrical charge

Barrier is a dielectric film (silicon oxide)

Quantum Tunneling

(transmission coefficient for a particle tunneling through a single potential barrier)

Hot Carrier InjectionStorage / Erase uses Fowler-Nordheim Tunnel Injection / Release

Doped Silicon

Single Layer Cell (SLC)

Multi-Layer Cell (MLC)

Triple-Layer Cell (TLC)

Use charge pumps to get through the barrier Each charge level has a binary state - 1 or 0

Gradual Destruction

Energy increaseswith cell layers

Multiple cells needmultiple writes

Barrier accumulateselectrons

Electrical potential difference of barrier and cells disappears

Difficulty Going ForwardTLCTLC

000 100

001 101

010 110

011 111

4LC4LC4LC4LC

0000 0100 1000 1100

0001 0101 1001 1101

0010 0110 1010 1110

0011 0111 1011 1111

Density

• 3-Dimensional

• Charge levels

• Size of cells

• “Dot Pitch” (Cells Per Inch)

• 5nm, 3nm, 2nm

• Varies with “level” count

SLC / ESLC

• Low Density

• Single (bit) Level Cell

• Quick: 25µs Read / 200-300µ Write

• More robust & long wear time

• Write endurance near 100,000 cycles

Capacity expensive | only in 5nm / 3nm densities |

MLC / EMLC

• Reasonably High Density

• Two (bit) Level Cell

• Decently fast: 50µs Read / 600-900µs Write

• Medium lifetime

• Write endurance near 3,000 cycles

• Very High Density

• Three (bit) Level Cell

• Decently fast: 75µs Read / 900-1350µs Write

• Not very robust or durable :-(

• Write endurance ~ 1,000 cycles

Write Amplification and Garbage Collection

Block Sizes

• Read Block

• 4k (aka “page”)

• Erasure Block

• (Large) multiple of 4k

• aka “block” 256KB erasure

block size

e-ink parallel

Write Amplification

Written Data

Empty Cell

next - want to change the data in the upper right quadrant

Write Amplification

Written Data

Empty Cell

New Data

Old Data

next - big chunk of new data to write

Write Amplification

Written Data

Empty Cell

Old Data

New Data

Where does this go? We’re out of empty erasure blocks!

Write Amplification

Written Data

Empty Cell

New data writtenover old cell

w/o TRIM

Write Amplification

Written Data

Empty Cell

New data writtenover old cell

w/ TRIM

Garbage Collection

IO Performance Profiles

Remember:

• Spinning Disks

• Linear is fast

• Random is slow

• Read marginally faster than writes (sometimes)

writes slower when switching tracks

With SSDs:

• Reads are fast

• Writes are slow(ish)

• Random or linear doesn’t matter (as much)

SSD Performance Overview

• Depends on

• Number of flash chips in use

• Number of busses from the processor

• Performance of controller CPU

• Contention

• Bus speed

• Number of erasure blocks used

• Number of previous writes to flash cells

• Chips

• IO Busses

• CPU Cores

Causes of Contention

• Legitimate use

• Garbage collection

• Legitimate (but latent) useage

• IO Blender!

(Bender Blender: http://bit.ly/10vc7Sf)Saturday, November 16, 13

Latent: updatedb? atime? app-level garbage collection? (t-shirt at threadless)

Bus Speed

• SATA - 3 or 6 Gb/s?

• IOPS Calc

• Can your controller handle your disks?

• Very fast

• No seek time

• moderately improved over spinning disk (linear - random greatly)

• Causes no damage to the media

• Generally scales up with capacity

• Usually fast (depending on drive usage)

• No seek time

• highly improved over spinning disk

• Causes no damage to the media

• Generally scales up with capacity

Spinning DiskRead/Write Matrix

Read Write

Linear

Random

SSDRead/Write Matrix

Read Write

Linear

Random

Solid State in Practice

Solid State Form Factors

Removable Media

Drives

PCI Cards

Next slide: parts of an SSD

Parts of an SSD

Interface

SATA/SAS

IDE (sadly?)

ControllerMain

Processor

I/O Bus Lanes

RAMCache

Battery / SuperCapacitor

Flash Chips

If individual chip capacity is finite, how do bigger drives increase capacity? What does this mean for performance?

Flash Controllers

• Flash Translation Layer (FTL)

• Stripe Writes

• Interpret bus instructions

• Wear Leveling

• Garbage Collection

Do the heavy lifting - single largest problem with flash drives, without a doubt.

Flash Translation LayerLBA (0...n blocks)

F L A S H C H I P S

SSD Aspects & Concerns

Longevity

• Primarily determined by the class of flash

• (e)SLC, (e)MLC, TLC

• Related to wear-leveling

• Under-reported capacity

• Short-stroking improves lifetime (not speed)

Partition Alignment

• Performance and longevity

• As big (or bigger) issue than it was in spinning disks

• Native 4k read blocks

• Far larger erasure blocks

• larger than is practical for alignment

• As a command, refers to ATA-8 spec

• SCSI equivalent is UNMAP, but both are often referred to as TRIM.

• Does not immediately delete unused blocks

• Allows for GC

Linux calls this “discard” - TRIM refers

Linux TRIM Support

• EXT4 / XFS / JFS / BTRFS - Native using ‘discard’ option

• Consider NOOP or Deadline IO scheduler

• fstrim (part of util-linux) for R/W vols

• zerofree for R/O vols

fstrim & zerofree - userland - important for thin-provisioned volumes on SAN arrays which support it. Check docs on schedulers for details - deadline prefers read queues (under /sys/block)

OSX Trim Support

• Comes by default on factory-installed SSDs

• Trim-Enabler

• http://www.groths.org/trim-enabler/

ZFS and SSDs

• ZFS Intent Log (ZIL)

• Adaptive Replacement Cache (ARC)

• arc_summary can help you decide

ZIL is almost like a journal - ARC is a RAM cache that has disk backing it. SSDs can be L2ARC - https://code.google.com/p/jhell/wiki/arc_summary

Filesystems in General

• Standard journaling filesystems

• Mount options (atime/relatime, etc), /tmp->tmpfs

• Next-Gen

• ZFS / BTRFS

• Distributed Filesystems

• DRBD

ZFS - SSD cache pool | ZFS/BTRFS are COW | DRBD no trim

Monitor Health w/ S.M.A.R.T.

• S.M.A.R.T. information

• vendor-specific

• Includes flash erase count

• smartctl on Linux and Mac

• Dozens of tools on Windows (check wiki)

Forensics

(http://bit.ly/fast11-wei-paper)

...Our results lead to three conclusions:

First, built-in commands are effective, but manufacturers sometimes implement them incorrectly.

Second, overwriting the entire visible address space of an SSD twice is usually, but not always, sufficient to sanitize the drive.

Third, none of the existing hard drive-orientedtechniques for individual file sanitization are effective on SSDs

Reliably Erasing Data From Flash-Based Solid State DrivesMichael Wei∗, Laura M. Grupp∗, Frederick E. Spada†, Steven Swanson∗

∗Department of Computer Science and Engineering, University of California, San Diego†Center for Magnetic Recording and Research, University of California, San Diego

SSD-enhanced RAID Array Considerations

Hardware / Software

• Dedicated CPU Power

• Battery-backed storage

Hardware RAID Controllers

• Trust (eyes on code)

• Excessive cost of HW

Software RAID Controllers

• Commercial Support

• Proprietary Tech

• Portability

• Spare CPU Cycles

single point of failure

TRIM / GC?

• Does the RAID software/device know enough to pass along TRIM?

• Will the array eventually crawl because of ongoing GC issues?

No software RAID that I know of supports it. Intel chipset for RAID0 with TRIM

Access Bandwidth

• How much data can a single drive transmit?

• How many drives are in the array?

• What is the aggregate bus speed to the array controller?

• What is the bus speed to the host(s)?

SSD Throughput Example

From Tech Radar: http://bit.ly/100UhvY

4.15Gb/s

Controller / Bus

• Speed / Ports

• How mature / reliable / tested?

RememberMe?

Just because buses exist in a storage array oesn’t make them magic and infinite in size

Tiering / Caching

Very slow, cheap disks

Faster spinning disks

SSD tier - hot blocks

Very fast SDRAM

Future Technology

Enhanced Capacity

Kowloon Walled City

Enhanced Longevity

Telomeres in chromosomes

Smart SSDs

Active Flash

What should I buy?

Questions?