XPS:EXPL:FP:Architecture and Software for Scalable Persistent Memory

Peter Varman Rice University Houston, Texas pjv@rice.edu

NSF XPS Workshop, June 1-2, 2015, Arlington, VA

Outline

•  Background •  WrAP Architecture •  Hardware Controller Approach •  Software Approach (SoftWrAP)

Background Storage Class Memory (SCM): •  Byte-Addressable •  High-Speed •  Density •  DRAM form factors •  No Passive Power Consumption •  Persistent •  Types:

•  Memristor •  Magnetic Spin •  Phase Change Memory (PCM)

•  Supports New Applications (no longer require slow, block-based persistence)

Persistent

Byte Addressable

SRAM L2/L3 DRAM SCM FLASH HDD

Access Latency

Adapted form ISCA 2009 (Qureshi et al)

Background •  In Memory Databases (IMDB) & Caches

–  CSQL, Volt DB, SAP HANA and Memcached –  Fast Access Speeds –  Low Passive Power Consumption –  Persistence –  Fast Crash Recovery

•  Support for Unstructured Pointer Based Data –  New Graph Databases (Neo4J)

•  Single programmer interface –  Blurs distinction between in-memory and disk-based data

•  Supports New Data Management Applications

Durability Problem

CACHE HIERARCHY

PERSISTENT MEMORY

B 2

C 0

A 1

•  Durability of writes (even a single write) not guaranteed

•  A=1 is only in cache hierarchy and not in memory

C 0

D 0

A 0 STORE A, 1

Transaction

D 0

B 0 B 0

Durability Problem

CACHE HIERARCHY

PERSISTENT MEMORY

B 2

C 0

A 1

•  Durability of writes are not guaranteed.

•  A=1 is now only in a store buffer and not memory.

•  Unfortunate side effect of also invalidating cache entry.

C 0

D 0

A 0 STORE A, 1 CLFLUSH A

Transaction

D 0

B 0 B 0

A 1

Store Buffer

Invalidate

Durability Problem

CACHE HIERARCHY

PERSISTENT MEMORY

B 2

C 0

A 1

Durability of a store, A=1 now in persistent memory.

C 0

D 0

A 1 STORE A, 1 CLWB A PCOMMIT

Transaction

D 0

B 0 B 0

Store Buffer

Fortunately, 2/2015 Intel ISA Manual introduces: CLWB – write back PCOMMIT – persistent commit - $$$

Problem solved?

What about multiple stores?

Failure Atomicity

STORE A, 1 CLWB PCOMMIT STORE B, 2 CLWB PCOMMIT STORE C, 3 CLWB PCOMMIT STORE D, 4 CLWB PCOMMIT

Transaction STORE A, 1 STORE B, 2 STORE C, 3 STORE D, 4 Commit

Transaction

CRASH!! Data Is Inconsistent

CRASH!! Data Is Inconsistent

•  Sequence of coupled writes that must be updated as a group •  Pointer changes to a linked data structure

•  Failure can leave data structure in SCM in inconsistent state

Cache Eviction

B 2

CACHE HIERARCHY

PERSISTENT MEMORY

B 2

C 3

A 1

EVICTED

Another problem Cache evictions can leave SCM in inconsistent state.

C 0

D 0

A 0 STORE A, 1 STORE B, 2 STORE C, 3 STORE D, 4 Commit

CRASH!!

Transaction

D 0

Outline

•  Background •  WrAP Architecture •  Hardware Controller Approach •  Software Approach (SoftWrAP)

WrAP Approach Write Aside Persistence (WrAP)

•  Propagate updates along two paths

•  Use cache hierarchy for value communication

•  Log updates to SCM as write-combined streaming stores

•  Avoid fine-grained synchronous front-end operations

•  Hardware Wrap: Victim Cache to hold cache spills

•  SoftWrAP: Alias SCM locations to DRAM locations

Outline

•  Background •  WrAP Architecture •  Hardware Controller Approach •  Software Approach (SoftWrAP)

WrAP Architecture

COHERENT(CACHE(

HIERARCHY(

CPU( CPU(

L1(L1(

BUFFERS'SCM'

VICTIM'PERSISTENCE'CACHE''(VPC)'

Fast(Paths(

Background(Path(

Control(Engine(

Log

•  Victim Persistence Cache –  Contains evicted entries from

open wraps

•  Background Logging Path –  SCM Address/Value Pairs –  Log in persistent memory –  Streaming stores

Write Aside Persistence wrapOpen(); *X=5; *Y=7; wrapClose();

Hardware Controller

Accepted -- Computer Architecture Letters (2015)

Outline

•  Background •  WrAP Architecture •  Hardware Controller Approach •  Software Approach (SoftWrAP)

SoftWrAP Approach

§  Aliasing SCM locations aliased to DRAM locations

wrapOpen() Creates Log wrapStore(x, val) Streams location x and value to log Writes x to Alias Table wrapLoad(x) Load x from alias table or SCM if not present wrapClose() Close log & PCOMMIT // Can process table.

SoftWrap

STORE a, 5

STORE a’, 5 Append to Log

Alias Table

Address Value Size W 5 4 X -1 4 Y 7 4 Z 8 8 A 5 4 … … …

Hash(W)

Data Object

Address Value Size … … … … … … M 1 8 Z 3 8 N 1024

… … …

Hash Table A State: Retiring

Hash Table B State: Active

-  Double Buffered (2 Hash Tables)

-  Concurrent Retirement

-  Supports primitives and object types

-  Reads check both tables

-  5 table states.

-  Locks only on state change and openWrap.

Full

Publications •  E. Giles, K. Doshi, P. Varman, “Write Aside Persistence for SCM

in HPC,” Poster Finalist, ACM SRC, SC’14.

•  E. Giles, K. Doshi, P. Varman, “Wrapping Operations for Atomicity and Durability: A Position Paper on How to Simplify NVM Programming for Extreme Performance”, (Short Paper) IEEE HPEC ’14

•  . •  Giles E., Doshi K., Varman P., “Free Atomic Consistency in

Storage Class Memory Using Software-Based Write-Aside Persistence”, (Short Paper + Poster) CF’15, Ischia, Italy

•  Giles E., Doshi K., Varman P: “SoftWrAP: A Lightweight Framework for Transactional Support of Storage Class Memory” MSST, June 2015 (to appear).

Related Work

•  Battery Backup Based Work –  Whole System Persistence

•  Recoverable Memory Techniques •  Up-Front Cache Line Changes & Data Copying

–  BPFS (epoch barriers on cache eviction, copy-on-write) –  NV-Heaps (Logging) –  Kiln (NV-Victim Cache provides transaction buffering; 2 phase commit)

•  Software Control Layer and Compiler –  Mnemosyne (STM based interception of all reads and writes; undo) –  ATLAS (Automatically generates atomic regions; Undo Log)

•  Software Solutions and Versioning –  Consistent Durable Data Structures (versioning and garbage collection) –  REWIND (In-place updates; atomic double linked list)

Summary •  Storage Class Memories are an exciting new area of

research that will give rise to new software applications and architectures.

•  WrAP Architecture presented is a fast, straightforward approach to ensure transactional support in writing byte-addressed persistent data.

•  Scaling to distributed storage

•  Wrapping complete applications