PCI Express (PCIe)

Overview

Flash Memory Summit 2013

Santa Clara, CA

1

Peter Onufryk Sr. Director, Product Development

PMC-Sierra

NVMe and PCIe Layering

NVMe

• Driver interface (registers)

• Queuing interface

• Command set

• Command processing model

PCIe

• Reliable memory read/write transactions

• Discovery and configuration

• Switching & routing

• Physical layer

Flash Memory Summit 2013

Santa Clara, CA

2

PCIe Characteristics

Specification defined by PCI-SIG

• www.pcisig.com

Packet based protocol over serial links

• Software compatible with PCI and PCI-X

• Reliable in-order packet transfer

High performance and scalable from consumer to enterprise

• Scalable link speed (2.5 GT/s, 5.0 GT/s, 8.0 GT/s)

• Scalable link width (x1, x2, x4, …. x32)

Primary application is as an I/O interconnect

• Not a CPU interconnect

• Some multi-host applications (NTB, MR-IOV)

• Some outside the box applications (PCIe cable)

Flash Memory Summit 2013

Santa Clara, CA

3

PCIe and Server Architecture

Flash Memory Summit 2013

Santa Clara, CA

4

*Source - Intel

PCIe is Everywhere

Flash Memory Summit 2013

Santa Clara, CA

5

Blade Server

SERVERS

Mid-range / High End

COMMUNICATIONS

Metro Switch / Router Enterprise Switch / Router Wireless

STORAGE

RAID System SAN Switch NAS

CONSUMER

Small Business

Entertainment Imaging Mobile

PCIe Fabric Topology

Flash Memory Summit 2013

Santa Clara, CA

6

Root Complex

DP DPDP

PCIe Switch

En

dp

oin

t

UP UP

DP DPDP

PCIe Switch

UP

DP DPDP

PCIe Switch

UP

DP DPDP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

Data Transfers

Flash Memory Summit 2013

Santa Clara, CA

7

Root Complex

DP DPDP

PCIe Switch

En

dp

oin

t

UP UP

DP DPDP

PCIe Switch

UP

DP DPDP

PCIe Switch

UP

DP DPDP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UPE

nd

po

int

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

PCIe and NVMe

Flash Memory Summit 2013

Santa Clara, CA

8

NVMe Host

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSD

NVMe

SSD

Root Complex

DP DPDP

PCIe Switch

NVMe

SSD

En

dp

oin

t

UP UP

DP DPDP

PCIe Switch

UP

DP DPDP

PCIe Switch

UP

DP DPDP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

tUP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

PCIe Layers

Flash Memory Summit 2013

Santa Clara, CA

9

TransactionLayer

Data LinkLayer

PhysicalLayer

TransactionLayer

Data LinkLayer

PhysicalLayer

PCI

Architecture

PCI

Architecture

PCIe Link

Upstream Port

PC

Ie Lane

Downstream Port

PC

Ie Lane

PC

Ie Lane

PC

Ie Lane

PCIe Link

Physical Layer

Scalable Speed

• Gen1 – 2.5 GT/s

• Gen2 – 5.0 GT/s

• Gen3 – 8.0 GT/s

Scalable Width

• x1, x2, x4, x8, x12, x16, x32

Encoding

• 8b10b: 2.5 GT/s & 5.0 GT/s

• 128b/130b: 8 GT/s

Flash Memory Summit 2013

Santa Clara, CA

10

PCIe Performance

Flash Memory Summit 2013

Santa Clara, CA

11

Generation Raw

Bit Rate

Bandwidth

Per Lane

Each Direction

Total x16

Link Bandwidth#

Gen 1 2.5 GT/s ~ 250 MB/s ~ 8 GB/s

Gen 2 5.0 GT/s ~500 MB/s ~16 GB/s

Gen 3 8 GT/s ~ 1 GB/s ~ 32 GB/s

Source – PCI-SIG PCI Express 3.0 FAQ

# Link bandwidth in each direction x 2 (full duplex)

Data Link Layer

Primary Function

• Reliable exchange of Transaction Layer Packets (TLPs)

between the two components of a Link

Other Functions

• Initialization (flow control credits)

• Power management

• Track and report link state to transaction layer (i.e., DL_up &

DL_Down)

Flash Memory Summit 2013

Santa Clara, CA

12

Transaction Layer

Primary Function

• Assembly and disassembly of

Transaction Layer Packets

(TLPs) for read and write

transactions

Other Functions

• Event signaling (e.g., interrupts,

power management, errors)

• Management of TLP credit

based flow control

Flash Memory Summit 2013

Santa Clara, CA

13

MR

d

Cpl

D

Cpl

D ... Cpl

D

Completer

Requester

Address Spaces

Flash Memory Summit 2013

Santa Clara, CA

14

Address Space Function

Memory Data transfer to/from memory mapped locations

- 64-bit memory address space

I/O Data transfer to/from IO-mapped locations

- 32-bit I/O space

Configuration Device Function configuration & setup

- 16-bit configuration space

Message Event signaling & general purpose messaging

Functions and Configuration Space

Function – an addressable entity in configuration

space

• Architecturally visible (i.e., can be discovered and configured)

• Capable of issuing requests and generating completions

Flash Memory Summit 2013

Santa Clara, CA

15

Bus FuncDev

8 5 3

Bus Func

8 8

Alternative Routing-ID Interpretation (ARI) IDTraditional ID

Type 0 and Type 1 Functions

Flash Memory Summit 2013

Santa Clara, CA

16

Type 0

Function

Function 0

Type 0

Function

Function 0

Type 0

Function

Function 1

Type 0

Function

Function 2

Bus Bus

Single Function

DeviceMulti-Function Device

Type 1

FunctionPCI-to-PCI

Bridge

Bus

Bus

Function Config. Space Registers

Flash Memory Summit 2013

Santa Clara, CA

17

PCI Compatible

Configuration Space

Header000h

0FFh

FFFh

PCI

Configuration

Space

PCI Express Extended

Configuration

Space

PCI Express

Extended Capability

MSI

Capability Structure

PCI Express

Capability Structure

PCI Express

Extended Capability

Advanced Error

Reporting

PCIe Switches and I/O Fan-out

Flash Memory Summit 2013

Santa Clara, CA

18

Root Complex

DP DPDP

PCIe Switch

En

dp

oin

t

UP UP

DP DPDP

PCIe Switch

UP

DP DPDP

PCIe Switch

UP

DP DPDP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

En

dp

oin

t

UP

PCIe Switch

Flash Memory Summit 2013

Santa Clara, CA

19

PCIe Switch

UP

DP DPDP Type 1

FunctionPCI-to-PCI

Bridge

Type 1

FunctionPCI-to-PCI

Bridge

Type 1

FunctionPCI-to-PCI

Bridge

Type 1

FunctionPCI-to-PCI

Bridge

PCIe Switch

Physical View

PCIe Switch Logical View

UP

DP DP DP

Virtual PCI Bus

Non-Transparent Bridge (NTB)

Flash Memory Summit 2013

Santa Clara, CA

20

Root

Complex

DP

PCIe Switch

UP

DP DPDP

En

dp

oin

t

UP

En

dp

oin

t

UP

Root

Complex

DP

PCIe Switch

UP

DPDP DP

En

dp

oin

t

UP

En

dp

oin

t

UPType 0

Function

Non-Transparent Bridge

Proprietary

Logic

Type 0

FunctionUPUP

PCIe Multi-Path Usage Model

Flash Memory Summit 2013

Santa Clara, CA

21

PCIe

SSD

PCIe

SSD

PCIe

SSD

PCIe

SSDPCIe

SSD

PCIe

SSD

PCIe Switch

PCIe

SSD

PCIe

SSD

PCIe Switch

Host Host

Inerconnect

PCIe PCIe

PCIe Overview Summary

PCIe is ubiquitous

PCIe provides a scalable interface for SSDs

• Scalable link width and speed

PCIe is not a bottleneck

• Highest performance standard I/O attach point

PCIe switches provide I/O fan-out

• Allows multiple SSDs to be connected to a Root Port

NVMe works within the standard PCIe framework

• Allows use of off-the-shelf Root Complexes and Switches

PCIe may be used to connect multiple hosts to an SSD

Flash Memory Summit 2013

Santa Clara, CA

22