2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 1
liblightnvmThe Open-Channel SSD User-Space Library
Simon A. F. LundCNEX Labs
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 2
Open-Channel SSD
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 3
Open-Channel SSD
• Media• Controller
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 4
Open-Channel SSD
• Media• Controller
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 5
Open-Channel SSD: Drive Model
Logical Block Chunk Parallel Unit Group of Parallel Units
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 6
Open-Channel SSD: Addressing
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 7
Open-Channel SSD: Chunk
Contains Minimal Addressable UnitsEach unit has size in bytes e.g. 4096Nomenclature: logical block, sector, address
Addresses within a chunk are contiguousE.g. Address range [0, naddrs-1]
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 8
Open-Channel SSD: Chunk IO Constraints
1. Erase before write2. Write contiguously3. Write WS_MIN multiple # of addresses pr. cmd4. Read no further than address WP – MWC
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 9
Open-Channel SSD: Chunk IO Constraints
1. Erase before write2. Write contiguously3. Write WS_MIN multiple # of addresses pr. cmd4. Read no further than address WP – MWC
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 10
liblightnvm: Related
nvme-clihttps://github.com/linux-nvme/nvme-cli
SPDKhttps://github.com/spdk/spdk
libnvmehttps://github.com/hgst/libnvme
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 11
liblightnvm: overview
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 12
liblightnvm: Usages
Pure User-Space driven IODedicated application-integration
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 13
liblightnvm: Usages
Pure User-Space driven IODedicated application-integration
Hybrid Kernel and User-SpacePBLK + User-Space for application IO
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 14
liblightnvm: Usages
Pure User-Space driven IODedicated application-integration
Hybrid Kernel and User-SpacePBLK + User-Space for application IO
Tooling for Open-Channel SSD kernel servicesFile-system and FTL management and
maintenance
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 15
liblightnvm: OCSSD Device Attributes
nvm_cmd_idfy struct nvm_devnvm_dev_openf(ident, flags)nvm_dev_get_wsopt(dev)nvm_dev_get_geo(dev)
struct nvm_geo
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 16
liblightnvm: OCSSD Device Attributes
DEMO: nvm_cmd_idfyhttps://asciinema.org/a/WJJMxRKsgAq0GIbWlfhIAGZDl
DEMO: nvm_dev and nvm_geohttps://asciinema.org/a/DCr9ak5VdnClpJjjvxjKQQNIg
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 17
liblightnvm: OCSSD Media State
nvm_cmd_rprtRetrieve chunk descriptors for all chunksRetrieve chunk descriptors for all chunks in a
parallel unit
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 18
liblightnvm: OCSSD Media State
nvm_cmd_rprtRetrieve chunk descriptors for all chunksRetrieve chunk descriptors for all chunks in a
parallel unit nvm_cmd_rprt_arbsProvides N arbitrary chunk addresses in the
requested state in distinct parallel units
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 19
liblightnvm: OCSSD Media State
DEMOhttps://asciinema.org/a/XGppr2Yjdc90fsoyLCPVCx0sc
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 20
liblightnvm: OCSSD Addressing
struct nvm_addrGeometric accessorsAddress translation is handled by the libraryUser does not need to know about the LBAF
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 21
liblightnvm: OCSSD Addressing
struct nvm_addrGeometric accessorsAddress translation is handled by the libraryUser does not need to know about the LBAF
nvm_dev_gen2dev nvm_dev_dev2gen
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 22
liblightnvm: OCSSD Addressing
DEMOhttps://asciinema.org/a/tFwlWRMq0DwwvK5oq5bCuBpty
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 23
liblightnvm: OCSSD IO Commands
nvm_cmd_erase – Vector Reset / DSM deallocate nvm_cmd_write – Vector / Scalar Write nvm_cmd_read – Vector / Scalar Read nvm_cmd_copy – Vector Copy
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 24
liblightnvm: OCSSD IO Commands
DEMOhttps://asciinema.org/a/iq8hoPAYpXSqY5Jgq67SrbA1Q
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 25
liblightnvm: OCSSD IO Command Options
IO Addressing Mode (SCALAR or VECTOR)
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 26
liblightnvm: OCSSD IO Command Options
IO Addressing Mode (SCALAR or VECTOR) NVM_CMD_SCALARerase / write / read mapped to NVMe spec.
defined opcodes
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 27
liblightnvm: OCSSD IO Command Options
IO Addressing Mode (SCALAR or VECTOR) NVM_CMD_SCALARerase / write / read mapped to NVMe spec.
defined opcodes NVM_CMD_VECTORerase / write / read mapped to OCSSD spec.
defined VECTOR opcodes
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 28
liblightnvm: OCSSD IO Command Options
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 29
liblightnvm: OCSSD IO Command Options
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 30
liblightnvm: OCSSD IO Command Options
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 31
liblightnvm: OCSSD IO Command Options
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 32
liblightnvm: OCSSD IO Command Options
IO Execution Mode (SYNC or ASYNC)
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 33
liblightnvm: OCSSD IO Command Options
IO Execution Mode (SYNC or ASYNC) NVM_CMD_SYNCSubmits and blocks until completion
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 34
liblightnvm: OCSSD IO Command Options
IO Execution Mode (SYNC or ASYNC) NVM_CMD_SYNCSubmits and blocks until completion
NVM_CMD_ASYNCReturns after submissionCallback function called upon completion
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 35
liblightnvm: Striping
HORZVERT
chunk0 chunk1 chunk2 chunk3 chunk0 chunk1 chunk2 chunk3
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 36
liblightnvm: Striping
HORZVERT
chunk0 chunk1 chunk2 chunk3 chunk0 chunk1 chunk2 chunk3
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 37
liblightnvm: Striping Caveat
HORZ
chunk0 chunk1 chunk2 chunk3
Constraints amplified Write-cache increase MWC x k
Optimal write-sizeWS_OPT x k
Minimal write is intact
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 38
liblightnvm: OCSSD ASYNC IO Example
DEMOhttps://asciinema.org/a/8bo7Ma0DWqqZaMQRelGWDNTaf
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 39
liblightnvm: Abstractions
Reduce the cognitive load on the OCSSD user
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 40
liblightnvm: Abstractions
Reduce the cognitive load on the OCSSD user Provide traditional IO semanticswrite(fd, *buf, count) / read(fd, *buf, count)pread(fd, *buf, count, offset)
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 41
liblightnvm: Abstractions
Reduce the cognitive load on the OCSSD user Provide traditional IO semanticswrite(fd, *buf, count) / read(fd, *buf, count)pread(fd, *buf, count, offset)
Use them when you need themPeel them off and take control when you don’t
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 42
liblightnvm: Virtual Block
Encapsulates IO to a disjoint set of k chunks Dynamic / Runtime control of parallel unitsUser provisioned set of chunks
HORZ striping on WS_OPT for throughput
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 43
liblightnvm: Virtual Block
Traditional IO Semanticsnvm_vblk_write(*vblk, *buf, count) nvm_vblk_read(*vblk, *buf, count) nvm_vblk_pread(*vblk, *buf, count, offset)
Agnostic to media and spec. variation
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 44
liblightnvm: Virtual Block
DEMOhttps://asciinema.org/a/HnPa9smu8W6HoeyaqC6DavBeo
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 45
liblightnvm: Append-Only Streams
Encapsulates IO to a disjoint set of k chunks Dynamic / Runtime control of parallel unitsLibrary provisioned set of chunksProvisioning strategy e.g. HORZ or VERT
HORZ striping on WS_OPT for throughput Stream states are persistent!
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 46
liblightnvm: Append-Only Streams
Traditional IO Semanticsaos_write(sid, *buf, count)aos_read(sid, *buf, count)aos_pread(sid, *buf, count, read)
Agnostic to media and spec. variation Encapsulates geometry and addressing
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 47
liblightnvm: Append-Only Streams
DEMO https://asciinema.org/a/ljb7fhetCKmRCd79G8cbYpaic
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 48
liblightnvm: Summary
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 49
Roadmap: Persistent CMB interface
Rawnvm_cmb_writenvm_cmd_read
IO orientednvm_cmb_io_writenvm_cmb_io_readnvm_cmd_io_push
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 50
Roadmap: Spec. support
Expand support in the evolving spec. spaceDenali / OCSSD 2.1 / NVMeRaw support via nvm_cmd_*Encapsulation in upper-level abstractions
Virtual Block and Append-Only interfaces
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 51
Roadmap: Related tools
nvm_uiWeb interface for management of PBLK
instances, NVMoF targets, subsystems and ports
Visualization of IO stats. in real-time CIJOEToolchain for QA, test, and development
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 52
Roadmap: Collaboration
What are you missing from liblightnvm?
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 53
Roadmap: Collaboration
What are you missing from liblightnvm? Regarding SGL support, would you prefer …An array of buffersA list of SGL segments Iterator / function-pointerSomething else? All of them?
2018 Storage Developer Conference. © CNEX Labs, Inc. All Rights Reserved. 54
Thanks
SRC http://github.com/OpenChannelSSD/liblightnvmDOC http://lightnvm.io/liblightnvmMAIL [email protected]
www.linkedin.com/in/simonlund