1

Agenda• SCSI Targets• STGT and Need of LIO• LIO Introduction• LIO Architecture• LIO Directory structure• Linux Storage Stack and LIO• LIO Configfs Interface• LIO Configfs tree• LIO Command Line interface• Use cases• LIO users and partners

2

SCSI targets

• The subsystem enables a computer node to behave as a SCSI storage device, responding to storage requests by other SCSI initiator nodes.

• Opens up the possibility of creating custom SCSI devices and putting intelligence behind the storage.

• Extra functionality can be added on top of it – RAID, snapshots, replication, HA, backup, data deduplication.

• Some existing SCSI target implementations – SCST, IET, STGT

3

STGT and Need of LIO

• Many major distributions included STGT support for userspace iSCSI target mode

• But is still unsupported in many commercial cases

• Major distributors wanted to see proper in-kernel iSCSI target mode support and the community offered out of tree modules for those interested

• Many developers, hardware vendors and users wanted upstream kernel level iSCSI target mode

4

LIO Introduction

• TCM/LIO = Target core module / Linux- IO

• Open source Linux SCSI target

• Developed by Datera Inc. (Rising tide systems)

• Merged in Linux Kernel on March, 2011

• Requirements

o Linux Kernel version >= 2.6.38

• Supported by almost all Linux distributions by default

5

LIO Introduction…Contd

• Supports SCSI Advanced featureso Persistent Reservations (SPC3/SPC4)o ALUA (Asymmetric Logical Unit Assignment)

Both explicit and implicit

o T10 Data Integrity Formato VAAI (vStorage APIs for Array Integration)o Multiple connection per session (MC/S)o iSCSI ERLs (Error Recovery Levels)

ERL 0 – Session Recovery ERL 1 – Digest failure RecoveryERL 2 – Connection Recovery

6

LIO Architecture

7

LIO Architecture…Contd

LIO consists of three group of modules :• Fabric modules

o Frontend of LIO, encapsulates supported storage protocols o Supports iSCSI, FC, FCOE, SRP, infiniband, vHost

• Storage modules / backstoreso Backend of LIO – provide access to imported devices via device driverso Supports –

Block ( HDDs, SSDs, LVM, Raids ) File RAMdisk

• Core Functionality (target_core_mod.ko)o Manages resources and memoryo Implements Persistent resvervation, ALUA and other core features.

8

Linux Storage Stack and LIO

9

VFS (Virtual File System)Memory Management

ext3 ext4 others

Block Layer and Device Mapper

SCSI Layer

target

/dev/sdb

LIO core modules

FC iSCSI SRP

Block Fileio Raw

/dev/mapper/xxx /dev/vg/lv ext4file

Logical Volume Manager

Low Level drivers

user

Kernel

Storage

LIO directory Structure

10

Storage and core modules

Core data structures

declarations

LIO dirstructure

LIO Configfs Interface

• Configfs is a RAM-based virtual file system similar to sysfs (but not same)

• Precisely, it is inverse of sysfs.• Provide better control from userland• Unlike sysfs, configfs allows to create, manage

and destroy kernel objects from user-space• Typically mounted at /config or /sys/kernel/config• LIO uses it for management and configuration of

storage and fabric modules.

11

LIO Configfs tree

12

LIO Command Line Interface• lio_node and tcm_node

o Low level utility to manage storage and fabric configfs nodeso Python based cli for developers, intergrators and advanced userso Different tcm utilities available for fabric modules

• lio-dump, tcm-dumpo Save configuration

• target-clio Provide unified single node SAN configuration shello Fabric agnostico Exports functionality via RTSlib library and API

python-rtslib (python API library over configfs)

13

Use Cases

• HA clustering• Openstack Cinder apis written to consume LIO

iSCSI storage• IO virtualization(IOV) with KVM on IOV capable

hardware• iSCSI LAN and WAN replication – DRBD and

pacemaker support• iSCSI boot using PXE• Implementing advanced features on top of LIO –

dedupe, thin provisioning, etc

14

LIO users and partners

15

References• RFC 3720• LIO wiki page• SCST vs LIO• iscsi targets wiki page• LIO-utils – github• Configfs• Targetcli and rtslib• Cinder plugin for LIO• Open iSCSI utility• STGT project• Distributed replicated block device

16

Questions ?

17

Author Biography

18

Kashish Bhatia – Member of Technical Staff, VMware

• Technology enthusiast, with most of the experience in storage domain• Has exposure to other Linux subsystems like filesystems, block layer, Selinux• Experience of working on distributed file system (Lustre) and storage virtualization• Interests : open source, linux kernel hacking• Published technical paper in Openstack Summit Nov 2013• Bachelor of Engineering, Computer Science, Pune University• To know more about Kashish connect on LinkedIn• Follow me on twitter