Containers and Ceph Haomai 2015.06.04

Hi, I’m Haomai Wang

❖ Join Ceph community Since 2013

❖ GSOC 2015 Ceph Mentor

❖ Maintains KeyValueStore and AsyncMessenger

❖ Active in RBD, Performance, ObjectStore things

❖ Newer to Containers!

❖ haomaiwang@gmail.com

Agenda

❖ Motivation

❖ Block ? File ?

❖ CephFS Update

❖ Orchestration

❖ Summary

Cloud Hodgepodge❖ Compelling clouds offer options

❖ Compute

❖ VM (KVM, XEN …)

❖ Containers (LXC, Docker, OpenVZ)

❖ Storage

❖ Block

❖ File

❖ Object

❖ Key/Value

❖ NOSQL

❖ SQL

App Server

App Server

App Server

App Server

Containers?❖ Performance

❖ Shared Kernel❖ Fast Boot❖ Lower baseline overhead❖ Better resource sharing

❖ Storage❖ Shared Kernel -> Efficient IO❖ Small Image -> Efficient deployment

❖ Emerging container host OSs❖ CoreOS❖ Atomic❖ Snappy Ubuntu

❖ New app provisioning model❖ Small, single-service containers❖ Standalone execution environment

Ceph Components

Block/File

VM + Block(RBD)❖ Model

❖ Nova → libvirt → KVM →librbd.so

❖ Cinder → rbd.py →librbd.so

❖ Glance → rbd.py → librbd.so

❖ Pros

❖ proven

❖ decent performance good security

❖ Cons

❖ performance could be better

❖ Status

❖ most common deployment model today (~44% in latest survey)

Container + Block(RBD)❖ The model

❖ libvirt-based lxc containers(Or Docker)

❖ map kernel RBD on host

❖ pass host device to libvirt, container

❖ Pros

❖ fast and efficient

❖ implement existing Nova API Cons

❖ weaker security than VM

❖ Status

❖ lxc is maintained

Follow VM to use mature Block(RBD)?

Different App Provision Model❖ Container VS Virtualization

❖ Hardware abstraction❖ Application Centric❖ Per VM Isolation, Guest Environment and

lifecycle defined by Application ❖ Application Isolation❖ Density

❖ New Provision❖ Micro-Service❖ Multi-instance, Multi-version, Maximal

flexible, Minimal overhead❖ Block

❖ Physical block abstraction❖ Unknown user data layout❖ Difficult to bind block to container(s)

Data Aware

RADOS(File alike Interface)

Ceph Storage Layout

Block Deivc

Block Deivc

Block Deivc

Block Deivc

Block Deivc

Block Deivc

Block Deivc

Block Deivc

OSD OSD OSD OSD OSD OSD OSD OSD

RBDOBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECTOBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT

BLOCK

CEPHFS(Strict POSIX)

DirectoryDirectory DirectoryFile

File

File

File

File

File

File

BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK

Containers may like file more than block

CephFS Update

File Storage

❖ Familiar POSIX semantics(POSIX is a lingua-franca)

❖ Fully shared volume – many clients can mount and share data

❖ Elastic storage – amount of data can grow/shrink without explicit provisioning

CephFS

CephFS Architecture❖ Inherit resilience and

scalability of RADOS

❖ Multiple metadata daemons (MDS) handling dynamically shared metadata

❖ Fuse & kernel clients: POSIX compatibility

❖ Extra features: Subtree snapshots, recursive statistics

Detecting failures❖ MDS

❖ “beacon” pings to RADOS MONs. Logic on MONs decides when to mark an MDS failed and promote another daemon to take its place

❖ Clients:

❖ “RenewCaps” pings to each MDS with which it has a session. MDSs individually decide to drop a client's session (and release capabilities) if it is too late.

The Now

❖ Priority

❖ Complete FSCK & repair tools

❖ Tenant Security/Auth

❖ Other work:

❖ Multi-MDS hardening

❖ Snapshot hardening

Orchestration

Existing VM & FileNFS + CEPHFS.KO VIRTFS/9P + CEPHFS.KO

Nova-Docker & CephFS❖ Model

❖ host mounts CephFS directly❖ mount --bind share into container

namespace❖ Pros

❖ best performance❖ full CephFS semantics

❖ Cons❖ rely on container for security

❖ Status❖ no prototype

Kubernetes & CephFS❖ Pure Kubernetes❖ Volume Driver

❖ AWS EBS, Google Block❖ CephFS❖ NFS❖ …

❖ Status❖ Under review(https://github.com/

GoogleCloudPlatform/kubernetes/pull/6649)

❖ Drivers expect pre-existing volumes❖ Expected deploy mode

❖ Pod(Shared File Volume)❖ Make micro-service ease with shared storage

Kubernetes on OpenStack❖ Provision Nova VMs

❖ KVM or ironic

❖ Atomic or CoreOS

❖ Kubernetes per tenant

❖ Provision storage devices

❖ Cinder for volumes

❖ Manila for shares

❖ Kubernetes binds into pod/container

❖ Status

❖ Prototype Cinder plugin for Kubernetes (https://github.com/spothanis/kubernetes/tree/cinder-vol-plugin)

Thank You!

Thanks toSage Weil(sage@redhat.com)

John Spray(john.spray@redhat.com)