Date post: | 28-Jul-2015 |
Category: |
Technology |
Upload: | ceph-community |
View: | 141 times |
Download: | 6 times |
Containers and Ceph Haomai 2015.06.04
Hi, I’m Haomai Wang
❖ Join Ceph community Since 2013
❖ GSOC 2015 Ceph Mentor
❖ Maintains KeyValueStore and AsyncMessenger
❖ Active in RBD, Performance, ObjectStore things
❖ Newer to Containers!
Agenda
❖ Motivation
❖ Block ? File ?
❖ CephFS Update
❖ Orchestration
❖ Summary
Cloud Hodgepodge❖ Compelling clouds offer options
❖ Compute
❖ VM (KVM, XEN …)
❖ Containers (LXC, Docker, OpenVZ)
❖ Storage
❖ Block
❖ File
❖ Object
❖ Key/Value
❖ NOSQL
❖ SQL
App Server
App Server
App Server
App Server
Containers?❖ Performance
❖ Shared Kernel❖ Fast Boot❖ Lower baseline overhead❖ Better resource sharing
❖ Storage❖ Shared Kernel -> Efficient IO❖ Small Image -> Efficient deployment
❖ Emerging container host OSs❖ CoreOS❖ Atomic❖ Snappy Ubuntu
❖ New app provisioning model❖ Small, single-service containers❖ Standalone execution environment
Ceph Components
Block/File
VM + Block(RBD)❖ Model
❖ Nova → libvirt → KVM →librbd.so
❖ Cinder → rbd.py →librbd.so
❖ Glance → rbd.py → librbd.so
❖ Pros
❖ proven
❖ decent performance good security
❖ Cons
❖ performance could be better
❖ Status
❖ most common deployment model today (~44% in latest survey)
Container + Block(RBD)❖ The model
❖ libvirt-based lxc containers(Or Docker)
❖ map kernel RBD on host
❖ pass host device to libvirt, container
❖ Pros
❖ fast and efficient
❖ implement existing Nova API Cons
❖ weaker security than VM
❖ Status
❖ lxc is maintained
Follow VM to use mature Block(RBD)?
Different App Provision Model❖ Container VS Virtualization
❖ Hardware abstraction❖ Application Centric❖ Per VM Isolation, Guest Environment and
lifecycle defined by Application ❖ Application Isolation❖ Density
❖ New Provision❖ Micro-Service❖ Multi-instance, Multi-version, Maximal
flexible, Minimal overhead❖ Block
❖ Physical block abstraction❖ Unknown user data layout❖ Difficult to bind block to container(s)
Data Aware
RADOS(File alike Interface)
Ceph Storage Layout
Block Deivc
Block Deivc
Block Deivc
Block Deivc
Block Deivc
Block Deivc
Block Deivc
Block Deivc
OSD OSD OSD OSD OSD OSD OSD OSD
RBDOBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECTOBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT OBJECT
BLOCK
CEPHFS(Strict POSIX)
DirectoryDirectory DirectoryFile
File
File
File
File
File
File
BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK BLOCK
Containers may like file more than block
CephFS Update
File Storage
❖ Familiar POSIX semantics(POSIX is a lingua-franca)
❖ Fully shared volume – many clients can mount and share data
❖ Elastic storage – amount of data can grow/shrink without explicit provisioning
CephFS
CephFS Architecture❖ Inherit resilience and
scalability of RADOS
❖ Multiple metadata daemons (MDS) handling dynamically shared metadata
❖ Fuse & kernel clients: POSIX compatibility
❖ Extra features: Subtree snapshots, recursive statistics
Detecting failures❖ MDS
❖ “beacon” pings to RADOS MONs. Logic on MONs decides when to mark an MDS failed and promote another daemon to take its place
❖ Clients:
❖ “RenewCaps” pings to each MDS with which it has a session. MDSs individually decide to drop a client's session (and release capabilities) if it is too late.
The Now
❖ Priority
❖ Complete FSCK & repair tools
❖ Tenant Security/Auth
❖ Other work:
❖ Multi-MDS hardening
❖ Snapshot hardening
Orchestration
Existing VM & FileNFS + CEPHFS.KO VIRTFS/9P + CEPHFS.KO
Nova-Docker & CephFS❖ Model
❖ host mounts CephFS directly❖ mount --bind share into container
namespace❖ Pros
❖ best performance❖ full CephFS semantics
❖ Cons❖ rely on container for security
❖ Status❖ no prototype
Kubernetes & CephFS❖ Pure Kubernetes❖ Volume Driver
❖ AWS EBS, Google Block❖ CephFS❖ NFS❖ …
❖ Status❖ Under review(https://github.com/
GoogleCloudPlatform/kubernetes/pull/6649)
❖ Drivers expect pre-existing volumes❖ Expected deploy mode
❖ Pod(Shared File Volume)❖ Make micro-service ease with shared storage
Kubernetes on OpenStack❖ Provision Nova VMs
❖ KVM or ironic
❖ Atomic or CoreOS
❖ Kubernetes per tenant
❖ Provision storage devices
❖ Cinder for volumes
❖ Manila for shares
❖ Kubernetes binds into pod/container
❖ Status
❖ Prototype Cinder plugin for Kubernetes (https://github.com/spothanis/kubernetes/tree/cinder-vol-plugin)