Storage as microservice (a.k.a OpenEBS)

Built using Docker

Docker Chennai Meetup

@openebs

#containerizedstorageforcontainers

3 June 2017

Let us revisit containers

Image source : https://newsroom.netapp.com/blogs/containers-vs-vms/

VM

Container

Docker -> Portability

Docker registry → DevOps bridging

Storage controllers software version control

Monolithic Storage Kernel

iSCSI Software

A

ReplicationSnapshot

EncryptionB

QoS&

MgmtC

volume1

volume2

Storage software is same for all

volumes

Should we containerize storage ?

Version control of application software is made easy

through Docker images

How about the storage software ?

When storage is containerized

Docker Hub

Storage can also be a micro service

Containerized Storage Kernel

(OpenEBS)

volume1

Storage ContainerVSM 1

SoftwareA, B, C

volume2

Storage ContainerVSM 2

SoftwareA’, B’, C

Dockerized storage

https://hub.docker.com/r/openebs/jiva/

What is OpenEBS ?

➔ OpenEBS is containerized storage for containers

➔ OpenEBS solves the persistent storage issue for containers

Legacy

storage

X

Application

Pod

★ Application pod management

automatically takes care of storage

volume management

Dedicated mode of deployment

Separate hosts

Network

Data

OpenEBS Maya

Hyperconverged mode of deployment

K8s master

K8s minions

VSM or Storage PodApplication Pod

TCMU/iSCSI

OpenEBS Maya Storage

Orchestrator

Application ( yaml )

What is Maya ?

Maya

Manifests express intent

K8S used with special care for apps requiring

persistence because they require brittle tight coupling.

Container 1 Container 2 Container 2

Hard wired connections via plug-in

stateless stateful

NAS SAN S3 NAS SAN S3

Without OpenEBS DevOps workflow is broken

Manifests express intent

stateless stateful

No changes to DevOps workflow even for containers requiring

persistence. Users manifest their intent and the storage and

storage controllers adjust automatically as needed.

OpenEBSPolicy engine

Remote Storage

Local Storage

OpenEBS Storage Hosts

OpenEBS VSMs

OpenEBS Storage ClusterEBS

APIs

Containers and underlying storage,

local on host or dedicated storage

pods OR remote S3 or EBS storage

all grouped into a storage cloud that

just works. Billing by AWS.

Manifests express intent

With OpenEBS the workflow works

DevOps operations with OpenEBS

OpenEBS got the storage ops off my back!

Because of OpenEBS we are rolling into

production.

How is OpenEBS built ?Storage containerization

Storage orchestration

Deep storage operations

Core Storage features

is built using

is integrated with

is built into (OpenEBS Maya)

is built into (OpenEBS)

Smart Capacity Management Volume Latency Monitoring and Auto-healingAdapt to varying workloads - Application Aware Storage (Zones)AWS EBS Like APIsStorage Analytics!

Enterprise HA Intelligent QoSDistributed Scaleout Smart Tiering for higher performance

OpenEBS - Word of mouth

Containerized storage for containers

Calico / Flannel owns networking for K8S. OpenEBS owns

storage.

The only open source solution built for this. CEPH is not built

to be containerized.

I run (use) OpenEBS just like I run (use) K8S (EBS).

OpenEBS got the storage ops off my back! Because of OpenEBS

we are rolling into production.

Persistent storage is still an issue today

https://clusterhq.com/2016/07/17/dockercon-ecosystem-clusterhq/

"Containers have moved

beyond ephemeral storage

to persistent storage to

hold data and protect your

applications."- 451

Research

Mesosphere Declares

‘Container 2.0,’ the Stateful

Era

Kubernetes Version 1.3

include additional

resources for supporting

stateful containers

Persistence Storage Types

https://portworx.com/stateful-enterprise-applications-docker/

Flocker

Convoy

Rex-ray

Torus

Infinit

PortWorx

OpenEBS*

Docker Plugin

K8s Plugin

Typical Use cases

Where does OpenEBS fit?

DevOps and containers are the futureWhich is already here, but unevenly distributed

Real computing needs persistenceBut - persistence for DevOps & containers is broken

Usecase : Reuse existing SAN in

your cloud to build container friendly storage

platform

Build container friendly storage platform

AWS

VMs

EBSDocker

Application Pod (Application has

container-native storage)

● The existing SAN storage (EBS or GCP PD or

any other block storage) in your cloud can be

integrated into K8s application pods by

OpenEBS

● Persistent storage need not be managed

separately in your cloud

● Your persistent storage is managed from within

your application yaml files

Usecase : Build AWS EBS or GCP PD type platform On-Premise

Deploy EBS style platform On-Premise (Dedicated)

Your data center

Bare metal servers

Docker engines (Compute)

Bare metal storage

servers

Docker engines (OpenEBS)

Application

pod

yaml

A single application yaml

file controls both the

compute and storage

needs

Deploy EBS style platform On-Premise (Hyperconverged)

Your data center

Bare metal servers

Docker engines (Compute+OpenEBS)

Application

pod

yaml

A single application yaml

file controls both the

compute and storage

needs

Comparing AWS EBS & OpenEBS

Feature AWS EBS OpenEBS

Network Block Storage as a

service

Yes No. This is a technology

that can be used to build a

service or platform like AWS

EBS

Technology form factor Unknown Open Source

High Availability grade Five nines of availability Five nines of availability

Availability zones Yes Yes

Point in time Snapshots to

S3

Yes Yes, to S3 or any S3

equivalent (locally hosted

Minio for example)

Connectivity in AWS EBS (Optimized for Cloud service)

Login into AWS Console

Create an EBS volume

Attach to an instance

Format and put it to use

by application

Connectivity in OpenEBS (Optimized for Container service)

Login into Kubernetes

Set up an application

yaml with OpenEBS as

the storage provider

Volume is created on

OpenEBS and mounted

on K8S minion

Format and put it to use

by application

kubectl create -f demo-mysql-iscsi.yaml

Snapshots on AWS EBS vs OpenEBS

Incremental

Snapshot

Restore

Amazon

Cloud

Enterprise

data center

Incremental

Snapshot

Restore

Amazon

CloudAny other S3 Cloud

storage provider

S3 storage

in the data center

Scality

Minio

OpenEBS

volumes

Apps in

Containers

Incremental

Snapshot

Restore

Incremental

Snapshot

Restore

Incremental

Snapshot

Restore

Other features comparison of AWS EBS & OpenEBS

Feature AWS EBS OpenEBS

Volume migration across tiers

(Magnetic, General Purpose

SSD, Provisioned IOPS)

Yes Yes, VSMs can be migrated

seamlessly across tiers.

QoS control Yes (Provisioned IOPS) Yes, granular control of IOPS

on a per volume basis

Burst IOPS, credit bucket for

unused IOPS

Yes Yes. Min/Max IOPS, burst

IOPS, credit bucket

functionality

Auto scaling of the storage

nodes

Yes (Not sure. Internal to EBS

architecture)

Yes. OpenEBS storage nodes

are added to clusters to allow

auto scaling of either capacity

or performance

OpenEBS Roadmap

0.1

0.4

1.0

0.3

0.2

● Soft launch / Basic version

● Containerized controller

● Longhorn integration basics

0.2 - Basic k8s integration

0.3 - Full k8s integration(Hyper-Converged)

0.4 - Tiering and distributed storage

1.0 - First enterprise edition(Full functionality for basic production usage)

K8s Provisioning via EBS-like driver

S3 building blocks

Complete longhorn integration

First usable release for community

consumption

Complete K8s integration

Fully hyper-converged

Support for local backing store

Building blocks of Q0S

Building blocks of Tiering

Tiering and QoS demonstrated

Building blocks of ML for storage analytics

DevOps operator use cases demonstrated

Tiering and QoS demonstrated

Building blocks of ML for storage analytics

DevOps operator use cases demonstrated

Nov, 2017

Sep, 2017

Jun, 2017

Mar, 2017

Jan, 2017

Take a peek at our github project and star us if you like

https://github.com/login/oauth/autho

rize?client_id=768d7617e686c9c2b

d81&scope=user%20public_repo&s

tate=openebs1234cloudbyte1234&

allow_signup=true

github.com/openebs

What do you think?

https://github.com/openebs

Join the community

#slackSlack.openebs.io

blog.openebs.io

@openebs