libradosAdventure! Excitement! Octopus-wrangling!

Jesse Williamson, SUSE (“jwilliamson@suse.de”)

2

Disclaimer:

The views expressed here are my own and do not necessarily reflect those of my employer, SUSE, nor of the Ceph authors and contributors.

(Whew!)

3

● RADOS: What the Heck Is It?

...in which I try to explain a programmingtool while showing very little code.

4

● RADOS: What the Heck Is It?

● Reliable, Autonomic Distributed Object Store“RADOS: A Scalable Reliable Storage Service for Petabyte-scale Storage Clusters”. Weil, Leung, et. al., 2007.

● “...[provides] applications with the illusion of a single logical object store with well-defined safety semantics and strong consistency guarantees.”

● Distributes data and workload across dynamic storage cluster

● “...[leverages] device intelligence to distribute the complexity surrounding consistent data access, redundant storage, failure detection, and failure recovery in clusters consisting of many thousands of storage devices.”

5

● RADOS: What the Heck Is It?

● Reliable, Autonomic Distributed Object Store“RADOS: A Scalable Reliable Storage Service for Petabyte-scale Storage Clusters”. Weil, Leung, et. al., 2007.

● “...[provides] applications with the illusion of a single logical object store with well-defined safety semantics and ● strong consistency guarantees.”

● Distributes data and workload across dynamic storage cluster

● “...[leverages] device intelligence to distribute the complexity surrounding consistent data access, redundant storage, failure detection, and failure recovery in clusters consisting of many thousands of storage devices.”

● Integral to Ceph!

● cluster management, device state, etc. (CRUSH maps)

● data distribution (“placement”) policies

● “smart” propagation of rules; data replication, consistency, etc.

6

● RADOS: What the Heck Is It?

● RADOS “the idea”● papers

● RADOS implementation “the thing”: ● Ceph:

- mons, OSDs, storage “smarts”

● Interfaces to the implementation:

● Command-line interface:- the “rados” tool

● Service interface:- RGW (RADOS Gateway); filesystems, ...

7

● RADOS: What the Heck Is It?

● RADOS “the idea”● papers

● RADOS implementation “the thing”: ● Ceph:

- mons, OSDs, storage “smarts”

● Interfaces to the implementation:

● Command-line interface:- the “rados” tool

● Service interface:- RGW (RADOS Gateway); filesystems, ...

● Programmatic interface:- librados

8

● librados:

● What is it, and what does it do?

● What uses it?

● What can you make with it?

9

● librados: ● What is it?

Librados is a programming-language level interface to Ceph.

Bindings available at-least for (varying levels of “official” standing):

C++, C, Erlang, Go, PHP, Java, Python, Ruby, Haskell

10

● librados: ● What is it?

Librados is a programming-language level interface to Ceph.

Bindings available at-least for (varying levels of “official” standing):

C++, C, Erlang, Go, PHP, Java, Python, Ruby, Haskell

● Lower (librados.h) and higher-level (librados.hpp, RadosClient)behaviors

11

● librados: ● What is it?

Librados is a programming-language level interface to Ceph.

Bindings available at-least for (varying levels of “official” standing):

C++, C, Erlang, Go, PHP, Java, Python, Ruby, Haskell

● Lower (librados.h) and higher-level (librados.hpp, RadosClient)behaviors

● ...librados is “batteries included”-- just link in!

12

● librados: ● What is it?

Librados is a programming-language level interface to Ceph.

Bindings available at-least for (varying levels of “official” standing):

C++, C, Erlang, Go, PHP, Java, Python, Ruby, Haskell

● Lower (librados.h) and higher-level (librados.hpp, RadosClient)interfaces

● ...librados is “batteries included”-- just link in!!

● librados is a platform you can build your application on top of--a client interface to Ceph.

13

● librados: ● What is it?

Librados is a programming-language level interface to Ceph.

Bindings available at-least for (varying levels of “official” standing):

C++, C, Erlang, Go, PHP, Java, Python, Ruby, Haskell

● Lower (librados.h) and higher-level (librados.hpp, RadosClient)interfaces

● ...librados is “batteries included”-- just link in!

● librados is a platform you can build your application on top of--a client interface to Ceph

● Hides complexity: networking, replication, scaling, dealingwith filesystems...

14

● librados: ● What is it?

librados (C++, C, Python, ...)

RADOS (Ceph infrastructure-- distributed object store)

RGW(Web services)

CephFS(filesystem) Your application! :-)

15

● librados: ● What is it?

import rados, sys

cluster = rados.Rados(conffile='ceph.conf')

cluster.connect()print "Cluster ID: " + cluster.get_fsid()

cluster_stats = cluster.get_cluster_stats()

for key, value in cluster_stats.iteritems(): print key, value

ioctx = cluster.open_ioctx('data')

ioctx.write_full("hi_there", "Hello World!")

ioctx.close()

16

● librados: ● What uses it? What can you make with it?

17

● librados: ● What uses it? What can you make with it?

Ceph components and tools:RADOS gateway (rgw);“rados” command-line tool;rdb-mirrorlibradosstriperlibrdb...among others...

18

● librados: ● What uses it? What can you make with it?

Ceph components and tools:RADOS gateway (rgw);“rados” command-line tool;rdb-mirrorlibradosstriperlibrdb...among others…

Potential project using librados to provide a backend for the Dovecot mail system:http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/24092

19

● librados: ● What uses it? What can you make with it?

Ceph components and tools:RADOS gateway (rgw);“rados” command-line tool;rdb-mirrorlibradosstriperlibrdb...among others…

Potential project using librados to provide a backend for the Dovecot mail system:http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/24092

RadosFS: A Ceph RADOS API for Hadoop (“https://github.com/rootfs/RadosFs”)Zlog: a distributed shared log for Ceph (“https://github.com/noahdesu/zlog”)Apache and nginx modulesDocker and Vagrant goodies

...many more examples on github.

20

● librados: ● What uses it? What can you make with it?

Ceph components and tools:RADOS gateway (rgw);“rados” command-line tool;rdb-mirrorlibradosstriperlibrdb...among others…

Potential project using librados to provide a backend for the Dovecot mail system:http://permalink.gmane.org/gmane.comp.file-systems.ceph.devel/24092

Librados is a powerful Ceph interface-- build anything you can imagine! :-)

21

● librados: ● What does it do?

Some major functional areas:

Configuration: handle environment variables, config files, command-line options in a consistent way;

22

● librados: ● What does it do?

Some major functional areas:

Configuration: handle environment variables, config files, command-line options in a consistent way;

Connections: abstract away networking, connection management, some threading;

Pools: manage namespaces, iterate over objects in them

23

● librados: ● What does it do?

Some major functional areas:

Configuration: handle environment variables, config files, command-line options in a consistent way;

Connections: abstract away networking, connection management, some threading;

Pools: manage namespaces, iterate over objects in them

I/O: Synchronous and Asynchronous read/write/update on objects

24

● librados: ● What does it do?

Some major functional areas:

Configuration: handle environment variables, config files, command-line options in a consistent way;

Connections: abstract away networking, connection management, some threading;

Pools: manage namespaces, iterate over objects in them

I/O: Synchronous and Asynchronous read/write/update on objects

More:Extended attributes, omaps, client watch/notify events, osd commands, statistics, snapshots, ...all “baked in”.

25

● librados:

Resources:

Ceph documentation (C, C++, Python, Java examples):http://docs.ceph.com/docs/master/rados/api/

Other Bindings (Erlang, Go, Haskell, PHP, Ruby-- many more (Common Lisp, anyone?)):https://github.com/renzhi/erlradoshttps://github.com/mrkvm/rados.gohttps://hackage.haskell.org/package/rados-haskellhttps://github.com/ceph/ceph-rubyhttps://github.com/ceph/phprados

Source code:https://github.com/ceph/ceph/tree/master/src/include/rados

A more in-depth presentation from Sage Weil:http://events.linuxfoundation.org/sites/events/files/slides/20150311%20vault15%20librados.pdf

Further reading:Weil, Leung, et. al.. “RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters”.Weil, Brandt, et. al.. “CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data”.

Questions? :-)