Ceph Day SF 2015 - Big Data Applications and Tuning in Ceph

transcript

Big Data Applications and Tuning in Ceph

Noah WatkinsRed Hat

Who am I?

● Noah Watkins● Red Hat engineer

○ Big Data applications on Ceph● PhD candidate at UC Santa Cruz

○ Use Ceph as a research platform● Contact

○ noahwatkins@gmail.com

Today’s Agenda

● Non-technical talk about technical stuff● Less visible projects that deserve attention● Ceph is a big ecosystem

○ Running Hadoop on Ceph○ Tracing and debugging features○ Custom object interfaces

Big Data with Hadoop and Ceph

Big Data with Ceph and Hadoop

● Do you Hadoop?

● Do you Hadoop?● Are you running a Ceph cluster?

● Do you Hadoop?● Are you running a Ceph cluster?● Combined, they work. End of talk.

National System

Administrator Appreciation

Why should you care? Consolidation

FooStore!

FooApp!

FooStore!

FooApp!

FooStore!

FooApp!

FooStore!

FooApp!

FooStore!

FooApp!

FooStore!

FooApp! FooApp!

How does it work?

● A shim layer translates file system APIs○ CephFS <-> Hadoop Common File System

● Opens up the entire Hadoop ecosystem○ MapReduce○ Spark○ Storm○ Impala○ HBase○ The list goes on and on

HDFS vs CephFS, 1TB Terasort

17http://www.mellanox.com/related-docs/whitepapers/wp_hadoop_on_cephfs.pdf

1 Year Old Results!

What Works and What Doesn’t (yet)

● Locality-aware scheduling○ The rumors aren’t true :)

● Variable replication and erasure coding○ Select from existing pools

● Snapshots● How’s the stability

○ Terasort, HBase, DFSIO○ Bug fixes and performance tuning (HDFS isn’t strict!)○ Gets better with each new MDS update 18

Test driving Hadoop on Ceph

● Github○ https://github.com/ceph/cephfs-hadoop

● Tutorial○ http://ceph.com/docs/master/cephfs/hadoop/○ Streamlined installation and new docs coming soon!

● Mailing list○ Best resource right now○ http://tracker.ceph.com

Tracing and Debugging with LTTng

Things always go according to plan

FooApp!

Things always go according to plan

FooApp!

Complex Systems Can’t Be Grokked

FooApp!

Complex Systems Can’t Be Grokked

FooApp!

What is tracing & why should I care?● Tracing allows us to see exactly what

happens inside the system[11:41:53.226668003] (+0.000270968) issdm-45 librbd:aio_read_enter: { cpu_id = 0 }, { pthread_id = 140267464296512 }, { imagectx = 0x7F929308F600, name = "kubuntu", snap_name = "", read_only = 0, offset = 2078900224, length = 31232, completion = 0x7F92935AE190 }

[11:41:53.226730019] (+0.000062016) issdm-45 librbd:aio_read_exit: { cpu_id = 0 }, { pthread_id = 140267464296512 }, { retval = 31232 }

[11:41:53.228001617] (+0.001271598) issdm-45 librbd:aio_complete_enter: { cpu_id = 1 }, { pthread_id = 140266098906880 }, { completion = 0x7F92935AE190, rval = 31232 }

[11:41:53.228007204] (+0.000005587) issdm-45 librbd:aio_get_return_value_enter:{ cpu_id = 1 }, { pthread_id = 140266098906880 }, { completion = 0x7F92935AE190 }

[11:41:53.228009718] (+0.000002514) issdm-45 librbd:aio_get_return_value_exit: { cpu_id = 1 }, { pthread_id = 140266098906880 }, { retval = 31232 }

[11:41:53.228016702] (+0.000006984) issdm-45 librbd:aio_complete_exit: { cpu_id = 1 }, { pthread_id = 140266098906880 }, { } 25

With tracing anything is possible

Queue Depth over TimeLatency vs Sector Size

Example: Ceph Request Latency

Trace processing pipeline

● Processing step examines trace events● Typically written in Python● Looking for pairs is a common pattern

○ Time spent in queue○ Time spent in I/O○ Client processing time

● Requires knowledge of internal workings

Zipkin, Blkin, and LTTng

● Dapper is a Google system○ Traces causal

relationships● Zipkin implemented by Twitter

○ Look at the pretty GUI○ Ignores data sources

● Huge number of raw LTTng tracepoints in Ceph○ LTTng → Zipkin (Blkin)

■ Marios Kogias○ Andrew Shewmaker○ Adam Crume

Getting started with tracing!

● Lots of tracepoints exist!○ Adding new points is easy :)

● RBD-Replay○ Collect and replay RBD traces○ http://ceph.com/docs/master/rbd/rbd-replay/

● Adding points and discussionhttp://noahdesu.github.io/2014/06/01/tracing-ceph-with-

lttng-ust.html 34

TRACEPOINT_EVENT(librados, rados_write_enter,

TP_ARGS(

rados_ioctx_t, ioctx, const char*, oid,

const void*, buf, size_t, len, uint64_t, off),

TP_FIELDS(

ctf_integer_hex(rados_ioctx_t, ioctx, ioctx)

ctf_string(oid, oid)

[Scripting] Storage and Compute with RADOS

A different version of a better talk

● The objects in RADOS can have arbitrary code associated with them○ Think: “remotely compress object “foo”, please.”

● "Distributed Storage and Compute with Ceph’s librados”○ Great talk by Sage Weil○ Check out YouTube

● Scripting compute with librados36

How does an OSD handle a request?

Client OSDlibradosread-object(foo)

read-object

transaction● Client reads an object from an OSD● The OSD executes a “read” operation

○ The read operation knows how to access data managed by the OSD

● All operations are executed in a transactional context

● Exact function can be swapped out

What are RADOS object classes?

Client OSDlibradosget-md5(foo)

transaction● Client writes some C++ code● Compiles it into an OSD plugin● After installing this code can be invoked● Avoids data transfer

○ Can cache results● Can simplify application design

MD5-Hash

out-of-band install

Example RADOS object class pluginint compute_md5(cls_method_context_t hctx, bufferlist *in, bufferlist *out){ size_t size; int ret = cls_cxx_stat(hctx, &size, NULL); if (ret < 0) return ret;

bufferlist data; ret = cls_cxx_read(hctx, 0, size, data); if (ret < 0) return ret;

byte digest[AES::BLOCKSIZE]; MD5().CalculateDigest(digest, (byte*)data.c_str(), data.length());

out->append(digest, sizeof(digest)); return 0;}

Input provided by client, and output returned to client.

Stat the object to query its size.

Read the entire object into a buffer.

Pass this data buffer to the MD5 algorithm

Return the MD5 digest to the client.

All in a transactional context.

Dynamic object classes with Lua

Client OSDlibradoscall-lua(script, foo)

transaction● Lua is great as an embedded language● LuaJIT is a high-performance

implementation● Allow clients to construct and modify

object classes without compiling or restarting OSDs

LuaJIT VM

dynamically generated interface

Example: Lua Thumbnail Generator function thumb(input, output)

-- apply thumbnail spec to original image

local spec_string = input:str()

local blob = get_orig_img()

local img = assert(magick.load_image_from_blob(blob:str()))

img = magick.thumb(img, spec_string)

-- append thumbnail to object

local obj_size = cls.stat()

local img_bl = bufferlist.new()

img_bl:append(img)

cls.write(obj_size, #img_bl, img_bl)

-- save location in leveldb

local loc_spec = #img_bl .. "@" .. obj_size

local loc_spec_bl = bufferlist.new()

loc_spec_bl:append(loc_spec)

cls.map_set_val(spec_string, loc_spec_bl)

Original Ver.1 Ver.2 Ver.3

Thumbnail Index

● Read object and apply ImageMagick transformation

● Append (cache) the new version of the image to the object

● Save the location of the version indexed by its specification

● Write a smart read function to consult the cache

● Application can dynamically alter the transformation applied

App-specific Object Interface

img_bl:append(img)

Thumbnail Index

img_bl:append(img)

Thumbnail Index

img_bl:append(img)

Thumbnail Index

img_bl:append(img)

Thumbnail Index

Getting started with scripted RADOS

● Buyer beware!○ Experimental code○ Works and fairly stable

● Code available on github○ http://github.com/ceph/ceph○ branch: cls-lua

● In-depth explanation and examples○ http://ceph.com/rados/dynamic-object-interfaces-with-lua/

That’s it!

● Lot’s of interesting development● Ceph is a great platform for experimentation● Q&A

Ceph Day SF 2015 - Big Data Applications and Tuning in Ceph

Technology