Date post: | 15-Jan-2015 |
Category: |
Technology |
Upload: | inktank |
View: | 1,588 times |
Download: | 1 times |
DreamObjects Cloud Object Storage
Powered by Ceph
About Me
• Senior systems engineer at dreamhost • Been with dreamhost since 2007 • Systems lead for DreamObjects • Excited about storage, distributed systems,
networking, linux internals, automation and monitoring
• I built my first ceph cluster in april, 2010
twitter: @mmgaggle
Anatomy of DreamObjects
Nuts and Bolts - Storage Nodes
• Intel nehalem or AMD lisbon • 32 GB of ram • 8-12 enterprise 3.5" sas disks • LSI megaraid 1078/2108 • 5x Gb nics • IPMI 2.0 BMC
Nuts and Bolts - Support Nodes
• Intel westmere-ep • 8 GB ram • 3x Gb nics • 4x SAS drives • IPMI 2.0 BMC
ceph monitors, radosgw, proxies, object sync
Tubes
• IPv6 native • Dual-stack application proxy / balancer • Application, rados, and cluster networks • One gig links, bonded • LLDP for discovery
Traffic
• IPv6 and IPv4 api endpoints • IPv6 to radosgw nodes • Stud for ssl termination • HAproxy for load balancing • 10Gb backend and frontend
Ceph Numbers
• Two plus petabytes of raw storage • Triplicate replication • 900+ OSDs • 7 nines durability • Single datacenter
CRUSH Configuration
• Row, rack, node, and disk fault tolerant • Weighted, even distribution of data • Object striping • Online expansion and contraction
OSD Configuration
• Single disk RAID0 • Two partitions per RAID0, data + journal • Writeback enabled • Disk cache disabled • Using XFS for filesystem
Application Tuning
• No keepalive on apache • Increase apache thread pool • Increase radosgw thread pool • Raise radosgw ulimits
Argonaut
• Lots of failure testing • Tested on large clusters (~900 osds) • Weight osds in/out of the cluster
Bobtail
• Tunables for less aggressive remapping • CNAME support • OSD performance ++
Control and Management
• chef • pdsh • runit/upstart • jenkins • sensu • collectd • graphite • logstash
Chef
• Bootstrap cluster, new nodes • Packages and configuration • Creates, destroys and encrypts osds • Search node metadata, restful api • Roles mapped to pdsh genders • User and ssh key management • Loads more..
pdsh
Parallel distributed shell • Ceph monitors are a natural control point • pdsh from monitor to other nodes • ssh-agent + ssh agent forwarding • Works when chef api doesn't
runit/upstart
Keep ceph daemons running • ceph-osd • ceph-mon • radosgw
Jenkins
• Continuous api functional tests • Monitor chef convergence • Generates alerts on failures
Sensu
• Relatively new monitoring system • Open source • Written in ruby • State stored in Redis • Pub-sub with local agent • Handlers monitor state and generate alerts
collectd
System metrics • cpu • memory • disks • entropy • interfaces
graphite
• bucky - collectd, statsd, metricsd api • carbon cache • whisper for metric storage (sequential writes) • graphite webui
logstash
• lumberjack agent to ship logs to logstash • logstash graphite output • logstash elasticsearch output • elasticsearch restful api • kibana webui sits on top of elasticsearch
Future
• East coast region • Pool replication to another datacenter • Object expiration, root site support • Content delivery network
..not necessarily in that order ;)
Thanks
Questions?
kyle.bader [at] dreamhost.com @mmgaggle
Inktank’s Professional Services Consulting Services:
• Technical Overview • Infrastructure Assessment • Proof of Concept • Implementation Support • Performance Tuning
Support Subscriptions: • Pre-Production Support • Production Support
A full description of our services can be found at the following:
Consulting Services: http://www.inktank.com/consulting-services/
Support Subscriptions: http://www.inktank.com/support-services/
Check out our upcoming and on demand webinars from Inktank
Upcoming: Advanced Features of Ceph Distributed Storage (delivered by Sage Weil, creator of Ceph)
February 12, 2013 10:00AM PT, 12:00PM CT, 1:00PM ET http://www.inktank.com/news-events/webinars/
On Demand:
Getting Started with Ceph http://www.inktank.com/news-events/webinars/ Intro to Ceph with OpenStack http://www.inktank.com/news-events/webinars/
Contact Inktank
[email protected] and 1-855-INKTANK Don’t forget to follow us on: Twitter: https://twitter.com/inktank Facebook: http://www.facebook.com/inktank YouTube: http://www.youtube.com/inktankstorage
THANK YOU!