A public cloud on Ceph and Apache CloudStack

Who am I?

● Wido den Hollander (1986)● Co-owner and CTO @ PCextreme B.V.● Ceph community member● CloudStack committer and PMC member

● Wrote the Ceph (RBD) integration● Work on the KVM Hypervisor code● Maintain Ubuntu .deb packages

Who is PCextreme B.V.?

● Dutch hosting company since 2004● 25 employees● 50.000 customers● >100.000 shared hosting customers● Running a public cloud on Apache CloudStack with Ceph storage

● Servers in Amsterdam, Rotterdam, Miami, Tokyo and Los Angeles

This is not a commercial talk

I am just showing how great CloudStack works for my company

I have €20 gift cards to give away

Our public cloud: Software

● Apache CloudStack 4.4.1● KVM hypervisors on Ubuntu 14.04● Ceph Primary Storage● NFS Secondary Storage

Our public cloud: Hardware

● 100% SuperMicro hardware● Intel Xeon E5 processors with 256GB of RAM per hypervisor

● Tiered SSD and HDD storage with Ceph● 10Gbit networking

Our public cloud: Hypervisors

Our public cloud: The numbers

● 2.000 Running Instances● 2 zones in Amsterdam

● Miami, Tokyo and Los Angeles within a month● 44 Hypervisors

● 11TB of RAM● 704 cores

● 700TB of Ceph storage● 30.000 API calls per day

● Customers have API access

Our public cloud: Network

● Basic networking● Keep it simple● Let real routers do the routing● Public Clouds get a lot of DDoS attacks...● Security Grouping for isolation

● IPv4 and IPv6● No private networks, all public IP-space● IPv6 is currently not fully supported by CloudStack

Our public cloud: Ceph

● Ceph is a Open Source unified distributed storage platform which can expose Block Devices

● Integrated into various clouds● We use it as Primary Storage for our cloud

Our public cloud: Ceph

● Running 52 Ceph servers spread out over three racks

● Roughly 700TB of RAW storage● Mix between 1TB SSD and 3TB HDD● 10Gbit network for low-latency

Our public cloud: Ceph

● Running 52 Ceph servers spread out over three racks

● Roughly 700TB of RAW storage● Mix between 1TB SSD and 3TB HDD

Our public cloud: Ceph

Our public cloud: Ceph

Our public cloud: Ceph

● Ceph is has no Single Point of Failure. If fault tolerant and self-healing.

● We never schedule a maintenance window and do maintenance during office hours.

● Ceph can scale from a couple of TB into multiple PBs without downtime● We started with 7 machines and grew to 52 without downtime

Our public cloud: Scaling

● The main problem with scaling is storage● Ceph solves this problem for us

● Adding Hypervisors is very simple● Instantly provides more RAM and CPU● Live Migrate Instances to new hardware

Why CloudStack?

It works

Why CloudStack?

● Open project, driven by the Apache Foundation● Easy access to the project and code● Stable code and understandable code● Very rich set of features● Fast and rich API

● And it just works

Did we encounter issues?

● Yes, we did. None of them were very serious:● High Availability kicked in wrongly a couple of times causing some Instances to be restarted

● Firewalling issue which caused some Instances to loose their network

● We fixed them upstream

Stamina vs Agile

● Stamina Instances are Highly Available and use Ceph storage● For persistent storage and long running applications

● Agile Instances are running on local SSD storage and are not HA● For short-running applications, eg webservers

Stamina vs Agile

Stamina vs Agile

● Using different Service Offerings and Host tags we deploy the Instances on the right hardware

Our Control Panel

● We wrote our own Control Panel which talks to the CloudStack API

● Custom made for our customers● Developed in-house● Written in PHP (Laravel framework)● Scales on mobile devices

● URL: https://cp.pcextreme.nl/

Our Control Panel

Our Control Panel

Conclusion

● CloudStack has been working for over 2 years for us

● Reliable and easy to maintain

Thank you!

● I have gift cards to try out our public cloud

● Twitter: @widodh● E-Mail: wido@widodh.nl● Blog: http://blog.widodh.nl/

● https://www.pcextreme.com/