Adventures & Challenges building an OpenStack public cloud
Walter Heukels, Koert van der Veer en Pim van Riezen
The Sysadmin Experience
Walter Heukels
About Me
• Walter Heukels – Senior Engineer at CloudVPS – Working on OpenStack Infra
About OpenStack
• Free soJware (Apache License) • WriOen in Python
• Big project • Very flexible
OpenStack AssumpSons
• Flexible, but it does have a philosophy – CaOle servers, not pets – Makes certain assumpSons
– We came up against some of those
• Don't go against the grain
OpenStack AssumpSons
• Flexible, but it does have a philosophy – CaOle servers, not pets – Makes certain assumpSons
– We came up against some of those
• Don't go against the grain – Unless you really want to
OpenStack Structure
• Lots of sub-‐projects – Nova – SwiJ – Quantum / Neutron
– Keystone – ...
• Distributed architecture
Our Cloud
• Object Store – Since April 2013 – Very Cool
• Compute – Started free public beta – ZFS Storage – KVM VirtualisaSon
Lessons Learned: Technical
• People are mostly running private clouds at the moment • Not much informaSon available on running a public cloud
• Examples – MulSple external networks
– Keystone (authenScaSon) performance
Lessons Learned: ExpectaSons
• Customers make assumpSons • Especially our customers
• Examples: – IP spoof protecSon (“my VPN router doesn't work”)
– HA for VM's ..we're working on this!
Lessons Learned: ExpectaSons
• Security groups?!? • Bitcoins!! • Some customers don't know what to expect “Will my
Wordpress site scale automaScally?”
Lessons Learned: Debugging
• Distributed system • Race condiSons can occur • Hard to find the logging you need • Graph everything
The Road Ahead
• New features – LBaaS – VPNaaS – Database as a Service – PaaS
The Dev Experience
Koert van der Veer
About Me
• Koert van der Veer – Senior Developer at CloudVPS – Working on OpenStack features
My Role Before OpenStack
• Development responsible for every detail of cloud management system
• Large CompeStors are moving incredibly quickly – High pressure to add new features – No Sme to fix technical debt
My Role With OpenStack
• Responsible for custom features only • Bugs are usually fixed by others • Large acSve community helps diagnosing problems • ContribuSng is very saSsfying and results in goodwill
Our Work on OpenStack
• Core features (contributed) – ZFS block storage – SwiJ features – Bugfixes
• Deployment • Billing • Interfacing
Development Tools
• Python with geventlet, kombu, sqlalchemy, etc. • DevStack • Unit tests • Tempest
• Grenade
Development Environment
• ProducSon close to git head • Rapidly re-‐deployable testcluster • pip instell –e “.”
Development Work Flow
• PreparaSon • Write code
• Review • Merge
• Maintain
Step 1 -‐ PreparaSon
• Launchpad blueprints • Launchpad bugs • IRC • Mailinglist
Step 2 -‐ Write Code
• Create feature branch • Write code
• Write unit tests • Run unit tests and staSc analysis • Commit
Step 3 -‐ Review
• Submit to Gerrit • Jenkins tests • Other reviews • Core reviewer
approves
Step 4 & 5 – Merge and Maintain
• Zuul reviews and audits code
• Jenkins merges code
Case Study 1: Bug in Cinder-‐Rootwrap
• Bug detected (Jan 7th) • Bug fixed (Jan 9th) • SubmiOed to Gerrit (Jan 9th, Jan 10th) • Approved (Jan 14th) • Zuul rejected (Jan 15th) • SubmiOed to Gerrit (Jan 16th) • Approved (Jan 17th) • Zuul accepted (Jan 19th)
Case Study 2: Custom Cinder Driver
• Goal • Challenges
– Bug in cinder-‐rootwrap – Feature completeness – Unit tests
• Progress
Future Plans for ContribuSon
• High availability for VMs • Per-‐port IP spoofing control • Extra security msg queue • Requests?
The Frontend Experience
Pim van Riezen
About Me
• Pim van Riezen – Senior developer at CloudVPS – Working on OpenStack GUI
Interfacing Goal
• Goal: “Make it easy to get started with a first VM”
Interfacing Challenges
• Lots of dependencies: – Create a keypair – Create a private network – Create a NAT router – Create security groups and rules – Create Server
Interfacing Challenges
• Decisions to make: – Networking – Key management
– …..
Interfacing Challenges
• Security group abstracSon: – The double funcSon as membership tag and access rule grouping
confuses users
– It takes a lot of words to actually explain the concept
Interfacing Challenges
• API documentaSon: – Hard to figure out what extensions are relevant – Different parts of an openstack cloud may be out of sync
– A lot of perculiar choices made in v1 APIs sSll leak through in v2
– Most command line tools also default to v1 APIs
– Lots of documentaSon lacks basic descripSons of parameters – CombinaSon of tracing command line client, making wild guesses, luck
Interfacing SoluSons
• SoluSons: – Comprehensive wizard
– Clear choices – Image metadata
– Predefined security groups