Date post: | 19-Aug-2014 |
Category: |
Engineering |
Upload: | animesh-singh |
View: | 7,755 times |
Download: | 13 times |
1
OpenStack Summit May 12-16, 2014 Atlanta, Georgia
Optimizing OpenStack for large scale Cloud Foundry deployments
Animesh Singh, Daniel Krook, Jason Anderson, Kalonji Bankole@animeshsingh @danielkrook @andersonljason @k_bankole
2 © 2014 IBM Corporation
OAuth
OSLC Infrastructureas a Service
Platformas a Service
Softwareas a Service
API economy
Cloudoperating
environment
Software-defined
environment
TOSCA
http://www.ibm.com/developerworks/cloud/library/cl-open-architecture/
IBM cloud and open technologies
At all tiers, IBM is committed to building its cloud on an open cloud architecture
Products and services built on open source and open standards benefit IBM and customers
3 © 2014 IBM Corporation
Gold Sponsors
IBM is #2 in contributions to integrated projects
IBM is working to accelerate OpenStack Foundation success…
Because an open interoperable Cloud is critical for flexible cloud deployment and customer success…
IBM has 15 core contributors15
IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function380
IBMers active in the projects
2
107
Mar2013
May2014859
Contributors8,500 Individuals
2130Contributors16,100+ Individuals
Exponential growth
Platinum Sponsors
OpenStack Participant Growth
4 © 2014 IBM Corporation4
©2014 IBM Corporation
…And embracing Cloud Foundry as an open source PaaS
Continuing our history of strengthening and extending open source
TOTAL CONTRIBUTORS
LINES OF CODE
PULL REQ / WK COMPANIES
1,274 Average per month2013 average: 13312mo average: 98
Lifetime average: 58
711k
50+ 43
5 © 2014 IBM Corporation
Cloud Foundry PaaS
Cloud Foundry services registry and runtime management layer.
Components are dynamically discoverable and loosely coupled, exposing health through HTTP endpoints so agents cancollect state and act on it.
Cloud Foundry architectural overview
User Authentication and Authorization
Router
DEA Pool
Service Gateway Apps
Service Connector
Health Manager
Messaging
Cloud Controller
Build Packs
Cloud Foundry BOSH
MicroClouds
PrivateClouds
PublicClouds
Service Nodes
6 © 2014 IBM Corporation
Integrate!
7 © 2014 IBM Corporation
PaaS
BOSH Cloud Provider Interface (CPI) Initially written for VMware and Amazon Web Services OpenStack CPI• OpenStack CPI written by PistonCloud Apache2 license• PistonCloud - An enterprise OpenStack company, founded by few of OpenStack
founders
Cloud Foundry – OpenStack integration
OpenStack
Cloud
Servers Networking Storage
Integration
IaaS
Hardware
8 © 2014 IBM Corporation
BOSH deployment process
Deployment Manifest• Release
name/version• # VMs, job params• Stemcells to use
Stemcell• Base OS• BOSH agent
Release• Name
Jobs• Software packages• Config templates• Scripts
BOSH
DeployedEnvironment Virtual Machine
• Configuration• Software Packages
Virtual Machine• Configuration• Software Packages
Virtual Machine• Configuration• Software Packages
Virtual Machine• Configuration• Software packages
9 © 2014 IBM Corporation
Sample manifest
10 © 2014 IBM Corporation10
Cloud Provider Interface for OpenStack
• OpenStack CPI is an implementation of the BOSH CPI.
• Leverages the Fog Ruby gem for OpenStack
• OpenStack CPI gem (Ruby package), the OpenStack Registry and Agents all reside on the Cloud Foundry Bosh repository https://github.com/cloudfoundry/bosh
• Stemcell Image (Base OS Image) and sample manifest files for OpenStack exist
11 © 2014 IBM Corporation
Requirements:• Static / floating ips• Persistent disks• Outbound Internet connectivity• Custom flavors• Increased quota• Security groups
11
Integration requirements for OpenStack
12 © 2014 IBM Corporation
Automate!
13 © 2014 IBM Corporation
OpenStack discovery:
• Extend the Fog OpenStack gem used by CPI to discover and OpenStack artifacts in an automated manner in the Cloud Foundry manifest
• Pass OpenStack credentials and discover OpenStack compute and network information.
Deployment automation: Tip one
Discover Flavors
Discover Networks
Discover VM Subnet
Discover DHCP IP
Discover Keypairs
Discover Gateway IP
Discover Security groups
14 © 2014 IBM Corporation
Deployment automation: Tip twoOpenStack pre-req setup:
• Extend the Fog OpenStack gem used by CPI to setup Cloud Foundry OpenStack pre-reqs• Setup tenant quota• Keypairs• Custom Flavors• Security Groups setup
Create Keypair
Create FlavorsRouter, DEA, Cloud Controller, Service Nodes
Create Security GroupsCf-private, cf-public
Setup tenant quota
15 © 2014 IBM Corporation
Deployment automation: Tip threeCloud Foundry manifest generation:
• Hard to manually create and edit Cloud Foundry manifests – which can extend up to 1000+ lines • Automation around template creation – an extension from the previous efforts of OpenStack Discovery
and Cloud Foundry pre-req setup. • Can create templates for both nova-network and neutron based environments• Iterates through jobs and assigns static IPs to them if we are using an environment without PowerDNS.
16 © 2014 IBM Corporation
Deployment automation: Tip fourStemcell automation:
• Community stemcell image has empty fstab and this makes the root partition auto detection to fail, and consequently provisioning to fail.
Multiple Solutions: 1. Inject file into stemcell
• Mount stemcell image using qemu-nbd• Inject /etc/fstab into mounted os• Create new image based on snapshot of modified os
2. Modify Openstack Configuration: • Change “libvirt_inject_partition” parameter in nova.conf from -1 (autodetect) to 1 (use first partition)
3. Modify bosh “Stemcell Builder” script• Script injects /etc/fstab into CentOS, but not Ubuntu (as seen here)
4. Install cloud-init in the image• Best solution to make it work in conjunction with metadata service
Stemcell• Base OS• BOSH Agent
17 © 2014 IBM Corporation
Scale!
18 © 2014 IBM Corporation
Sample CF sizing for 1,000 small applications
~60 virtual machines
• 20 high memory DEAs
• 11 CF fabric components
• 26 service broker and service nodes
160 CPU
~500 GB memory
~1.5 TB VM disk
~200 GB block storage
19 © 2014 IBM Corporation
Controller Node
Our initial OpenStack cluster was built on powerfulhardware…
…but it was not ideal for Cloud Foundry
Compute NodesCompute NodesCompute Node
Nova Compute
Nova Network
Storage Node
Cinder (volume)
Glance (image)
1x
20x
Dedicated Servers
(bare metal)
Nova API
Nova Scheduler
Keystone
Horizon
Database (MySQL)
Messaging (Qpid)
1x
128 GB memory6 cores36 TB diskeach
20 © 2014 IBM Corporation
Weaknesses in the OpenStack layout affectedthe stability of our Cloud Foundry configuration
Single points of failure in the networking, messaging, and storage configuration made BOSH unhappy.
‒ Qpid on Folsom memory leaks made the head node a bottleneck. ‒ Lost messages affected VM and storage creation, leaving many
resources in stuck states.‒ Single Nova API process design was overloaded with BOSH traffic‒ VLAN per tenant design routed much traffic through the single
head node, impacting both the fabric and brokered services.‒ Cinder and Glance storage and APIs were located on a single
node.
Bare metal allocation of resources was also inefficient‒ Every OpenStack component ran as a process directly on the OS‒ Made reacting to workload changes difficult, i.e., difficult to
redistribute OpenStack components to react to Cloud Foundry demands
21 © 2014 IBM Corporation
We’ve addressed the major issues OpenStack by distributing components more intelligently
Moving some components to VMs, assigning controller, compute, data, storage roles to groups of components improves availability, flexibility, scalability and maintainability
Compute NodesCompute NodesCompute Node
Nova Compute
Cloud Controller Data Nodedatabase (mysql)
messaging (qpid)
Cloud Controller Master Node
nova-api
nova-scheduler
Keystone
Horizon
Cloud Controller Data Nodedatabase (MySql)
messaging (Qpid)
Cloud ControllerStorage Node
Cinder (volume)
Glance (image) Nova Network
Controller Node
Nova API
Nova Scheduler
Keystone
Horizon
Storage Node
Cinder (volume)
Glance (image)
2xData Node
Database (MySQL)
Messaging (RabbitMQ)
2x 15x
Load Balancer NodeHA Proxy
Keepalive
Load Balancer NodeHA Proxy
Keepalive
Load Balancer NodeHA Proxy
KeepAlive
6x3xVirtual
Machines(VMs)
Dedicated Servers
(bare metal)
22 © 2014 IBM Corporation
Matching the right hardware for OpenStack to host Cloud FoundryLearn more about these topics at sessions from our team Wednesday
23 © 2014 IBM Corporation
Optimize!
24 © 2014 IBM Corporation
Deploy optimization tips• Increase OpenStack API rate limits (/etc/nova/api-paste.ini)
[filter:ratelimit]paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factorylimits = (POST, "*", .*, 9999, MINUTE);(POST, "*/servers", ^/servers, 9999, DAY);(PUT, "*", .*, 9999, MINUTE);(GET, "*changes-since*", .*changes-since.*, 9999, MINUTE);(DELETE, "*", .*, 9999, MINUTE)
BOSH
BOSHAdministrator
NATS
PostGres
Redis
Blob Store
Health Monitor
Director
Registry
Clo
ud P
rovi
der I
nter
face
(CP
I)
Cloud Foundry Environment
mongo_gw
mysql_gw
redis_gw
postgres_gw
cloud_ctrler
ccdb_ng
router
nats
debian_nfs
dea
uaa_db
uaa
loggregator
health_mgr rabbit_gw
mongo_node
mysql_node
redis_node
postgres_node
rabbit_node
OpenStackEnvironment
Ope
nSta
ck
AP
I
25 © 2014 IBM Corporation
Deploy optimization tips• Increase BOSH’s NATS timeout
• A large number of BOSH agents can overwhelm NATS• Place the following in your MicroBOSH manifest file:
apply_spec: properties: nats: ping_interval: 30 ping_max_outstanding: 30
BOSH NATS
mongo_gwmysql_gw redis_gwpostgres_gw
cloud_ctrler
ccdb_ng
router
nats
debian_nfs
dea
uaa_db
uaa
loggregator
health_mgr
rabbit_gw
mongo_nodemysql_node redis_nodepostgres_node rabbit_node
26 © 2014 IBM Corporation
Deploy optimization tips• Avoid name based security groups with nova-network
• Name based security groups require message bus activity and database updates proportional to the number of existing VMs
• Use an OpenStack scheduler which distributes VM load• Ensure that jobs are distributed across compute nodes instead of grouping• The default scheduler has the correct behavior
compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
27 © 2014 IBM Corporation
Security optimization tipsGuiding principle: Use the most limited permissions required to complete the job
1. Use tenant credentials• Do not use full admin credentials in your BOSH manifest
2. Only open the security ports which are required
28 © 2014 IBM Corporation
Security optimization tips3. Separate your VM network and OpenStack management network
• One challenge with BOSH VMs deployed on OpenStack• Setup pin holes for those particular VMs (Director, OpenStack Registry)
4. Separate Cloud Foundry’s core fabric from other brokered services• Services connected using the service broker should be on a separate network• Prevents compromised services from effecting Cloud Foundry
29 © 2014 IBM Corporation
What next? Join us at the Cloud Foundry Summit
30 © 2014 IBM Corporation
Monday, May 12 – Room B314
12:05-12:45
Wednesday, May 14 - Room B312
9:00-9:40
9:50-10:30
11:00-11:40
11:50-12:30
OpenStack is Rockin’ the OpenCloud Movement! Who‘s Next to Join the Band ?Angel Diaz, VP Open Technology and Cloud LabsDavid Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure
Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships
Taking OpenStack beyond Infrastructure with IBM SmartCloud Orchestrator.Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator
IBM, SoftLayer and OpenStack - present and futureMichael Fork - Cloud Architect
IBM and OpenStack: Enabling Enterprise Cloud Solutions Now.Tammy Van Hove -Distinguished Engineer, Software Defined Systems
IBM Sponsored Sessions
31 © 2014 IBM Corporation
Monday, May 12
3:40 - 4:20
3:40 - 4:20
Tuesday, May 13
11:15 - 11:55
2:00 - 2:40
5:30 - 6:10
5:30 - 6:10
Wednesday, May14
9:50 - 10:30
2:40 - 3:20
Thursday, May 15
9:50 - 10:30
1:30 - 2:10
2:20 - 3:00
IBM Technical Sessions
32
Be sure to stop by the IBM booth to see some demos and get your rockin’ OpenStack t-shirt while they last.
Thank you !