Post on 13-Apr-2017
transcript
As a Service: Cloud Foundry on OpenStack - Lessons Learnt
Apps
@AnimeshSingh
OpenStack Summit Barcelona
October 2016 @blueboxjesse
A polyglot “platform for the people”
• The de facto open PaaS platform
• Foundation established Dec. 2014, under Linux Foundation umbrella
Cloud Foundry
PaaS
Cloud Foundry on OpenStack
IaaS
UAA
Router
DEAPool
ServiceGateway Apps
ServiceConnector
HealthManager
Messaging
CloudController
BuildPacks
ServiceNodes
CloudFoundryBOSH
CloudProviderInterface
BOSH Deployment Process
Deployment Manifest • Release name/version • # VMs, job params • Stemcells to use
Stemcell • Base OS • BOSH agent
Release • Name • Software packages • Config templates • Scripts
BOSH
Cloud Foundry
Virtual Machine • Configuration • Software Packages
Virtual Machine • Configuration • Software Packages
Virtual Machine • Configuration • Software Packages
Virtual Machine • Configuration • Software packages
Problems we faced As CF and OpenStack deployers we experienced various issues deploying CF and OpenStack, and CF on OpenStack
§ Instability: Instability of OpenStack environments from various distros
§ APIs: Lack of compliance in APIs and new releases. Some of the API behavior also differs based on the plugins used
§ Capacity: Right from cpu/mem/disk etc. to HA, persistent disk, floating ips etc. – sizing needs to be precise
§ Network: Should CF co reside with other management components in a network?
§ OpenStack for Enterprise Software: Enterprise software lags, with most variations still available only for VMware.
§ Generic Deployment: Consistency when deploying CF on OpenStack, VMware or any other IaaS § Combined OpenStack and Cloud Foundry Usage: How to allow seamless usage and consumption of OpenStack
services along with CF services at the same time?
§ CF HA: What works, what doesn’t?
§ Proxy: Environments behind customer firewalls with restricted outbound access § Constant Release Cycles: Both CF and OpenStack have frequent releases, patches, updates etc. How do we
reconcile?
7
50% experience significant issues deploying CF onto OS
45% are intermediate OS users
Problems Community Faces: Survey Results
Close to 50% of users had to customize their environment
for CF
Problems Community Faces: Survey Results
Most users on Juno/Kilo Close to 50% of users run their own local OpenStack
Problems Community Faces: Survey Results
APACHE HAPROXY APACHE HAPROXY
HA MYSQL ARBITER
(PERCONA)
HA NEUTRON !
HA KEYSTONE!
HA GLANCE!
HA HORIZON!
NOVA!
CINDER API & ISCSI!
HA NEUTRON !
HA KEYSTONE!
HA GLANCE!
HA HORIZON!
NOVA!
CINDER API & ISCSI!
HA MYSQL (PERCONA) RABBITMQ HA MYSQL
(PERCONA) RABBITMQ
Our OpenStack Offering - Bluemix Private Cloud (IaaS)
NOVA
CINDER API & ISCSI
NOVA
CINDER API & ISCSI
§ IBM Platform as a Services offering • Three deployments styles: public, dedicated, private • 1M+ registered users (20K+/month) • 100K+ running apps and 500+ services
Bluemix Bare Metal (public and dedicated) and OpenStack/VMware (private)
Our Cloud Foundry Offering - Bluemix (PaaS)
Services
Lifecycle Management
IDS
Application Runtime
Runtimes & Frameworks
Middleware Application Operational Mobile External Data
Node Java Ruby Worklight WebSphere Liberty
Eclipse IDE Application
Composition Environment
Create & Manage Services
Test/Run Test/Run
Explore Services
Explore Services
IBM Bluemix
Check In Code Check In Code
Web IDE (Eclipse Orion)
Test if your OpenStack is fit for Cloud Foundry deployment
Manually, you can run the validations here to see if your OpenStack is a good fit: https://docs.cloudfoundry.org/deploying/openstack/validate_openstack.html § Can you access the OpenStack APIs for your instance of OpenStack?
§ Can you access OpenStack metadata service from a virtual machine?
§ Can you ping one virtual machine from another?
§ Can you invoke large numbers of API calls?
§ Can you create and mount a large volume?
§ Can you upload and deploy an Ubuntu Server Cloud Image?
§ Can networking be configured for both external and internal IPs?
§ Can you access the Internet from within instances?
Test if your OpenStack is fit for Cloud Foundry deployment To check in automated fashion if your OpenStack is ready to run BOSH and Cloud Foundry, you can run this CF Incubator project : https://github.com/cloudfoundry-incubator/cf-openstack-validator
16
Testing the CPI API Upload stemcell Create VM Find VM Create disk Find disk Attach disk to VM Detach disk from VM Create disk snapshot Delete disk snapshot Delete disk Delete VM Delete stemcell
Other OpenStack tests Check API rate limit Check required versions of OpenStack projects CPI requires API version 1 for glance and cinder Security group settings Check if security group rules allow necessary incoming/outgoing ports Outbound internet access Timeservers can be reached Attach a floating IP Set VM metadata tags Access a VM over ssh from the outside Access one VM from another VM Create a large volume
Get the right sizing for Cloud Foundry • Sample sizing are available online as references e.g
https://docs.cloudfoundry.org/deploying/openstack/required-flavors.html https://www.mirantis.com/blog/full-stack-devops-with-pivotal-cloud-foundry-on-mirantis-openstack https://docs.pivotal.io/pivotalcf/1-7/customizing/openstack.html#versions
• Increase default quota of the tenant to meet minimum requirement e.g number of ports always gets hit first • Recommended ‘OpenStack flavors’ disk space should be go in ephemeral disk.
Keep root disk to 10GB minimum
Optimize OpenStack scheduling and communication
Optimize Internal Communication: • Configure OpenStack for high volume API calls e.g Increase OpenStack API rate limits (/etc/nova/api-paste.ini) • Avoid name based security groups with nova-network. Name
based security groups require message bus activity and database updates proportional to the number of existing VMs
• If Neutron is configured with VXLAN via the Open vSwitch mechanism, the MTU should be 1400. For GRE, the recommended number is 1460.
Use the right Scheduler
Use an OpenStack scheduler which evenly distributes load e.g compute_scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
Account for different OpenStack configurations • If your OpenStack setup requires you to use disks from block
storage instead, that will work with Cloud Foundry as well. properties. openstack.boot_from_volume: true
• By default, the VMs created try to receive data from OpenStack's HTTP metadata service. If your OpenStack installation doesn't provide metadata and userdata over HTTP, but requires you to use config-drive instead of metadata, you need to specify this in the property properties.openstack.config_drive: cdrom
Optimize Cloud Foundry / BOSH bandwidth and communication
Optimize Internal Communication: • Increase BOSH NATS time out for high concurrency
e.g Configure NATS messaging bus to increase ping interval
Optimized routing and bandwidth allocation • Isolate Cloud Foundry components using multiple networks
e.g DEAs in their own network, brokered services in their own network
Optimize log forwarding
• Supporting services like logging, report generation etc should go in their own tenant network. Also any communication between the VM(s) sending logs, and log receiving component(s) should go over the private network. Don’t use floating ips as destination, else you are paying double the cost.
Optimize for Security and Network
• Only open ports which are needed. Use the most limited permissions required to complete the job.
• If OpenStack is using a self-signed certificate, configure properties.openstack.connection_options to include the property ca_cert
• Use tenant credentials: Do not use full admin credentials in your BOSH manifest
• Minimize floating ips: Except for the incoming Gateway device(HA Proxy, Datapower, F5 etc), none of the fabric VMs should be on public network or need a floating ip
Account for customer boundary firewall or proxy • BOSH will retrieve CF release components via the
URL provided in deployment manifest. Also services and CF apps may need outbound access. In environments behind firewall or proxy with restricted access, this could be a major issue.
• If the firewall or proxy requires both destination and source ips to be enlisted, prepare a list of destination IPs/URLs you need to reach out and hand it to datacenter admin
• If your OpenStack VMs don’t have a floating ip from external network, the source ip presented to firewall will be Neutron gateway ip
• Cloud Foundry doesn’t support out of the box SSL packet inspection where the SSL certificate for a given sight is replaced by your own self signed certificate. Inspection is only supported using an Internet Authority signed certificate.
Work on seamless OpenStack update/upgrade OpenStack separates the availability of workloads (Data Place) from the availability of the API services (Control Plane) OpenStack Code Major Upgrade Maintenance: Moving from one OpenStack release to another, e.g. from Kilo to Mitaka with data migration • Infrequent, twice a year at most. New OpenStack code is put in place for all of the services that make up a cloud (e.g. Nova,
Neutron, Glance, Heat, Cinder, Ceilometer, Aodh, Swift, etc..). • ~10 minutes for each service
OpenStack Code Minor Upgrade Maintenance: Upgrading OpenStack service, without data migration • Typically only bug fixes. New OpenStack code is put in place and the services associated with that code are simply restarted.
• ~5 seconds for each service
OpenStack Config Maintenance: Changing the configuration for one or more of the OpenStack services, needed to correct bugs in the service, or to tune the service due to consumer requests. • Require the restart of a particular service. The disruption to the API control plane is isolated to the services receiving configuration
updates. • ~ seconds for each service
OpenStack Installation: • Leverage the open source Ursula Ansible and Rally Cloud infrastructure Automation framework • Requires information about hardware, network environment and software repositories.
Bluemix Private Cloud Install Automation
Setup Storage
Setup OpenStack
Setup Network
Run validation
Setup Hardware
Ursula, Rally-OpenStack
OpenStack Discovery:
• Leverage the open source Fog gem to discover OpenStack artifacts in an automated manner • Require OpenStack credentials and discover OpenStack compute and network information.
Cloud Foundry on OpenStack deployment Automation
Discover VM Configuration Sizes
Discover Network Subnets
Discover Network Security Rules
Discover DHCP , DNS Gateway and floating IPs
Discover Security Credentials
Cloud Foundry Pre-req setup on OpenStack:
• Leverage the open source Fog gem to setup Cloud Foundry requirements in an automated manner • Setup according to best practices and guidelines – still giving users the flexibility to change if desired
Create Security Credentials
Create VM configs for Router, DEAs, Cloud Controller, Service Nodes
Create network Security Rules
Setup tenant quota
Cloud Foundry on OpenStack deployment Automation
Cloud Foundry Deployment Automation • Automate stemcell modification • Automate Cloud Foundry deployment manifest file genration using Ruby ERB • Automate upload of Cloud Foundry core release, services and runtime frameworks, followed by Cloud
Foundry deployment
Stemcell Creation and Upload
Generate BOSH and Cloud Foundry Manifest
Upload Cloud Foundry core, Services and runtime
Deploy Cloud Foundry
Deploy bosh director
RUBYBOSH
Cloud Foundry on OpenStack deployment Automation
© IBM Corporation 29
As a Service: As a provider, look into offering Cloud Foundry and OpenStack as a Service
Why as a Service? Cloud Foundry:
– New release every 2-3 weeks
– Bluemix PaaS is a combination of CF and 150+ Services
– Older versions will lead to huge version mismatches, and lead to version sprawl
– Keeps Public/Dedicated and Local Bluemix in sync
OpenStack:
– Twice annual releases that touch the entire code base.
– Upgrading sequentially is important: stay up to date!
– OpenStack’s complexity requires expertise in many operational areas
– Focus on higher business value. Work with OpenStack, not on OpenSTack
Private Cloud Hardware
Bluemix Private Cloud: IaaS Relay: Box Panel
Box Panel
Site Controller (Software) B
luem
ix
Priv
ate
Clo
ud
Ope
nSta
ck
Box Panel Formations
Central Authentication Customer Relationship Management Service Catalog and Metering Billing and Invoicing
Obj
ect
Sto
rage
Blo
ck
Sto
rage
Cor
e N
etw
orki
ng
Inventory Management Network Management Reporting and Analytics Support Ticking, Chat and Email
IaaS Relay: Site Controller
Box Panel
Site Controller
Customer Cloud
Resides on-premises adjacent to customer clouds, providing real-time administrative control of cloud environments.
– Network Automation
– Power Distribution Unit Automation
– System and Network: • Monitoring
• Telemetry
• Logging
– Secure Remote Control and Access
– Bare Metal Provisioning
– Package and Container Repo
SDN router
The Internet Customer Network
IBM Urban Code Deploy
Softlayer Server
Bluemix Platform
Stemcells Releases Manifests
BOSH CLI
Automated
Management Processes
(Deploy, Upgrade etc.)
IBM Urban Code Deploy
Relay
Customer Hardware & Infrastructure
Bluemix Core Services • Monitoring & Logging
• Cache
• Cloudant (Data Store)
• Qradar EPS
• IEM Relay
Configuration Store
(per customer)
Bluemix Code &
Automation Repository
Opensource Code
IBM Test & Staging Validation
IBM Production Deployment &Validation
Bluemix Local
Inception VM
UCD Agent
• Secure connection • Connection originated from customer premise
• Restricted access (agent-only)
Cloud Foundry
ACE UI
Enterprise ITSM
Customer Services
Customer Premises
Premises
On-premise data store for logs, monitoring data etc.
Bluemix Ops Console
IBM Cloud Security Services
Qradar Console IEM Server
Bluemix Ops Directory Server
Privileged ID Governance
IPSec Tunnel
DataPower
• Customer’s Service traffic
• Syndicated & 3rd party service traffic
• App staging artifacts • Inbound & outbound user to app traffic
• LDAP
• Enterprise services
• Other SaaS services • vCenter
Network Isolation Customer Services
Bluemix Private Cloud PaaS Relay IBM Premises
VPN Tunnel
Inception VM
Stemcells
Releases
Manifests
BOSH CLI
DataPower
ACEUI
Metering
AdminUI
NATS
BMDB
Backup
Loginserver
UAA CC
Blobstore
HMCCDB
Loggregator
Gorouter
DEAs
UAADB
Logging
UCD Agent Ops Center Agent
Enterprise ITSM
NetworkIsolation• LDAP
• Enterprise services • Other SaaS services
Customer Services
VPN Tunnel
Bluemix Private Cloud: PaaS and IaaS As a Service!
Bluemix Relay
Bluemix Private Cloud Relay
Site Controller
Repo / Deploy Server Monitoring Server
(Sensu) Logging (ELK)
Bastion (Access)
Cloud Foundry on OpenStack – A Great Fit!
• 100% Open PaaS and IaaS solutions – No vendor lock-ins • Strong and growing community of contributors and sponsors on both sides
• Power of Open Source community can be leveraged to automate the deployment and
lifecycle management of Cloud Foundry on OpenStack
• OpenStack meets Cloud Foundry integration requirements, and is totally configurable and adaptable to handle the scale of a PaaS solution like Cloud Foundry
• Bottom Line: It’s a match made in Heaven !