1
Managing Heterogeneous Cloud Infrastructure with OpenStack
Example: Hosting ‘Hybrid’ Applications Comprising Bare-Metal and Virtualized Resources
OpenStack Summit, May 12th, 2014, Atlanta
IBM Research – Haifa
Alex Glikson
Ezra Silvera
2 © 2014 IBM Corporation
Outline
� Heterogeneous Clouds
‒ Background & Motivation
‒ Opportunities and Challenges
� Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications
‒ Goals
‒ Scenarios
‒ Solution Approach
‒ Technical Challenges and Design Considerations
‒ Example
� Summary, Q&A
3
Background & Motivation
4 © 2014 IBM Corporation
Evolution of IaaS Usage Patterns
� Early days of IaaS (‘infancy’)‒ Value proposition:
• Agility, efficiency, elasticity, self-service, pay-per-use
‒ Main usage patterns: • Dev & test, relatively simple ‘cloud-native’ web applications
‒ Offered hardware configurations:• Small set of standardized (virtual) hardware configurations
e.g., AWS:
– started with 1 instance type (2006),
– grew to 5 in 2008, …
Offered highly standardized virtualized infrastructure, used to host small set of applications
5 © 2014 IBM Corporation
Evolution of IaaS Usage Patterns
� Now (‘adolescence’)‒ Value proposition:
• Agility, efficiency, elasticity, self-service, pay-per-use
‒ Usage patterns:
• Rapidly increasing spectrum of applications seeking the benefits of the cloud model
– E.g., ‘legacy’ business applications, HPC, analytics, etc
• Increasing adoption in large organizations, owning many ‘non-cloud native’ applications
‒ Offered hardware configurations:
• Greatly increased heterogeneity is expected
– E.g., AWS now offers >60 instance configurations
• Rise of bare-metal providers
– IBM/SoftLayer, Internap, etc
Evolving to a much larger variety of resources (including bare-metal),used to host much larger variety of applications
6 © 2014 IBM Corporation
Underlying Heterogeneity
� Heterogeneity spectrum (examples)‒ Different CPU/memory/disk model & ratio’s‒ Compute:
• CPUs vs GPGPUs vs accelerators (e.g, FPGAs)
‒ Storage: • Local HDD vs SSD vs SAN vs NAS
‒ Network: • regular vs high-speed vs Infiniband
‒ Virtualization:• Bare-metal vs KVM vs LXC vs Xen vs VMware
7 © 2014 IBM Corporation
Example: Collaborative 3D Design
� Example mapping considerations:
‒ Tier 1 (Web app): regular KVM-based ephemeral VMs
‒ Tier 2 (3D rendering): bare-metal with GPGPUs
‒ Tier 3 (Hadoop): LXC with SSDs, Infiniband interconnect
‒ Tier 4 (Designs DB): VMware VMs supporting DR with SRM, reliable and fast SAN-backed storage
Tier-1: web app
Tier-2: 3D rendering
Tier-4: Designs DB Tier-3: Hadoop
8 © 2014 IBM Corporation
Two Main Drivers of Heterogeneity
� User-driven: new & challenging application requirements
� Provider-driven: natural evolution in hardware over time
9 © 2014 IBM Corporation
New Application Requirements
� Key inhibitors preventing broader IaaS adoption‒ Special resource requirements
• Legacy applications, resource-intensive applications
‒ Isolation• Performance (e.g., due to strict SLAs)
• Security (e.g., due to compliance)
A modern cloud environment must offer HW configurations optimally designed for classes of applications with special requirements
10 © 2014 IBM Corporation
Natural Evolution in Hardware
� HW models and technologies evolve over time
‒ E.g., EC2 Instance Types:
Original (since 2006)
New since December 2013
New since April 2014
Deprecated since April 2014
Model vCPUMem (GB) Storage (GB)
m3.medium 1 3.75 1 x 4 SSD
m 3.large 2 7.5 1 x 32 SSD
m3.xlarge 4 15 2 x 40 SSDm3.2xlarge 8 30 2 x 80 SSD
c3.la rge 2 3.75 2 x 16 SSD
c3.xlarge 4 7.5 2 x 40 SSD
c3.2xlarge 8 15 2 x 80 SSD
c3.4xlarge 16 30 2 x 160 SSDc3.8xlarge 32 60 2 x 320 SSD
r3.large 2 15 1 x 32 SSD
r3.xlarge 4 30 .5 1 x 80 SSD
r3.2xlarge 8 61 1 x 160 SSDr3.4xlarge 16 122 1 x 320 SSD
r3.8xlarge 32 244 2 x 320 SSD
g2.2xlarge 8 15 1 x 60 SSD
i2.xlarge 4 30 .5 1 x 800 SSD
i2 .2xlarge 8 61 2 x 800 SSDi2 .4xlarge 16 122 4 x 800 SSD
i2 .8xlarge 32 244 8 x 800 SSD
hs1.8xlarge 16 117 24 x 2048
t1 .micro 1 0.613 EBS Only
m1.small 1 1.7 1 x 160m1.medium 1 3.75 1 x 410
m 1.large 2 7.5 2 x 420
m1.xlarge 4 15 4 x 420
c1 .medium 2 1.7 1 x 350c1.xlarge 8 7 4 x 420
cc2.8xlarge 32 60 .5 4 x 840
cg1.4xlarge 16 22 .5 2 x 840
m2.xlarge 2 17 .1 1 x 420
m2.2xlarge 4 34 .2 1 x 850m2.4xlarge 8 64 .8 2 x 840
cr1.8xlarge 32 244 2 x 120 SSD
hi1.4xlarge 16 60 .5 2 x 1,024 SSD
A typical Cloud environment must support multiple ‘generations’ of HW
simultaneously
11 © 2014 IBM Corporation
Two Main Drivers of Heterogeneity
� User-driven: new & challenging application requirements
� Provider-driven: natural evolution in hardware over time
Support for infrastructure heterogeneity is essential in any modern IaaS cloud
12
Opportunities and Challenges
13 © 2014 IBM Corporation
Opportunities and Challenges
� Business opportunities enabled by infrastructure heterogeneity
‒ Service/application providers:
• Cloud adoption in ‘traditional’ market segments � agility and efficiency
– E.g., manufacturing, finance, scientific, etc
– IDC: the (extended) HPC market alone is larger than IaaS ($15B vs $9B)
‒ Cloud/infrastructure providers:
• Ability to better fit to specific classes of applications � differentiation
– Especially relevant given the overall commoditization of the IaaS market, recent price wars, etc
14 © 2014 IBM Corporation
Opportunities and Challenges
Challenge: cloud infrastructure natively designed for:
‒ Underlying infrastructure heterogeneity
‒ Ability to seamlessly host applications with special requirements
‒ Still preserve the characteristics of the cloud model
• Agility, efficiency, elasticity, etc
15 © 2014 IBM Corporation
Heterogeneous Cloud Requirements: Cloud User
� Cloud user perspective
‒ Access to a wider range of offered HW/SW configurations
• Performance, security/isolation
• Compute, storage, network
‒ Same or better user experience as with ‘regular’ cloud, e.g.:
• Same: single/unified catalog of configuration ‘flavors’, images, etc
• Same: seamless network connectivity, storage access, etc
• Same: higher-level services (orchestration, LBaaS, DBaaS, etc)
• Better: higher-level application abstraction, hiding the details of the underlying HW (critical with increased complexity of underlying HW)
– Specify goals/requirements, rather than specific resource types/models etc
16 © 2014 IBM Corporation
Heterogeneous Cloud Requirements: Cloud Provider
� Cloud provider perspective
‒ Minimize management cost by unified processes and APIs
• Provisioning, inventory, monitoring, life cycle operations, visualization, etc
– Unified resource model
‒ Secure multi-tenancy across heterogeneous resources
‒ Elasticity & efficiency
• Smart scheduling, to overcome resource fragmentation
• Dynamic balancing between HW-compatible pools by “repurposing”(subject to compatibility)
– Capacity monitoring & prediction
– Automated evacuation & re-provisioning
17 © 2014 IBM Corporation
Heterogeneous Cloud: OpenStack Perspective
� OpenStack is following a similar pattern‒ Started primarily with dev & test and cloud-native applications
• Simplistic approach to flavors, scheduling, performance optimization, etc
‒ Extensive efforts to enable adoption by additional applications, e.g.:
• Hadoop (Sahara)
• DBaaS (Trove)
• Bare-metal (Ironic, TripleO, Tuskar)
• Scheduling (Gantt)
• Reservations (Climate)
• Policies (Congress)
• Etc
‒ Growing community interest in supporting heterogeneous clouds, e.g.:• Multi-Tenant Bare Metal Provisioning (Mon, 11:15, B206; Tue, 2:50pm, B301)
• Deploying OpenStack in a Multi-Hypervisor Environment (Wed, 11am, B101)
• IBM, SoftLayer and OpenStack - Present and Future (Wed, 11:50am, B312)
• Many more related sessions…
18 © 2014 IBM Corporation
Heterogeneous Cloud: OpenStack to the Rescue!
Claim: OpenStack provides a good basis for building heterogeneous clouds
• But few gaps yet to be addressed
19
Hosting ‘Hybrid’ (Bare-Metal + Virtualized) Applications
20 © 2014 IBM Corporation
Hybrid Applications – Background and Goals
� Hybrid environment – includes both physical and virtual resources
‒ Pools: Resources are grouped into bare-metal and virtual pools
� Hybrid application – a composite (e.g., multi-tier) application that spans both virtual and physical server(s)
� Goals
‒ Admin: Minimize management complexity and overhead. Unified management across bare-metal and virtualized pools.
‒ User: Simplified ongoing management and application life cycle.
� Design guideline: If possible try to use OpenStack native solution
‒ E.g., no external coordinators, orchestration, management
21 © 2014 IBM Corporation
Pool 3: Virt Type 2 Pool 4: Bare metalPool 1: Spare (BM) Pool 2: Virt Type 1
Tenant ATenant B
Hybrid Applications – Background and Goals
22
Scenarios
23 © 2014 IBM Corporation
Scenario 1: Node Repurposing
Support native policies that dynamically balance the virtual andphysical pools (e.g., moving servers from one to another)
� Repurposing Virtualized node (Hypervisor) � Physical node
1. Detect resource congestion in the physical pool
2. Identify a candidate host to be re-purposed into physical
3. Trigger “evacuate“ for all VMs on the selected host
4. Assign physical host to the BM pool
� Preliminary network and storage configuration
‒ Server is ready to be deployed as bare-metal application (later on)
24 © 2014 IBM Corporation
Scenario 2: Runtime Decision on Target Pool
Scenario: decide on the server type (BM/VM) upon deployment‒ E.g., according to performance requirements
1. User deploy an application without specifying the server type‒ Using generic definitions (e.g., image, flavor)
2. Automatically choose server type‒ Potentially using scheduler, Heat, ceilometers monitoring, etc.
3. Choose/construct actual image for deployment‒ Option 1: Maintain single base copy of the image and perform
necessary adaptation to match the target server type (BM/VM)
‒ Option 2: Maintain multiple ‘versions’ of the image and select the appropriate one according to target type.
4. Deploy using corresponding provisioning mechanism‒ E.g., leveraging Ironic
‒ Apply adaptation upon boot (e.g., leveraging Heat)
25
Solution Approach
26 © 2014 IBM Corporation
Management Approaches
� Possible solutions range from 'share nothing to 'share everything'
‒ 'share nothing’ - Each pool is managed by a separate instances of OpenStack with dedicated services (e.g., Nova, Cinder).
‒ ‘share everything’ - All pools are managed by single OpenStack “instance”. Use single copy of each service
‒ Some of the services are shared
• Region and cells can be considered special cases for this
� OpenStack can natively handle bare metal and virtual hosts, so why not combining them together in a single OpenStack?
� We focused on the 'shared everything' architecture
‒ Defined as 'Integrated Management‘
27 © 2014 IBM Corporation
Integrated Management: Basic Architecture
� Special resource types, mapped to specially designed resource pools� Use multiple host aggregates, including ‘bare-metal’
‒ Use aggregate filters for scheduling� Networking
‒ Admin network for provisioning and management of compute nodes‒ Separate management network for BM machines hosting user applications‒ Data network combining both VMs and BM nodes
Controller node
mysql-server
rabbit-server
neutron-server
glance
Network node
Neutron.*-plugin-agent
Neutron-l3-agent
Neutron-dhcp-agent
Neutron.*-plugin-agent
Neutron-l3-agent
BM computeCompute node
nova-compute
Neutron.*-plugin-agent
Compute node
nova-compute
Neutron.*-plugin-agent
Compute node
nova-compute
Neutron.*-plugin-agent
BM nodeBM node
BM node
External
Data/Guests
Management
API
Internet
BM MgtBM nodeBM nodeBM spare
28 © 2014 IBM Corporation
Advantages of Integrated Management approach
� Native and simple OpenStack-based solution
‒ Doesn’t require external orchestration
• E.g., Dynamic repurposing of physical nodes can be driven by Heat
‒ No need to coordinate between services of different pools
• e.g., two Neutron instances to achieve unified NW domain
� Simplified administration
‒ Controlling all pools/resources from one place in a unified manner
‒ Reduce configuration complexity and overhead
• Reduce the number of services need to be managed
‒ Better diagnostics and root cause analysis
� Improved user experience and manageability
‒ Maintain/expose combined topology view for different server types
‒ Hide the underlying infrastructure details from the user (if needed)
29
Technical Challenges and Design Considerations
30 © 2014 IBM Corporation
Challenges Examples
� OpenStack has some gaps when managing bare-metal and virtual machines simultaneously (**)
� Networking‒ Manage simultaneously both the virtual and physical networks
• Network isolation for tenants should combine both BM and VMs
• E.g., Use tunnels/overlays connecting physical nodes and VMs
• E.g., Use multiple plugins (e.g., OVS, openflow) simultaneously to gain advantages for each pool (if possible)
** Note: we focus on issues related to Hybrid management
‒ E.g., general BM issues should/are addressed as part of Ironic.
31 © 2014 IBM Corporation
Challenges Examples (Cont.)
� Compute‒ Enhanced scheduler to better support heterogeneity
• Use different placement policies for different pools (e.g., look at GPU consumption rather then CPU/memory)
‒ Support instance management for BM
• Actions for BM instances may differ from actions on VMs
• E.g., implement: Console, View log. Explore applicability of: Pause, Suspend, Resize
‒ Support aggregates for specific BM nodes
‒ Add hierarchical relations between BM instance and the compute node running on it
• E.g., Propagate delete command to the VM level as well (mark VMs as deleted when deleting the BM instance).
32 © 2014 IBM Corporation
Challenges Examples (Cont.)
� Image Management
‒ Enhance access control for images in Glance
• E.g., Don’t expose Ironic’s RAM disk images to users
‒ Support run-time image selection
• E.g., Display single instance of the image
‒ Support dynamic adaptation of images to match target pool type
� UI: extend UI to support Hybrid applications/instances
‒ Add bare-metal node management in Horizon
• Currently can be done through CLI (e.g., ironic node- create/delete)
‒ Physical machines are shown as Hypervisors
33
Hybrid Application example
34 © 2014 IBM Corporation
Example – Deploying Hybrid WordPress
� Apache+WordPress on VM and MySQL on physical node
� Setup and configuration‒ A single HEAT template used for the two nodes
‒ Using host aggregates for scheduling
‒ Used diskimage-builder to build the baremetal images• cloud-init element installed
35 © 2014 IBM Corporation
Example – Deploying Hybrid WordPress
� Example gaps reflected in Horizon:‒ BM nodes are shown as Hypervisors
‒ Deployment images are exposed to users
36
Summary
37 © 2014 IBM Corporation
Summary
� Heterogeneous cloud environments are gaining momentum
‒ Critical in order to host a broad spectrum of applications
� OpenStack is a promising solution to manage hybrid clouds
‒ By using integrated management we can have
• Simple native OpenStack solution
• Simplified administration and enhanced user experience
� Need a careful design to ensure all requirements (e.g., security, isolation) are maintained across the hybrid environment
� Some gaps need to be addressed in OpenStack
38 © 2014 IBM Corporation
Be sure to stop by the IBM booth to see some demos and get your rockin’ OpenStack t-shirt while they last.
Don’t miss Monday evening’s booth crawl where you can enjoy Atlanta’s own SWEET WATER IPA!
Thank you !
39 © 2014 IBM Corporation
Q & A
40 © 2014 IBM Corporation
Monday, May 12 – Room B314
12:05-12:45
Wednesday, May 14 - Room B312
9:00-9:40
9:50-10:30
11:00-11:40
11:50-12:30
OpenStack is Rockin’ the OpenCloud Movement! Who‘s Next to Join the Band ?Angel Diaz, VP Open Technology and Cloud LabsDavid Lindquist, IBM Fellow, VP, CTO Cloud & Smarter Infrastructure
Getting from enterprise ready to enterprise bliss - why OpenStack and IBM is a match made in Cloud heaven. Todd Moore - Director, Open Technologies and Partnerships
Taking OpenStack beyond Infrastructure with IBM SmartCloudOrchestrator.Andrew Trossman - Distinguished Engineer, IBM Common Cloud Stack and SmartCloud Orchestrator
IBM, SoftLayer and OpenStack - present and futureMichael Fork - Cloud Architect
IBM and OpenStack: Enabling Enterprise Cloud Solutions Now.
Tammy Van Hove -Distinguished Engineer, Software Defined Systems
IBM Sponsored Sessions
41 © 2014 IBM Corporation
Monday, May 12
3:40 - 4:20
3:40 - 4:20
Tuesday, May 13
11:15 - 11:55
2:00 - 2:40
5:30 - 6:10
5:30 - 6:10
Wednesday, May14
9:50 - 10:30
2:40 - 3:20
Thursday, May 15
9:50 - 10:30
1:30 - 2:10
2:20 - 3:00
IBM Technical Sessions