Date post: | 16-Apr-2017 |
Category: |
Science |
Upload: | cloudlightning |
View: | 278 times |
Download: | 2 times |
SELF-ORGANISING, SELF-MANAGING HETEROGENEOUS CLOUD
OVERVIEW
1. Funding and Challenge
2. Consortium
3. Current IaaS Cloud Usage
4. Objectives
5. Approach
6. CloudLightning Architecture
7. HPC Use Cases
8. Challenges Ahead
HORIZON2020
Horizon 2020 is the biggest EU Research and Innovation
programme ever with nearly €80 billion of funding available over 7 years (2014 to 2020) – in addition to the private investment that this
money will attract.
It promises more breakthroughs, discoveries and world-firsts by taking
great ideas from the lab to the market.
The principal goals:
• world-class science• removal of barriers to innovation• enable public and private sectors to work together
The CloudLightning project was funded under Call H2020-ICT-2014-1Advanced Cloud Infrastructures and Services High performanceheterogeneous cloud infrastructures and runs from Feb 2014 - January 2017
SPECIFICCHALLENGE
The aim is to develop infrastructures, methods
and tools for high performance, adaptive cloud applications and
services that go beyond the current capabilities.
Cloud computing is being transformed by new requirements such as
• heterogeneity of resources and devices,• software-defined data centres,• cloud networking, security, and • the rising demands for better quality of user experience.
Cloud computing research will be oriented towards
• new computational and data management models (at both infrastructure and services levels) that respond to the advent of faster and more efficient machines,
• rising heterogeneity of access modes and devices, • demand for low energy solutions, • widespread use of big data, • federated clouds and • secure multi-actor environments including public administrations.
CONSORTIUM
CloudLightning comprises of eight partners from academia
and industry and is coordinated by University
College Cork.
CURRENT IAAS CLOUD USAGE
Consider the two actors: the customer, looking to
build a solution on provider’s infrastructure,
and the cloud service provider
Customer:
• Hard work • Research various offerings
and build/compile solutions accordingly.
• Sub-optimal• Create a generic solution
to facilitate portability• Opt for provider specific
offering and risk vendor lock-in
Provider:
• Relinquishes control• Over resource utilization
and power management• The cloud is now
approaching 10% of the world’s electricity consumption!
• Offers Resources in limited, discrete sizes
• Precipitates over-provisions and exacerbates waste
PROJECT OBJECTIVES
Customer Level Objectives
• Make cloud computing more accessible
• Make cloud computing more efficient
• Move towards “ease of everything”
Provider level objectives
• Re-establish control over their IaaSofferings
• Facilitate better power management
• Enable fast resource provisioning for quicker service initiation
• Enable seamless exploitation of heterogeneous hardware
• Exploit faster and cheaper service delivery offered by hardware accelerators
• Employ different heterogeneous hardware types for different services or for different invocations of the same service
Project level objectives
• Demonstrate our approach in a very challenging HPC application domain
• Construct small-scale test-bed
• Construct Large-scale simulation
OUR APPROACH
Separate the concerns of the customer and the
provider
1. Create a Service Orient Architecture for the Heterogeneous Cloud
2. Customer focuses on service requirements, workflows and SLAs rather than resources
3. Provider focuses on efficient resource management and service delivery
SERVICE ORIENTED
ARCHITECTURE
This approach moves the management burden
from the customer to the provider.
The resulting complexity for the
provider is very high.
Creator forms the work-flow and stores the Blueprint in the Blueprint Catalogue; the Operator selects a Blueprint from the Blueprint Catalogue and optionally edits its constraints and parameters.
The Operator launches the Blueprint by:(1) requesting an appropriate solution from the CL and(2) deploying the Blueprint on the resources returned as part of that solution.
The End User then interacts with the deployed Blueprint.
SELF-ORGANISATIONAND
SELF-MANAGEMENT
Moving from a distributed customer-based IaaS
Management to centralised provider-based IaaS
management introduces enormous complexity.
This complexity can be addressed by self-organization
and self-management.
Basic tenets:• component autonomy• awareness of the environment• goal-driven behaviour of individual components• self-configuration
Goals include:• minimize energy consumption• Improve service delivery
Goals achieved by collaboration.
Coalitions of resources, working in concert to respond to the needs of a specific service request rather than offering a menu of a limited number of resource packages.
CONCEPTUAL ARCHITECTURE A Cell is a collection of resources.
There may be multiple CellsA Cell Manager is associated witheach CellThe Cell resources are grouped into anumber virtual racks, called vRacks.A vRack Manager is associated witheach vRack. It is self-managed,identifying and creating coalitions, todeliver on specific service requests.vRack Managers cooperate to formvRack Manager Groups.vRacks Managers in the same Groupself-organize to meet specificobjectives. They are aware ofchanges in the environment includingnew and disappearing resources andadapt, on a negotiated basis, withother vRacks Managers within thesame vRack Manager Group to meetsystem objectives.
ARCHITECTURE COMPONENTS
CLOUDLIGHTNING RESOURCES • In a heterogeneous cloud, there will be many different types of
compute resources.• These resources may be available individually or they may be
bundled into subsystems. • Individual resources and subsystems may have pre-installed
software stacks• They may be physically located on interconnects with different
characteristics• A CL-Resource is a generic term used to refer to any of the
above• CL-Resources can thus be bare metal; virtual machines;
containers; networked commodity or specialized hardware, servers with accelerators such as GPUs, MICs and FPGAs; pre-built HPC environments
• In response to a service request, the CL system identifies specific CL-Resources to be used for the delivery of that service.
• Workflows of services may be implemented on a mixture of CL-Resources – one resource type per service
RESOURCE COALITION
A collection of CL-Resources used to execute a service is called aCoalition. A coalition may be composed of one or more CL-Resources of the same type.Multi CL-Resource coalitions support multi-process services.Coalitions are formed by a vRack Manager in response to specific service requirements. Coalitions may be persisted to eliminate delays in CL-Resource creation and so to improve service deliveryDynamic Coalition formation
respects SLA requirements minimizes provider overheadsmaximizes resource utilization
The constituent CL-Resources of a Coalition
may span multiple servers within a single
vRack.
VRACK MANAGER TYPES AND
GROUPS
vRack Managers are typed to • reflect differences in the CL-Resources under their control• constrain how vRack Manager Groups are formed and self-
organized• leverage resource specific optimization opportunities
resulting from grouping vRack Managers together
PLUG AND PLAY
LEVERAGING EXISTING
OPENSTACK COMPONENTS
BENEFICIARIES
The primary beneficiary is the Infrastructure-as-a-Service provider. They benefit from activating
the HPC in the cloud market and a reduction in
cost related to better performance per cost and
performance per watt.
This increased energy efficiency can result in
lower costs throughout the cloud ecosystem and
can increase the accessibility and
performance in a wide range of use cases
including Oil and Gas discovery, Genomics and
Ray Tracing (e.g. 3D Image Rendering)
• Improved physics simulations and higher resolution RTM imaging.
• Energy and cost efficient scalable solution for RTM and OPM/DUNE simulations.
• Reduced risk and costs of dry exploratory wells.
• Improved performance/cost and performance/Watt.
• Faster speed of genome sequence computation.
• Reduced development times.
• Increased volume and quality of related research.
• Reduced CAPEX and IT associated costs.
• Extra capacity for overflow (“surge”) workloads.
• Faster workload processing to meet project timelines.
Ray Tracing (3D Image Rendering)
GenomicsOil and Gas
IN CONCLUSION The Challenges Ahead
• Separate the concerns of the IaaS consumer and the CSP
• Create a Service Oriented Architecture for the emerging heterogeneous cloud
• Reduce energy consumption by improved IaaS management
• Improve service delivery
• Leverage heterogeneity
• Bring HPC to the cloud
• Resource management in hyper-scale cloud deployments
THANK YOUTHANK YOUwww.cloudlightning.eu