2013 05-openstack-israel-heat

transcript

What’s Up Next for Heat?

Alex Heneveld / @ahtweetinOpenStack Israel27 May 2013

CloudIt’s not just about VM’s

It’s about storage, networking, maas, ...

It’s not just about VM’s

Ultimately it’s about applications

CloudIt’s not just about VM’s

It makes the clouds rise

Declarative — OpenStack Heat (Grizzly) and AWS CloudFormationProcedural — Apache Whirr, Gigaspaces Cloudify, Cloudsoft BrooklynOpenStack Summit — Red Hat, RAX, IBM, HP, Cloudsoft, OASIS TOSCA and CAMPDSLs in Progress — CAMP PDP and OpenStack Heat (Havana)

Approaches to Applications in the Cloud

AWS CloudFormation

{ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "WordPress is web software …", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair for SSH access", "Type" : "String" }, ... }, "Mappings" : { "AWSInstanceType2Arch" : { "m1.small" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, ... }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } }, "Resources" : { "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "gcc-c++" : [], "make" : [],

"Resources" : { "WebServer": { "Type": "AWS::EC2::Instance"

AWS CloudFormation

Parameters

Mappings(config metadata)

WaitConditions &WaitHandles

(orchestration)

Outputs

Functions

AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"

"Type": "AWS::EC2::EIP"

"Type": "AWS::SQS::Queue"

"Type": "AWS::CloudFormation::Stack"

"Type": "AWS::S3::Bucket"

AWS CloudFormation

"Type": "AWS::AutoScaling::AutoScalingGroup"

"Type": "AWS::EC2::EIP"

"Type": "AWS::SQS::Queue"

"Type": "AWS::CloudFormation::Stack"

"Type": "AWS::S3::Bucket"

AWS CloudFormation

Composition!

— Limited to AWS cloud

— Limited to AWS concepts

— Cumbersome to write

— Limited dependency injection

Nice, but:

AWS CloudFormation

— CloudFormation clone

— Targetting OpenStack resources and AWS compatibility

— Adds YAML support (comments and easier)

— Multi-cloud support using Apache Deltacloud

OpenStack Heat

(AWS CloudFormation){ "AWSTemplateFormatVersion" : "2010-09-09", "Description" : "WordPress is web software …", "Parameters" : { "KeyName" : { "Description" : "Name of an existing EC2 KeyPair for SSH access", "Type" : "String" }, ... }, "Mappings" : { "AWSInstanceType2Arch" : { "m1.small" : { "Arch" : "64" }, "m1.large" : { "Arch" : "64" }, ... }, "AWSRegionArch2AMI" : { "us-east-1" : { "32" : "ami-31814f58", "64" : "ami-1b814f72" }, ... } }, "Resources" : { "WebServer": { "Type": "AWS::EC2::Instance", "Metadata" : { "AWS::CloudFormation::Init" : { "config" : { "packages" : { "yum" : { "gcc-c++" : [], "make" : [],

OpenStack Heat

HeatTemplateFormatVersion: 2012-12-12 Description: WordPress is web software ... Parameters: KeyName: Description: Name of an existing EC2 KeyPair for SSH access Type: String ... Mappings: AWSInstanceType2Arch: m1.small: {Arch: '32'} m1.large: {Arch: '64'} ... DistroArch2AMI: F18: {'32': F18-i386-cfntools, '64': F18-x86_64-cfntools}, ... Resources: WebServer: Type: AWS::EC2::Instance Metadata: AWS::CloudFormation::Init: config: packages: yum: gcc-c++: [] make: []

— CloudFormation clone

— Targetting OpenStack resources and AWS compatibility

— Adds YAML support (comments and easier)

— Multi-cloud support using Apache Deltacloud

OpenStack Heat

Available NOW in Grizzly!

OpenStack Heat

— Still limited to selected clouds

— Still limited in concepts

— Only slightly less cumbersome to write

— Very limited dependency injection

But...

Procedural Approach

10 INPUT "What app do you like? ", A$20 INPUT "What cloud do you like? ", C$30 GOSUB 10040 END

100 REM TODO deploy C$ to A$110 RETURN

Procedural Approach

— Millions of lines of bash, chef, python, ruby, java, ... or ...

— Apache Whirr

— Gigaspaces Cloudify

— Brooklyn and Cloudsoft AMP

Apache Whirr

— Apache top level project

— Deploy multi-node applications— Hadoop, Hama, Mahout, web,

Cassandra & many more

— Define topologies declaratively— Add new roles, configuration,

constraints and wiring procedurally

provision

install

configure

manage

Apache Whirr

$ whirr launch-cluster \ --cluster-name=hbase \ --instance-templates='1 hbase-master,6 hbase-regionserver'

Apache Whirr

spec = new ClusterSpec();spec.setProvider("cloudservers-uk");spec.setIdentity(apikey);spec.setCredential(secret);spec.setClusterName("hbase");spec.setInstanceTemplates(ImmutableList.of( new InstanceTemplate(1, "hbase-master"), new InstanceTemplate(6, "hbase-regionserver")));

cluster = new ClusterController().launchCluster(spec);

— Limited parameterisation

— Defining new roles requires coding

— Strict phases limit applicability

But...

— Composable to one dimension

— Portable locations support

— Pluggable with bash, Chef, Puppet, and more

Apache Whirr

Gigaspaces Cloudify

— Apache-licensed Open PaaS Stack— Deployment and Management

— Build applications declaratively— Define services procedurally— Configure auto-scaling declaratively

Gigaspaces Cloudify

application { name="petclinic" service { name = "mysql" } service { name = "tomcat" dependsOn = ["mysql"] } }

Gigaspaces Cloudify

service { name "mysql" icon "mysql.jpg" type "DATABASE"

lifecycle{ install "mysql_install.groovy" start "mysql_start.groovy" ... }}

Gigaspaces Cloudify

def installLinuxMysql(config,context,builder,currVendor,installScript,myCnfObject) {" builder.sequential {" " chmod(dir:"${context.serviceDirectory}", perm:"+x", includes:"*.sh")" " exec(executable: "${context.serviceDirectory}/${installScript}",failonerror: "true")}...switch (currVendor) { case ["Ubuntu", "Debian", "Mint"]: context.attributes.thisInstance["binFolder"]="/usr/bin" installLinuxMysql(config,context,builder,currVendor,"installOnUbuntu.sh",myCnfObject) ...

Gigaspaces Cloudify

if sudo grep -q -E '[^!]requiretty' /etc/sudoers; then echo "Defaults:`whoami` !requiretty" | sudo tee /etc/sudoers.d/`whoami` >/dev/null sudo chmod 0440 /etc/sudoers.d/`whoami`fisudo apt-get -y -q update || error_exit $? "Failed on: sudo apt-get -y update"killMySqlProcesssudo apt-get -y -q purge mysql-client* mysql-server* mysql-common*sudo rm -rf /etc/mysql || error_exit $? "Failed on: sudo rm -rf /etc/mysql"sudo DEBIAN_FRONTEND='noninteractive' apt-get -o Dpkg::Options::='--force-confnew' -q -y \ install mysql-server-core mysql-server mysql-client mysql-common || error_exit $? "Failed ... ”

Gigaspaces Cloudify

scalingRule { serviceStatistics { metric "Requests per second" movingTimeRangeInSeconds 20 }

highThreshold { value 100 instancesIncrease 1 } }

— Defining new services requires coding

But...

— Composable to one dimension

— Parameterisable recipes

— Portable locations support

— Pluggable with bash, Chef, Puppet, and more

Gigaspaces Cloudify

Brooklyn & Cloudsoft AMP

— Brooklyn = Apache-licensed open-source

— AMP = Enterprise + Support

— Deployment and Management of Anything

— Hierarchical Autonomic Policies

— Define blueprints procedurally

Brooklyn

Contention ManagementCustom PaaSBig Data

public class MyWebCluster extends AbstractApplication implements MyWebClusterConstants { public void init() { MySqlNode mysql = addChild(EntitySpecs.spec(MySqlNode.class) .configure("creationScriptUrl", DB_SETUP_SQL_URL)); ControlledDynamicWebAppCluster web = addChild(EntitySpecs.spec( ControlledDynamicWebAppCluster.class) .configure(WebAppService.HTTP_PORT, PortRanges.fromString("8080+")) .configure(JavaWebAppService.ROOT_WAR, WAR_PATH) .configure(javaSysProp("brooklyn.example.db.url"), formatString("jdbc:%s%s?user=%s\\&password=%s", attributeWhenReady(mysql, MySqlNode.MYSQL_URL), DB_TABLE, DB_USERNAME, DB_PASSWORD)). .configure(JcloudsLocationConfig.MIN_RAM, 2048) .configure(JcloudsLocationConfig.TEMPLATE_SPEC, "osFamily=centos,os64Bit=true") ); }

Brooklyn: Nested Elastic Blueprints

public class MyWebCluster extends AbstractApplication implements MyWebClusterConstants { public void init() { MySqlNode mysql = ... ControlledDynamicWebAppCluster web = ... web.addEnricher(HttpLatencyDetector.builder(). url(ROOT_URL). rollup(10, SECONDS).build());

web.getCluster().addPolicy(AutoScalerPolicy.builder(). metric(REQUESTS_PER_SECOND_IN_WINDOW_PER_NODE). metricRange(10, 100).sizeRange(2, 5).build());

addEnricher(SensorPropagatingEnricher.newInstanceListeningTo(web, ROOT_URL, REQUESTS_PER_SECOND_IN_WINDOW, REQUEST_LATENCY_IN_SECONDS_IN_WINDOW)); }

Brooklyn: Policies, Enrichers and KPI’s

Brooklyn

— Defining blueprints requires coding

But...

— Composable and substitutable

— Parameterisable blueprints

— Portable and powerful locations support

— Pluggable with VM images, Heat, Chef, more

Applications

— Components (IaaS-resources)

— DeclarativeInfrastructure

— Procedural

— Extensible (new resources)

— Portable (hybrid, metal)

— Management

— Composable (stacks)

Heat in Grizzly

thanks largely to Red Hat

Portland Design Summit — Heat

lelots of new interest and ideas

leingredients from Brooklyn, Whirr, OoO and others

nga special ingredient from Rackspace

a special ingredient from IBM

Heat in Havana

ConcurrentDistributedResilientFunctions / DI

Heat in Havana

ConcurrentDistributedResilientFunctions / DIStandards ?

OasisTOSCA

http://www.oasis-open.org/committees/tosca/

Topology and Orchestration Specification for Cloud Applications

Modeling Topologies with TOSCA Service Topologies are described using the TOSCA “Meta-model”:

Artifacts Describe Installables and Executables required to

instantiate and manage a service. Currently, they include:

Implementation Artifacts: – Executables or Plans that implement a Node’s

or Relationship’s Operations (e.g. a Bash script) Deployment Artifacts:

– Installables of the components (e.g. a TAR file)

A service’s Topology Model is included in a TOSCA Service Template which is packaged and shared, along with all dependent artifacts, as a TOSCA Cloud Service Archive (CSAR)

Service Templates Group the nodes and relationships that make up

a service’s topology – Allowing modeling of sub-topologies

Service Templates “look like nodes” enabling: Composition of applications from one or more

service templates Substitution of abstract Node types with

available service templates of the same type

Nodes Represent Components of an application or service

and their Properties. Example nodes include: – Infrastructure: Compute, Network, Storage, etc. – Platform: OS, VM, DB, Web Server, etc. – Granular: functional Libraries, Modules, etc.

Include Operations which are the management functions for the node – e.g. deploy(), start(), stop(), connect(), etc.

Export their dependencies on other nodes as Requirement and Capabilities

Relationships Represent the logical Relationships between

nodes – e.g. “hostedOn”, “connectsTo”, etc.

Describes the valid Source and Target nodes they are designed to couple – e.g. source “web application” node is designed

to “connectTo” a target “database” node Have their own Properties and Constraints

Slide origin: IBM

TOSCA: Modelling Topologies

<Definitions id="PayrollDefinitions" targetNamespace="http://www.example.com/ste" xmlns:pay="http://www.example.com/ste/Types"> <Import namespace="http://www.example.com/ste/Types" location="http://.../PayrollTypes.tosca" importType=" http://docs.oasis-open.org/tosca/ns/2011/12"/> <ServiceTemplate id="Payroll" name="Payroll Service Template"> <TopologyTemplate ID="PayrollTemplate"> <NodeTemplate id="Payroll Application" type="pay:ApplicationNodeType"> <DeploymentArtifacts> <DeploymentArtifact name="PayrollEAR" type="http://..../CSARref"> EARs/Payroll.ear </DeploymentArtifact> </DeploymentArtifacts> </NodeTemplate> <NodeTemplate id="Application Server" type="pay:ApplicationServerNodeType"> <DeploymentArtifacts> <DeploymentArtifact name="ApplicationServerImage" type="http://.../AMIref"> ami-edf2cf99 </DeploymentArtifact> </DeploymentArtifacts> </NodeTemplate> <RelationshipTemplate id="deployed_on" type="pay:deployed_on"> <SourceElement ref="Payroll Application"/> <TargetElement ref="Application Server"/> </RelationshipTemplate> </TopologyTemplate> </ServiceTemplate></Definitions>

TOSCA: Sample Definition

— Fails the “ease-of-use” test

— Does not standardise an API

But...

— Powerful and broad modelling coverage

— Workflow, locations, substitution

— Relatively mature

OasisCAMP

www.cloudspecs.org

Cloud Application Management Platform

http://www.oasis-open.org/committees/camp/

CAMP: Objectives and Non-Objectives

— Software

— an Orchestration Spec

— a Modelling System

But CAMP is NOT

— a REST API

— for Deployment and Management

— of Applications

It provides a flexible way to use many platforms.Interoperability.

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

TemplatePlatformComponent

Template

RequirementType

App ComponentType

Clients Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

PDP = Platform Deployment Plan

(blueprint)

— Deployment PlanCAMP: Key Concepts

DSL & REST

App ComponentType

(blueprint)

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— Creating Templates

App ComponentTemplate

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— ResolutionCAMP: Key Concepts

AssemblyTemplate(blueprint)

(blueprint)POST

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— Resolution

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— Resolution

AssemblyTemplate

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

PlatformComponent

Template

RequirementType

App ComponentType

— Deployment

Assembly(blueprint) Platform

Component

PlatformComponent

App Component

CAMP: Key Concepts

Clients

Platform

PlatformComponent

Template

RequirementType

App ComponentType

— Deployment

PlatformComponent

Template

PlatformComponent

Template

Component

PlatformComponent

App Component

CAMP: Key Concepts

Clients

Platform

— Deployment

Component

PlatformComponent

App Component

PlatformComponent

Template

PlatformComponent

Template

Assembly

— depends on orchestrations supporting this

— wants standardised requirement and component types

— still needs a nice DSL

But...

— all very general

— designed to map on to a wide range of orchestrationsPaaS, TOSCA, Brooklyn, Heat

— OSS code in progressgithub.com/brooklyncentral/camp-server

CAMP: PDP/Blueprint DSL (work in progress)

Parameters

Components

Requirements

Functionsparameters & sensors

Outputs

CAMP: PDP/Blueprint DSL (work in progress)

Parameters

Functionsparameters & sensors

Outputs

components: my_app: type: WAR content: hello.war requirements: hosted_on: javaVersion: [1.7,)

Mappings(config metadata)

WaitConditions &WaitHandles

(orchestration)

Parameters

components: my_instance: type: server ram: 2048m requires: connects_to: target: my_database

Heat: New HOT DSL (also work in progress)

Functions

Outputs

HeatOrchestration

Template

Coming inHavana

Heat in Havana

ConcurrentDistributedResilientFunctions / DI

Wrap-Up

It’s about the apps.Come get involved!

Declarative — OpenStack Heat (Grizzly) and AWS CloudFormationProcedural — Apache Whirr, Gigaspaces Cloudify, Cloudsoft BrooklynOpenStack Summit — Red Hat, RAX, IBM, HP, Cloudsoft, OASIS TOSCA and CAMPDSLs in Progress — CAMP PDP and OpenStack Heat (Havana)

Thank you.alex.heneveld@cloudsoftcorp.com

@ahtweetin

2013 05-openstack-israel-heat

Technology