Post on 22-Mar-2017
transcript
1© Copyright 2016 EMC Corporation. All rights reserved.
2© Copyright 2016 EMC Corporation. All rights reserved. 2© Copyright 2016 EMC Corporation. All rights reserved.
INTRODUCTION TOMESOS AND MESOSPHEREMAY 2016
3© Copyright 2016 EMC Corporation. All rights reserved.
• Somik Behera, DCOS Product Manager - Mesosphere – Twitter: @strikesme– Blog: http://strikesme.com
• David vonThenen, Developer Advocate - EMC {code}– Twitter: @dvonthenen– Blog: http://dvonthenen.com
PRESENTERS
4© Copyright 2016 EMC Corporation. All rights reserved.
• What is Mesos?• What will Mesos do for me?• Who is Mesosphere?• How EMC{code} fits with Mesos• Demo• Q&A
AGENDA
5© Copyright 2016 EMC Corporation. All rights reserved. 5© Copyright 2016 EMC Corporation. All rights reserved.
WHAT IS MESOS?
6© Copyright 2016 EMC Corporation. All rights reserved.
• From a resource perspective, it’s a cluster manager:– Pools servers resources to be centrally managed as a single
unit• From an application perspective, it’s a scheduler:
– Dispatches workloads to consume pooled resources• Often described as a Data Center Operating System
(DCOS)
MESOS IS…
7© Copyright 2016 EMC Corporation. All rights reserved.
• Master– Manages Mesos Agents– Enables fine-grained sharing of resources (CPU, RAM, …)
across frameworks in the form of resource offers.• Zookeeper
– Enables highly reliable distributed coordination (cluster state)
• Agent (aka Slave)– The entity that will be running the task
MESOS COMPONENTS
8© Copyright 2016 EMC Corporation. All rights reserved.
MESOS ARCHITECTURE
9© Copyright 2016 EMC Corporation. All rights reserved.
EFFICIENT RESOURCE UTILIZATION
10© Copyright 2016 EMC Corporation. All rights reserved.
WHO DOESN’T LIKE EFFICIENCY?
11© Copyright 2016 EMC Corporation. All rights reserved.
• Generic– Anything that can be run on a standard Linux shell– Provides a mechanism for deploying applications packages– Supports Docker workloads
• Framework– A plug-in to Mesos that allows you to implement a sub-
scheduler– specialize a workload by providing custom placement and
resource allocation
APPLICATION SUPPORT
12© Copyright 2016 EMC Corporation. All rights reserved.
GENERIC APPLICATIONS
13© Copyright 2016 EMC Corporation. All rights reserved.
MESOS FRAMEWORKS
14© Copyright 2016 EMC Corporation. All rights reserved.
BRINGING EVERYTHING TOGETHER
15© Copyright 2016 EMC Corporation. All rights reserved. 15© Copyright 2016 EMC Corporation. All rights reserved.
WHAT WILL MESOS DO FOR ME?
16© Copyright 2016 EMC Corporation. All rights reserved.
• Bare metal x86 hardware• VMs (your favorite hypervisor)• Public, Private, Hybrid Cloud
MULTIPLE DEPLOYMENT VECTORS
17© Copyright 2016 EMC Corporation. All rights reserved.
MESOS WILL RUN ON…
18© Copyright 2016 EMC Corporation. All rights reserved.
• Heterogeneous nodes (different CPU models, memory footprints, etc)• Supports elastic scaling
– Nodes can be added for capacity– Nodes can be removed for maintenance
CLUSTER NODES
19© Copyright 2016 EMC Corporation. All rights reserved.
• Has been in production use for 3+ years• 10k+ node scale• Recognized by users and the community• Tasks dispatching scale linearly over # nodes (2010)
STABILITY AND PERFORMANCE
20© Copyright 2016 EMC Corporation. All rights reserved.
POWERED BY MESOS
21© Copyright 2016 EMC Corporation. All rights reserved.
• Marathon– manages long-running applications.
Provides both a UI and REST API• Chronos
– is a replacement for cron to schedule batch jobs and other workloads.
• Mesos-DNS– enables DNS based service discovery
• Etc, etc, etc…
RICH APPLICATIONS AND ECOSYSTEM
22© Copyright 2016 EMC Corporation. All rights reserved.
• Runs a wide variety of workloads• On commodity hardware• Efficient utilization of resources• High Availability / Fault Tolerant• UI and API management
WHAT WILL MESOS DO FOR ME?
23© Copyright 2016 EMC Corporation. All rights reserved.
OR TRY TO BUILD ONE FROM SCRATCH
24© Copyright 2016 EMC Corporation. All rights reserved. 24© Copyright 2016 EMC Corporation. All rights reserved.
WHO IS MESOSPHERE?
25© Copyright 2016 EMC Corporation. All rights reserved.
MESOSPHERE: BORN OF THE HYPERSCALE ERA
Mesosphere is well funded$50M by Tier 1 investors: Andreessen Horowitz & Khosla Ventures
2009 2013 2014 2015
First Mesosphere DCOS lighthouse customersTens of thousands of containers launched
Expanded OperationsSan Francisco (2013)Hamburg (2014)New York (2015)Berlin (2015)
Mesosphere FoundedKey engineering leaders from Twitter, Airbnb - companies behind open-source tech
Apache Mesos built at UC Berkeley
● Core technology from AMPlab
● Corporate large-scale production deployments begin (e.g., Apple, Twitter, Salesforce)
Growing Ecosystem & DCOS General AvailabilityNow a breeze to install modern app services (e.g., Hadoop, Spark, Cassandra)
26© Copyright 2016 EMC Corporation. All rights reserved.
DELIVERING OUTCOMES FOR ENTERPRISES
DEVELOPER AGILITY
HYPERSCALE OPERATIONS
DATA AGILITY
Highly Efficient & Scalable IT
Distributed Framework Pooling
Container Orchestration
Cont. Integration/Delivery
Developer Cloud
Self-Service Container Runtime
Big Data
Internet Of Things
Reactive applications
27© Copyright 2016 EMC Corporation. All rights reserved.
THE DATACENTER IS THE NEW SERVER
PHYSICAL (x86) VIRTUAL HYPERSCALEMAINFRAME
SERVER VIRTUAL MACHINE
PARTITION (LPAR)
UNIT OF INTERACTION
● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER
● LINUX, WINDOWS
● DATA / TRANSACTION PROCESSING
● UNIX, IBM OS/360DEFINITIVE
APPS AND OS
● ERP, CRM, PRODUCTIVITY, MAIL & WEB SERVER
● HYPERVISOR + GUEST OS
● BIG DATA, INTERNET OF THINGS, MOBILE APPS
● ???
???DATACENTER
NEW FORM FACTOR FOR DEVELOPING AND
RUNNING APPS
● BIG DATA, INTERNET OF THINGS, MOBILE APPS
● DATACENTER OPERATING SYSTEM (DCOS)
28© Copyright 2016 EMC Corporation. All rights reserved.
MESOSPHERE DCOS ACCELERATES ENTERPRISES TO A MODERN CLOUD STACK WITH THE DATACENTER FORM
Datacenter Operating System (DCOS)
Distributed Systems Kernel (Apache Mesos)
Big Data + Analytics EnginesMicroservices (in containers)
Streaming
Batch
Machine Learning
Analytics
Functions & Logic
Search
Time Series
SQL / NoSQL
Databases
Modern Cloud App Components
Distributed systems kernel to abstract resources
Killer “apps” ecosystem
Cont Integration/Cont Delivery
Framework “app” architecture to consistently run on top of kernel
User Interface (GUI & CLI)
Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage)
Any Infrastructure (Physical, Virtual, Cloud)
29© Copyright 2016 EMC Corporation. All rights reserved.
DCOS BRINGS HYPERSCALE TO ENTERPRISES
Desktop Apps (e.g., Safari, Adobe Photoshop, Itunes)
GUI (Aqua)/CLI
Personal computer
- App store - OpenGL - Advanced UI gestures
BSD Unix
Mobile Apps (e.g., Spotify, Evernote, WhatsApp)
Android GUI
Mobile devices
- Telephony Manager- Battery management- External storage support
Linux
Distributed systems (e.g., Docker, Spark, Hadoop, Cassandra)
GUI/CLI (DCOS CLI)
Full Datacenter / Cloud
- Container orchestration- Distributed batch jobs- Persistent storage mgmt
Apache MesosKernel
Applications
Form Factor
User Interface
OS Services(highlights)
Computing for the masses Mobile apps for the masses Datacenter-scale apps for mainstream enterprises
Benefits
30© Copyright 2016 EMC Corporation. All rights reserved. 30© Copyright 2016 EMC Corporation. All rights reserved.
HOW EMC {CODE} FITS WITH MESOS
31© Copyright 2016 EMC Corporation. All rights reserved.
• Containers are ephemeral therefore stateless• No state? Means boring Application• Container vendors started introducing state by way
of local disks• Something better?• EMC {code} brings storage expertise to the table
HOW EMC {CODE} FITS WITH MESOS?
32© Copyright 2016 EMC Corporation. All rights reserved.
• Enable external storage• Enhance frameworks to embrace and consume
external storage• Run EMC storage platforms as scale out data services
EMC {CODE} GOALS WITH MESOS
33© Copyright 2016 EMC Corporation. All rights reserved.
• Persistent– Containers can come and go,
but continue where they left off– Application Data, Logs, Audit
Trail, etc• External
– Storage lives outside the Mesos Agent node
– High Availability– Tolerates node failure
PERSISTENT EXTERNAL STORAGE
34© Copyright 2016 EMC Corporation. All rights reserved.
• REX-Ray– Provides a vendor agnostic storage orchestration engine– AWS, GCE, ScaleIO, XtremIO, many many more– Github: https://github.com/emccode/rexray
• mesos-module-dvdi– Provides a hook to Mesos agent nodes to provision, attach,
and detach external storage– Github: https://github.com/emccode/mesos-module-dvdi
HOW DO WE ACHIEVE THIS?
35© Copyright 2016 EMC Corporation. All rights reserved.
• We can run “real world” applications• Persistence – DBs• Highly Available, Fault Tolerant• Audit trail – Transaction Logs• Not worry about the “2am call”
STATELESS TO STATEFUL
36© Copyright 2016 EMC Corporation. All rights reserved.
THE 2AM CALL…
37© Copyright 2016 EMC Corporation. All rights reserved. 37© Copyright 2016 EMC Corporation. All rights reserved.
DEMO
38© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 1: TWEETER - ARCHITECTURE
Datacenter Operating System (DCOS)
Containerized Microservice
Kafka Distributed Messaging system
Cassandra Distributed Datastore
Spark Data Analytics Jobs
Distributed Software Load Balancers
“Tweeter” Service
39© Copyright 2016 EMC Corporation. All rights reserved.
• Running on Amazon EC2– Mesos– Marathon– Mesos DNS
• Production HA Cluster– 3 x Master + Zookeeper Nodes (t2.micros)– 4 x Slave Nodes with Docker Support (t2.medium)
DEMO 2: CONFIGURATION
40© Copyright 2016 EMC Corporation. All rights reserved.
• Production Website (MVC)– Web UI (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restclient– REST API (Golang)
• Github: https://github.com/dvonthenen/goprojects/tree/master/src/restserver– PostgreSQL (Docker Container) backed by Persistent
External Storage• Using REX-Ray• Using mesos-module-dvdi
• Simple Web App - User Account Management
DEMO 2: PRODUCTION WEBSITE
41© Copyright 2016 EMC Corporation. All rights reserved.
• Deploy Model, View, Controller using REST API– Observe creation of each component– Observe PostgreSQL DB attaching to EBS volumes– Using browser, log into Account Management UI and
add/delete records– Observe data from Web UI being added to PostgreSQL table
• Simulate Failure– Kill EC2 Instance– Watch task recreation and reattach to EBS volumes
DEMO 2: PRODUCTION WEBSITE
42© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 2: INITIAL DEPLOY
43© Copyright 2016 EMC Corporation. All rights reserved.
DEMO 2: SCALE IT UP
44© Copyright 2016 EMC Corporation. All rights reserved.
Data Persistence in the New Container WorldWednesday 3PM
Joshua BernsteinVP of Technology for ETD
Tobi Knaup CEO & Co-Founder of Mesosphere
Guru Session
45© Copyright 2016 EMC Corporation. All rights reserved.
@EMCcode@strikesme@dvonthenen
emccode.comcommunity.emccode.com
Come visit us at Booth #1044 or in the vLab
Questions?
46© Copyright 2016 EMC Corporation. All rights reserved. 46© Copyright 2016 EMC Corporation. All rights reserved.