Downtime is not an option - day 2 operations - Jörg Schad

1

Downtime is not an option How Fast Data and Microservices change the datacenter

@joerg_schad @dcos

© 2017 Mesosphere, Inc. All Rights Reserved. 2

Jörg SchadDistributed Systems Engineer

@joerg_schad


In the beginning there was a big

Monolith


© 2017 Mesosphere, Inc. All Rights Reserved.

Hardware

Operating System

Application

5

COMPUTERS


noun | ˈmīkrō/ /ˈsərvəs/ :

an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.*

Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable.

*From whatis.com

OVERVIEW


Operating System

Operating System

Operating System

ServiceApp ServiceServiceAppApp

7

MICROSERVICES- Polyglot- Single Responsibility- Smaller Teams- Utilization- Machine

types/groups- Dependency hell

Machine

Infrastructure

Machine Machine

ServiceService ServiceServiceServiceService

© Gerard Julien/AFP

Run everything in containers!



OS

9

CONTAINERS- Rapid deployment- Dependency

vendoring- Container image

repositories- Spreadsheet

scheduling

OS OS

Machine

Infrastructure

Machine Machine

Container Runtime Container Runtime Container Runtime



CONTAINERSCHEDULING

RESOURCE MANAGEMENT

SERVICE MANAGEMENT

- Load Balancing- Readiness Checking

CONTAINER ORCHESTRATION


CONTAINERSCHEDULING

- Placement- Replication/Scaling- Resurrection- Rescheduling- Rolling Deployment- Upgrades- Downgrades- Collocation

RESOURCE MANAGEMENT

- Memory- CPU- GPU- Volumes- Ports- IPs- Images/Artifacts

SERVICE MANAGEMENT

- Labels- Groups/Namespaces- Dependencies- Load Balancing- Readiness Checking

CONTAINER ORCHESTRATION


Orc

hest

ratio

n

12

Machine Infrastructure

Web Apps & Services

Scheduling

Resource Management

Container Runtime

Machine & OS

Service Management

CONTAINERORCHESTRATION

Machine & OS Machine & OS

Container Runtime Container Runtime


MapReduce is crunching Data

Meanwhile...


But then business demanded

FAST DATAWe need to turn faster!


Fast Data

Batch Event ProcessingMicro-Batch

Days Hours Minutes Seconds Microseconds

Solves problems using predictive and prescriptive analyticsReports what has happened using descriptive analytics

Predictive User InterfaceReal-time Pricing and Routing Real-time AdvertisingBilling, Chargeback Product recommendations


The SMACK Stack

EVENTSUbiquitous data streams from connected devices

INGEST

Apache Kafka

STORE

Apache Spark

ANALYZE

Apache Cassandra

ACT

Akka

Ingest millions of events per second

Distributed & highly scalable database

Real-time and batch process data

Visualize data and build data driven applications

Mesos/ DC/OS

Sensors

Devices

Clients


Datacenter


NAIVE APPROACH

Typical Datacentersiloed, over-provisioned servers,

low utilization

Industry Average12-15% utilization

mySQL

microservice

Cassandra

Spark/Hadoop

Kafka



MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS

Typical Datacentersiloed, over-provisioned servers,

low utilization

Mesos/ DC/OSautomated schedulers, workload multiplexing onto the

same machines

mySQL

microservice

Cassandra

Spark/Hadoop

Kafka


• A top-level Apache project• A cluster resource

negotiator• Scalable to 10,000s of

nodes• Fault-tolerant, battle-tested• An SDK for distributed apps• Native Docker support

21

Apache Mesos


Datacenter Operating System (DC/OS)

Distributed Systems Kernel (Mesos)

DC/OS ENABLES MODERN DISTRIBUTED APPS

Big Data + Analytics EnginesMicroservices (in containers)

Streaming

Batch

Machine Learning

Analytics

Functions & Logic

Search

Time Series

SQL / NoSQL

Databases

Modern App Components

Any Infrastructure (Physical, Virtual, Cloud)23

24

THEBASICS

DC/OS is … ● 100% open source (ASL2.0)

+ A big, diverse community● An umbrella for ~30 OSS projects

+ Roadmap and designs+ Docs and tutorials

● Not limited in any way● Familiar, with more features

+ Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages, ...


Container Options

Enhancements to the Mesos Containerizer to allow support launching specific container formats (Docker, AppC, OCI (future), etc)

● Reduces need to maintain and update multiple containerizers

● Support multiple container formats with a single containerizer

Image provisioner component added to the Mesos containerizer - responsible for pulling, caching, and preparing container root filesystems

Launcher Isolators

Universal containerizer

Provisioner

Process management

Container lifecycle hook

Container image support


DEMO


GEO-ENABLED IoT


DATA FLOW


Keep it running!


Monitoring- Collecting metrics- Routing events- Downstream processing

- Alerting- Dashboards- Storage (long-term retention)

Logging- Scopes- Local vs. Central- Security considerations

DAY 2 OPERATIONS


Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package Management- Networking Policies- Auditing- Backups & Disaster Recovery

Troubleshooting- Debugging

- Services- System- Access?

- Tracing- Chaos Engineering

DAY 2 OPERATIONS


Troubleshooting● Services: typically specific to service, use logging (for

example, dcos task log) and dcos node ssh for

per-node investigations

● dcos task exec

○ Permissions?

● System:

○ Simple diagnostics via dcos node diagnostics

○ Comprehensive dump via clump

https://dcos.io/docs/1.8/overview/telemetry/#diagnostics

https://github.com/mhausenblas/clump/tree/master/recipes/dcos


THANK YOU!

ANY QUESTIONS?@dcos

[email protected]

/groups/8295652

/dcos/dcos/examples/dcos/demos

chat.dcos.io


FailuresFramework

Scheduler

Executor

Task

Agent

LEADER STANDBY STANDBY

ZK

ZK

ZK

Executor

Task

Agent


Distributed Systems could be so easy...

35

1. The network is reliable.

2. Latency is zero.

3. Bandwidth is infinite.

4. The network is secure.

5. Topology doesn't change.

6. There is one administrator.

7. Transport cost is zero.

8. The network is homogeneous.

*) https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing

https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing




Questions?

Code: https://git.io/vXUoy

http://grnh.se/ie76ru

https://git.io/vXUoy








Monitoring


METRICS

Measurements captured to determine health and performance of cluster

- How utilized is the cluster?- Are resources being optimally used?- Is the system performing better or worse over time?- Are there bottlenecks in the system?- What is the response time of applications?


DC/OS METRIC SOURCES

● Mesos metrics ○ Resource, frameworks, masters, agents,

tasks, system, events ● Container Metrics

○ CPU, mem, disk, network● Application Metrics

○ QPS, latency, response time, hits, active users, errors

OS

Mesos

Container ContainerContainer

App App App


Before upgrading1. Make sure cluster is healthy!2. Perform backup

a. ZKb. Replicated logsc. other state

3. Review release notes4. Generate install bundle

a. Validate versions

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent

Executor

Task

Agent


ZK

ZK

ZK


MESOS MASTER METRICS

● Metrics for the master node are available at the following URL:○ http://<mesos-master-ip>/mesos/master/metrics/snapshot ○ The response is a JSON object that contains metrics names and values as

key-value pairs.● Metric Groups:

○ Resources○ Master○ System○ Slaves○ Frameworks○ Tasks○ Messages○ Event Queue○ Registrar


MESOS MASTER BASIC ALERTS

Metric Value Inference

master/uptime_secs is low The master has restarted

master/uptime_secs < 60 for sustained periods of time The cluster has a flapping master node

master/tasks_lost is increasing rapidly Tasks in the cluster are disappearing. Possible causes include hardware failures, bugs in one of the frameworks or bugs in Mesos

master/slaves_active is low Slaves are having trouble connecting to the master

master/cpus_percent > 0.9 for sustained periods of time DCOS Cluster CPU utilization is close to capacity

master/mem_percent > 0.9 for sustained periods of time DCOS Cluster Memory utilization is close to capacity

master/disk_used & master/disk_percent DCOS Disk space consumed by Reservations

master/elected is 0 for sustained periods of time No Master is currently elected


Operations

UPGRADES


1. Master rolling upgradea. Start with standbyb. Install new DC/OS

2. Agent rolling upgrade3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


1. Master rolling upgrade2. Agent rolling upgrade

a. Uninstall DC/OSb. Install new DC/OS

3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


1. Master rolling upgrade2. Agent rolling upgrade3. Framework upgrades

a. Orthogonal to DC/OSb. Ensure changes don’t

affect existing apps

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


Failure Handling


Failure Handling

MESOS TASK FAILURE


MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT AGENT

EXECUTOR

TASK


Status UpdateStatus Update

EXECUTOR

MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

TASK

AGENT

SEGFAULT :(


MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

TASK

Launch TaskLaunch Task

AGENT

Launch Task


EXECUTOR

Status UpdateStatus Update

MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

TASK

AGENT

Status Update


Failure Handling

MESOS AGENT FAILURE


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Re-register


Failure Handling

MESOS HOST FAILURE


LOCAL AGENT FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


LOCAL AGENT FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT


LOCAL AGENT FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

Status Update


MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

TASK

Launch TaskLaunch Task

Launch Task

Resource Offer


MESOS TASK FAILURE

ZK

MASTERMARATHON

CLIENT AGENT AGENT

EXECUTOR

TASK

Status Update

Status Update

Status Update


Failure Handling

MESOS MASTER FAILURE


MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Leading Master Leading Master

Leading Master


MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Reregister

Reregister

ReregisterReregister


Reregistered

MASTER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Reregistered

ReregisteredReregistered


Failure Handling

SCHEDULER FAILURE


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Framework IDLeading Master


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Reregister


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Reregistered


SCHEDULER FAILURE

ZK

MASTERMARATHON


EXECUTOR

TASK

Status Update

Reconcile Tasks


1. Master rolling upgradea. Start with standbyb. Uninstall DC/OSc. Install new DC/OS

2. Agent rolling upgrade3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


1. Master rolling upgrade2. Agent rolling upgrade

a. Uninstall DC/OSb. Install new DC/OS

3. Framework upgrades

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


1. Master rolling upgrade2. Agent rolling upgrade3. Framework upgrades

a. Orthogonal to DC/OSb. Ensure changes don’t

affect existing apps

UPGRADE PROCEDURE

Framework

Scheduler

Executor

Task

Agent


ZK

ZK

ZK

Executor

Task

Agent


FUTURES (TBD)

Leverage maintenance primitives in Mesos to drain host

Upgrade management through DC/OS to perform rolling upgrades


Monitoring- Collecting metrics- Routing events- Downstream processing

- Alerting- Dashboards- Storage (long-term retention)


DAY 2 OPERATIONS


Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package Management- Networking Policies- Auditing- Backups & Disaster Recovery


- Services- System


DAY 2 OPERATIONS


MONITORING


MONITORINGCONCEPT


MONITORINGTOOLINGEXAMPLES

● local scraping:

a. collectd

b. cAdvisor*

● event router:

a. fluentd

b. Flume

c. Kafka*

d. logstash*

e. Riemann

*) available via Mesosphere Universe

https://collectd.org/

https://collectd.org/

https://github.com/google/cadvisor

https://github.com/google/cadvisor

http://www.fluentd.org

http://www.fluentd.org

https://flume.apache.org

https://flume.apache.org

https://kafka.apache.org

https://kafka.apache.org

https://www.elastic.co/products/logstash

https://www.elastic.co/products/logstash

http://riemann.io

http://riemann.io



● storage:

a. Elasticsearch*

b. Graphite

c. InfluxDB*

d. KairosDB/Cassandra*

e. OpenTSDB/HBase

f. others such a local filesystem, Ceph FS*,

HDFS*, etc.


https://www.elastic.co/products/elasticsearch

https://www.elastic.co/products/elasticsearch

https://graphiteapp.org

https://graphiteapp.org

https://influxdata.com/time-series-platform/influxdb

https://influxdata.com/time-series-platform/influxdb

https://kairosdb.github.io

https://kairosdb.github.io

http://opentsdb.net

http://opentsdb.net



● dashboard:

a. D3

b. Grafana*

c. signal fx

● alerting:

a. BigPanda

b. PagerDuty

c. signal fx

d. VictorOps


https://d3js.org

https://d3js.org

https://grafana.net

https://grafana.net

https://signalfx.com


https://bigpanda.io

https://bigpanda.io

https://www.pagerduty.com

https://www.pagerduty.com



https://victorops.com

https://victorops.com


MONITORINGTOOLINGEXAMPLES(INTEGRATED)

● Amazon CloudWatch ● AppDynamics ● Azure Monitor ● Circonus ● DataDog* ● dcos/metrics● Ganglia ● Google Stackdriver ● Hawkular ● Icinga ● Librato ● Nagios ● New Relic ● OpsGenie ● Pingdom ● Prometheus ● Ruxit Dynatrace* ● Sensu ● Sysdig* ● Zabbix


https://aws.amazon.com/cloudwatch



https://www.appdynamics.com



https://azure.microsoft.com/services/application-insights



https://www.circonus.com



https://www.datadoghq.com

https://www.datadoghq.com/

https://www.datadoghq.com

https://github.com/dcos/dcos-metrics/

https://github.com/dcos/dcos-metrics/

http://ganglia.info

http://ganglia.info

http://ganglia.info

https://cloud.google.com/monitoring



http://www.hawkular.org/



https://www.icinga.com



https://www.librato.com



https://www.nagios.org



https://newrelic.com



https://www.opsgenie.com



https://www.pingdom.com



https://prometheus.io



http://www.dynatrace.com/en/ruxit



https://sensuapp.org



https://sysdig.com

https://sysdig.com

https://sysdig.com

http://www.zabbix.com

http://www.zabbix.com/

http://www.zabbix.com


LOGGING


LOGGINGSCOPES


LOGGINGTOOLINGEXAMPLES(PRIMITIVES) ● DC/OS logging overview

● Docker logging drivers

● systemd's journalctl

https://dcos.io/docs/latest/administration/logging/

https://dcos.io/docs/latest/administration/logging/

https://docs.docker.com/engine/admin/logging/overview/

https://coreos.com/os/docs/latest/reading-the-system-log.html


LOGGINGTOOLINGEXAMPLES(INTEGRATED)

● Centralized app logging with fluentd

● DC/OS

a. ELK stack log shipping

b. Splunk

● Graylog

● Loggly

● Papertrail

● Sumo Logic

http://www.fluentd.org/centralized_application_logging

http://www.fluentd.org/centralized_application_logging

https://dcos.io/docs/1.8/administration/logging/elk/

https://dcos.io/docs/1.8/administration/logging/elk/

https://dcos.io/docs/1.8/administration/logging/splunk/

https://dcos.io/docs/1.8/administration/logging/splunk/

https://www.graylog.org/

https://www.graylog.org/

https://www.loggly.com/

https://www.loggly.com/

https://papertrailapp.com/

https://papertrailapp.com/

https://www.sumologic.com/

https://www.sumologic.com/


MAINTENANCE


Overview

● How to install a new version of X?● When to scale what (service-level vs. nodes)● Who gets to access/install which services in what way?

Upgrades

Sizing

User and package management

● What services can talk to each other and in which way?● Who accessed what, when and how?● How is the continuous operation of the cluster and the services accomplished?

What happens when cluster (or critical infra components like ZK) go down?

Networking

Auditing

Disaster Recovery


OTHER TROUBLESHOOTING TECHNIQUES

● Tracing

○ Idea: identify latency issues and perform

root-cause analysis in a distributed setup

○ OpenTracing

● Chaos Engineering

○ Idea: proactively break (parts of) the system to

understand how it reacts

○ Chaos Monkey

○ DRAX

http://opentracing.io/

http://opentracing.io/

https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey

https://github.com/dcos-labs/drax

https://github.com/dcos-labs/drax



ARCHITECTUREMESOS FUNDAMENTALS

● Agents advertise resources to Master● Master offers resources to Framework● Framework rejects/uses resources● Agents report task status to Master














Questions?

Code: https://git.io/vXUoyPsssssssst …… we are hiring!


https://git.io/vXUoy








CONTAINER SCHEDULING

106


RESOURCE MANAGEMENT

107


SERVICE MANAGEMENT

108


Service Service Service

Web App Web App Web App

Hardware

Operating System

109

SERVICE-ORIENTEDARCHITECTURE

- Separation of concerns

- Optimization of bottlenecks

- Smaller teams- API Contracts- Data replication- Complicated

provisioning- Dependency

management

Operating System

Operating System

Hardware Hardware


Operating System

Operating System

Operating System


110

MICROSERVICES- Polyglot- Single Responsibility- Smaller Teams- Utilization- Machine

types/groups- Dependency hell

Machine

Infrastructure

Machine Machine



THE BIRTH OF MESOS

TWITTER TECH TALKThe grad students working on Mesos

give a tech talk at Twitter.

March 2010

APACHE INCUBATIONMesos enters the Apache Incubator.

Spring 2009

CS262BBen Hindman, Andy Konwinski and

Matei Zaharia create “Nexus” as their CS262B class project.

MESOS PUBLISHEDMesos: A Platform for Fine-Grained

Resource Sharing in the Data Center is published as a technical report.

September 2010

December 2010

DC/OS

April 2016


Monitoring- Collecting metrics

- Routing events- Downstream processing

○ Alerting○ Dashboards○ Storage (long-term retention)


Maintenance - Cluster Upgrades- Cluster Resizing- Capacity Planning- User & Package

Management- Networking Policies- Auditing- Backups & Disaster

Recovery


○ Services○ System


Date post:	21-Jan-2018
Category:	Technology
Upload:	codemotion
View:	241 times
Download:	0 times

Downtime is not an option - day 2 operations - Jörg Schad

Technology