Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB

transcript

#MDBW17

Andrew Morgan – MongoDB Product Marketing

POWERING MICROSERVICES WITH DOCKER, KUBERNETES, KAFKA, & MONGODB

andrewmorgan

#MDBW17

AGENDA

Microservice

What, Why,

Containers

Docker, Kafka

Orchestratio

Kubernetes,

Mesos, …

MongoDB

Why, How?

When to use Use cases

Who, Why?

1 2 3 4 5 6

MICROSERVICES

#MDBW17

WHY USE MICROSERVICES? (TL;DRWEBSCALE)

Speed Change Maintain Scale Empower

Build MVP quickly Rapid iterations Simple components Product Team == Component

React to market Isolated impact Team Committees

#MDBW17

Monolithic

Cross-Team

Tight Coupling

Small Change

Huge Impact

System Retests

#MDBW17

Microservices

Decoupled

Independent Dev

Isolated Impact

#MDBW17

MICROSERVICES EXAMPLE

Twitter

IngestGoogle+

Ingest

Snapcha

t Ingest

Faceboo

k Ingest

#MDBW17

Twitter

Ingest

Snapcha

t Ingest

Faceboo

k Ingest

#MDBW17

Twitter

Ingest

Snapcha

t Ingest

Faceboo

k Ingest

#MDBW17

Twitter

Ingest

Snapcha

t Ingest

Faceboo

k Ingest

Whatsap

p Ingest

#MDBW17

Twitter

Ingest

Snapcha

t Ingest

Faceboo

k Ingest

Whatsap

p Ingest

Snapcha

t IngestSnapcha

t Ingest

#MDBW17

DEVELOPMENT TEAMS

CONTAINERS

#MDBW17

CONTAINERS – POWERING MICROSERVICES

Real world shipping containers

• Road, rail & sea

• Contents untouched

• Ubiquitous & standardized

• Simple

• Contents protected

• Constraints

#MDBW17

CONTAINERS – POWERING MICROSERVICES

Software containers

• 1 image -> Many containers‒ Laptop, DC, cloud

‒ Dev, QA, production, support

• Simple, efficient

• Isolation

• Constraints

#MDBW17

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Host Operating System

Hypervisor

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Container ContainerContainer

Bare Metal

Docker Engine

Libraries

Service ServiceService

#MDBW17

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Hypervisor

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Bare Metal

Docker Engine

Libraries

#MDBW17

VMS VS. CONTAINERS

VM VMVM

Bare Metal

Hypervisor

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Guest OS

Libraries

Service

Bare Metal

Docker Engine

Libraries

#MDBW17

DOCKER

• Simple to use

• 100K+ images on Docker Hub

• Build images from images

• Platforms‒ Linux, OS X, Windows

‒ Laptop, VM, Cloud,…

‒ Cloud services

#MDBW17

RUN MONGODB

docker run -d mongo

#MDBW17

TITLE ONLY

#MDBW17

TITLE ONLY

#MDBW17

MICROSERVICE ARCHITECTURES BUILT ON CONTAINERS

Many small, focused containers ->

sophisticated services

• Well defined APIs

• Independent languages &

libraries

• Modular: easy maintenance +

• Fault tolerant

• Scalable

#MDBW17

CONNECTING THE MICROSERVICES –KAFKA

Producer

123...

Topic A

Consumer

#MDBW17

Producer

123...

Topic A

Consumer

Producer Consumer

#MDBW17

Producer

123...

Partition 0

Topic A

Consumer

Producer Consumer

123...

Partition 1

#MDBW17

Producer

LEADER

Topic A / Partition 0

Broker 1

FOLLOWER

Broker 2

LEADER

#MDBW17

Producer

123...

Partition 0

123...

Partition 1

123...

Partition N

Topic A

Topic B

123...

Partition 0

New Old

Consumer

ORCHESTRATION

#MDBW17

ORCHESTRATION

Automated deployment, connecting, and maintenance of multiple containers

• Provision hosts

• Containers‒ Instantiate

‒ Reschedule

‒ Link

‒ Scale Out/In

• Expose services

#MDBW17

KUBERNETES

Created by Google, feature-rich and widely adopted

• Deployment and ‘replication’

• On-line scale out/in

• Rolling upgrades

• High Availability

• Persistence

• Ports

• Load balancing

• Google Compute Engine

#MDBW17

APACHE MESOS

10,000s of physical servers; used by

Twitter, Airbnb & Apple

• Code (“frameworks) vs.

declarative

• Less feature rich than Kubernetes

• Kubernetes as a Mesos

framework

• Foundation for distributed systems‒ Apache Aurora, Chronos, Marathon

#MDBW17

CHOOSING AN ORCHESTRATION FRAMEWORK

• What you have:‒ Skills?

‒ DevOps frameworks?

‒ Number of hosts?

‒ Bare metal, VMs, or cloud?

• Lifecycle

• Features‒ Automated High Availability?

‒ Grouping and load balancing?

‒ As a service?

MONGODB

#MDBW17

WHY MONGODB IS A GOOD FIT FOR MICROSERVICES

Monitoring

Automation

Flexible Data

Redundancy Scalability Simplicity

#MDBW17

Monitoring

Automation

Flexible Data

#MDBW17

Monitoring

Automation

Flexible Data

#MDBW17

Monitoring

Automation

Flexible Data

#MDBW17

Monitoring

Automation

Flexible Data

#MDBW17

Monitoring

Automation

Flexible Data

#MDBW17

ORCHESTRATING MONGODB USING KUBERNETES

Distributed, stateful application

• Persistent volumes

• External IP addresses for internal

• Init MongoDB replica set

• Monitor

• Backup

Volumename:

mongo-persistent-storage1

pdName: mongodb-disk1

mongodb

-disk1

Containername: mongo-node1

image: mongo

command: mongod –replSet my_replica_set

containerPort: 27017

volumeMounts:

name: mongo-persistent-storage1

mountPath: /data/db

DockerHubRegistry

Podlabels: [name: mongo-node1; instance: rod]

ReplicationControllername: mongo-rc1

labels: [name: mongo-rc]

replicas: 1

selector: [name: mongo-node1]

Service:LoadBalancername: mongo-svc-a

labels: [name: mongo-svc-a]

ports: [port: 27017, targetPort: 27017]]

selector: [name: mongo-node, instance: rod]ExternalIP

Address

104.1.1.1

104.1.1.1:27017

ReplicationControlername: mongo-rc-europe

labels: [name: mongo-europe]

replicas: 1

selector: [name: mongo-node]

Volumename:

mongo-persistent-storage

pdName: mongodb-disk-europe

mongodb-

disk-europe

Containername: mongo-node

image: mongo

volumeMounts:

name: mongo-persistent-storage

mountPath: /data/db

Podlabels: [name: mongo-node]

Service:LoadBalancername: mongo-svc-europe

labels: [name: mongo-svc-europe]

ReplicationControlername: mongo-rc-asia

labels: [name: mongo-asia]

replicas: 1

Volumename:

pdName: mongodb-disk-asia

mongodb-

disk-asia

image: mongo

volumeMounts:

mountPath: /data/db

Service:LoadBalancername: mongo-svc-asia

labels: [name: mongo-svc-asia]

ReplicationControlername: mongo-rc-us

labels: [name: mongo-us]

replicas: 1

Volumename:

pdName: mongodb-disk-us

mongodb-

disk-us

image: mongo

volumeMounts:

mountPath: /data/db

Service:LoadBalancername: mongo-svc-us

labels: [name: mongo-svc-us]

#MDBW17

STATEFULSETS

Beta in Kubernetes 1.5/6

• Stable, predictable, unique network identifiers.‒ IP addresses may change

• Stable, persistent storage

• Ordered, graceful deployment and scaling (0 N-1)

• Ordered, graceful deletion and termination (N-1 0)

WHEN TO USE

MICROSERVICES

#MDBW17

WHEN TO USE MICROSERVICES

USE CASES

#MDBW17

MONGODB & MICROSERVICES IN THE WILD

#MDBW17

REFERENCES

• Enabling Microservices – Containers & Orchestration Explained

https://www.mongodb.com/collateral/microservices-containers-and-orchestration-explained

• Microservices: The Evolution of Building Modern Applications

https://www.mongodb.com/collateral/microservices-the-evolution-of-building-modern-

applications

• Data Streaming with Apache Kafka & MongoDB

https://www.mongodb.com/collateral/data-streaming-with-apache-kafka-and-mongodb

#MDBW17

RUNNING MONGODB AS A MICROSERVICE

Conway’s Law1967

Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.”

#MDBW17

ALTERNATE MICROSERVICES EXAMPLE

• Much larger Microservices1. User Account

2. Product Catalog

3. Inventory

4. Orders

• Best practice is each

microservice has its own

database

#MDBW17

SECURITY

Opportunities

• Isolation

• Constrain resources

• Narrow roles

• Keep images/containers current

• root: so restrict access

Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB

Technology