Post on 08-Aug-2015
transcript
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CoreOS automated MySQL Cluster Failover using Galera Cluster
Hewlett-Packard (Enterprise) Advanced Technology Group – CloudYazz Atlas
Patrick Galbraith Spring 2015
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
About the speaker ● Yazz Atlas● HP Advanced Technology Group● HP Cloud Services PaaS/DBaaS/
LBaaS Environment ● Has worked at Windermere Real
Estate, Keas, Slashdot, Cobalt Group, University of Iowa, Antioch College
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
About the speaker ● Patrick Galbraith ● HP Advanced Technology Group● Has worked at Blue Gecko,
MySQL AB, Classmates, Slashdot, Cobalt Group, US Navy, K-mart
● MySQL projects: memcached UDFs, DBD::mysql, federated storage engine
● Family● Outdoors
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.4
Purpose of this talk – why are you here?
Basic Pre-req• Overview of the Demo Setup
Docker• Containers vs. Virtulization
• Simple Docker usage
• Clustered Docker
CoreOS• Container optimized, stripped down Linux
distribution
• Overview of core components of CoreOS – fleet, etcd, systemd
Galera• Synchronous replication – excellent solution to
clustering MySQL
• HA Proxy for failover
Discussion• What is next
• What about Kubernetes
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
Laptop Host- VirtualBox- OpenStack- Vmware- Bare metal- et al.
Master-01
Basic Pre-reqs Setup
CoreOS + etcd + confdSome of the times you will need• CoreOS
• Go
– https://golang.org/doc/install
• Etcdctl– go get github.com/coreos/etcd/etcdctl
• Fleetctl– go get github.com/coreos/fleet/fleetctl
Core-01
Core-02
Core-03
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.6
What are containers?
• Operating-system-level virtualization• Encapsulated, hermetically sealed applications
• Relatively isolated• Small footprint
• Fast to launch!
• Portable. And did I mention, portable?!• Use of host OS and Kernel
• Execution consists of time to startup application in question• LXC, Docker, Solaris Zones, BSD Jails, Parallels Virtuozzo, OpenVZ, …
Photo credit: expo_2020 - http://goo.gl/OeSQhi License CC v2.0
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Containers vs. VMs
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
DockerWhat is Docker?
• Set of tools for managing containers
• Command line tool that doubles as a daemon
• Uses Linux Kernel Features:
– Kernel namespaces – the core ingredient to containers working: PID, IPC ,uts (what will be seen by a group of processes), mount, network and user
– Cgroups (control groups) -- limit, account and isolate resource usage (CPU, memory, disk I/O, etc.) of process groups
• Originally used lxc, now defaults to Libcontainer but meant for any containerization mechanism
• Much more light weight than VMs
• Encapsulated application containers in a relatively isolated but lightweight operating environment
• Written in Go
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Clustered Docker• CoreOS (https://coreos.com/)
• Kubernetes (http://kubernetes.io)
• Mesos + Marathon (http://mesos.apache.org/) Apache project, Zookeeper, etc
• Project Atomic (http://www.projectatomic.io/) RH/Fedora/Centos designed for running Docker
• Docker Openstack (https://wiki.openstack.org/wiki/Docker) Hypervisor Driver for Openstack Compute
• Swarm/Compose/Machine
• RancherOS (http://rancher.com/rancher-os) Minimalist Linux, Docker daemon runs as PID 1 first process the kernel starts known as “System Docker,”
• Flocker (https://clusterhq.com)
• Spotify Helios (https://github.com/spotify/helios) Zookeeper
• Flynn (https://flynn.io/)
• Deis (http://deis.io)
• Maestro (https://github.com/toscanini/maestro)
• Shipyard (http://shipyard-project.com)
• … others to come!
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
What is CoreOS?
CoreOS
• Minimalist Linux
• Optimized for containers• Easy to run containers
• Service discovery, container management,
• Docker -- Container runtime and management, though Rocket long-term• Etcd – distributed global key value store for config data on each node
• Fleet – Rudimentary Scheduler interacts with systemd and etcd• Systemd – Newer Linuxi – system and service manager for Linux
• Flannel – Networking across nodes
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.11
Connecting it all together
CoreOS• Etcd
• Fleetd
• Flanneld
Photo credit: Stéfan – http://goo.gl/ye8Icd License CC v2.0
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Galera secret sauce• Galera (Percona XtraDB Cluster) container
• FLEETCTL_ENDPOINT set to ETCD host:port
• etcd included
• Entrypoint script
• Generates SQL file to create SST (single state transfer) user and clustercheck user
• Builds cluster.cnf
• Wsrep_node_address set to its own IP
• Wsrep_cluster_address set using entries from etcd
• Server-id random number
• clustercheck on each galera node container (port 8000) to determine health and which nodes available using clustercheck user to connect
• Ports 3306, 4444, 4567, 4568, 8000 exposed provides access to Galera cluster node containers for external and HAProxy container
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Galera secret sauce – haproxy sprinkling• HAProxy container
• FLEETCTL_ENDPOINT set to ETCD host:port
• Confd included
• Confd builds haproxy.cfg from template using etcd entries for each Galera node
• Confd loop entered to start container
• HAProxy uses clustercheck on each node (port 8000) to determine health and which nodes available
• Port 33006 exposed provides access to Galera cluster node containers
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Laptop
Virtualbox
Master-01
Core-01
Core-02
Core-03
etcd
flanneld
Galera-1
Galera-2
Galera-3
haproxy
Sidekick-1
Sidekick-2
Sidekick-2Flannel network
Gal
era
repl
icat
ion
HAP
roxy
etcd
com
mun
icat
ion
Connecting it all together.
$> vagrant up
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Galera on CoreOS
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Disaster Strikes!
Don’t put all your containers on one ship (CoreOS host).
• HA Proxy is the middleman connecting to your Galera Cluster
– Minimal impact to connecting agents
• Adding a new CoreOS host and redeploy the Galera container
Photo credit Paul Townsend – http://goo.gl/ImL8hX License CC v2.0
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.17
Demo time…
DEMO• Vagrant - 1.7.2
• Virtualbox - 4.3.26r98988
– coreos-alpha (virtualbox, 660.0.0)
• Github Repos
– https://github.com/EntropyWorks/coreos-vagrant
– https://github.com/EntropyWorks/fleet-units-galera-cluster
– https://github.com/EntropyWorks/docker_haproxy
– https://github.com/CaptTofu/percona_xtradb_cluster_docker (coreos branch)
• Asciinema recordings
– https://asciinema.org/a/19559
© Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank You
Hewlett-Packard, Advanced Technology Group
Yazz Atlas yazz.atlas@hp.com
Patrick Galbaith patg@hp.com