PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...

transcript

How people build software

Automating Schema Changes using

Percona Live Open Source Database Conference 2017-04-27 | 12:50 PM - 1:40 PM | Ballroom D

How people build software!

Tom Krouper

• @CaptainEyesight • ! @tomkrouper

• Database Infrastructure Engineer

MySQL at GitHub• GitHub stores repositories in git, and uses MySQL

as the backend database for all related metadata: • Repository metadata, users, issues, pull

requests, comments etc. • Website/API/Auth/more all use MySQL.

• We run a few (growing number of) clusters, totaling

around 100 MySQL servers. • The setup isn’t very large but very busy. • Our MySQL service must be highly available.

GitHub Online Schema

Transmogrifier

gh-ost --host=replica.with.rbr.com --database="my_schema" --table="my_table" --alter="engine=innodb" --max-load=Threads_running=25 --critical-load=Threads_running=1000 --critical-load-interval-millis=3000 --throttle-http="http://freno/${cluster}"

—throttle-control-replicas=“replica1,replica2" --switch-to-rbr --exact-rowcount --concurrent-rowcount --panic-flag-file=/tmp/ghost.panic.flag --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --hooks-path=/path/to/hooks/ --hooks-hint=“@$(whoami)" ... [--execute]

!gh-ost

https://github.com/github/gh-ost

Migrationswithout the locking ALTER

!Rails

$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true

class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end

def down remove_column :example_table, :some_id end end

gh-ost ...

—table="example_table" —alter=“ADD COLUMN some_id int(11) DEFAULT NULL” ...

Hubot Chatops automation

!Chatops commands

.migration

.mysql

.queue for github

.deploy

.nagios

.xtrabackup

.mysqlproxy

.truck me

.where is

Putting it all together

automate all the things… well, at least some of the things

credit: https://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html

!Schema Migrations

$$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true

class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end

def down remove_column :example_table, :some_id end end

%!Schema Migration: Automation

Usage: .migration-queue <command> [<args>...] Workflow chatops for database migrations

show [needs_review|reviewed|scheduled] - Show all migrations in queue. add <pr-number> - Add a migration to the queue. schedule <pr-number> - Schedule a migration to be run. completed <pr-number> <migration-version> - Mark a migration as completed.

.migration-queue

!Schema Migrations

%!.migration queue show

%!.migration queue add

%!.migration queue schedule

%!Schema Migrations

•CREATE

•DROP

%!Schema Migrations

•CREATE

•DROP

!Schema Migrations

%•CREATE

• Scheduling migrations outputs the CREATE TABLE statement

•DROP • Outputs chatops command to “RENAME TABLE“

%!Schema Migrations

•ALTER

%!Schema Migrations

•ALTER • Runs a script that calls gh-ost

!gh-migrate-ghost

gh-ost \ --conf=/etc/mysql/gh-ost.cnf \ --host=$ghost_replica \ --database="$database_name" --table="$table_name" --alter="$ddl" \ --max-load=Threads_running=25 --critical-load=Threads_running=1000 \ --critical-load-interval-millis=3000 --chunk-size=$chunk_size \ --throttle-http="http://${freno}:8111/check/gh-ost/mysql/${cluster}" \ --max-lag-millis=500 --heartbeat-interval-millis=100 \ --switch-to-rbr --allow-master-master --cut-over=default \ --timestamp-old-table \ --exact-rowcount --concurrent-rowcount --default-retries=120 \ --panic-flag-file=/tmp/ghost.panic.flag \ --postpone-cut-over-flag-file=/etc/github/ghost.postpone.flag \ --serve-socket-file="$socket_file" \ --hooks-path=${hooks_directory} --hooks-hint="${hooks_hint_user}" \ --verbose \ $execute $extra_args

%!Running Migration Automation

Manage running gh-ost migrations

Usage: .migration sup show brief status of running migrations .migration status show full status of running migrations .migration local show status of running migrations on localhost (useful from SSH logins)

.migration nice-ratio <ratio> set new nice-ratio for active migration .migration max-lag-millis <maxlag> set new max-lag milliseconds for active migration .migration max-load <status=val> set new status check for gh-ost to pause on, e.g., threads_running=30 .migration critical-load <status=val> set new status check for gh-ost to stop on, e.g., threads_running=1000 .migration throttle-control-replicas <hosts> set the replicas using commma delimited list of hosts. .migration throttle-http <url> change freno URL for running migrations. Empty string to disable throttling via freno.

.migration throttle|pause|suspend force throttling of active migrations .migration no-throttle|continue|resume terminate forced throttling (other throttling reasons may still apply)

.migration unpostpone|cut-over <table_name> cease to actively postpone; proceed to cut-over and completion .migration panic kill the running migration (requires magic_word)

.migration

.migration sup

.migration status

.migration nice-ratio <ratio>

.migration max-lag-millis <maxlag>

.migration max-load <status=val>

.migration critical-load <status=val>

.migration throttle-control-replicas <hosts>

.migration throttle-http <url>

.migration throttle

.migration resume

.migration cut-over <table_name>

.migration panic

.migration cut-over <table_name>

%!Finishing the Migration

.migration-queue completed <pr>

&!Merging the PR

•qmtd <url for pr> •… wait •update branch •wait for ci tests •merge (deploy)

$ … &!Next Pull Request

#!Automation Issues

•Migration Scheduler •Automatic gh-ost runs •Automatic pull request merge

$!Lessons Learned

•Smaller changes •Ask for help •Dig in as deep as you can first •You can’t automate everything

!Opportunity for improvements

•Everything is iterative •More automation ideas

• PR approval could include `.migration-queue add`. • merge `.migration-queue add` & `schedule` • rails level changes to avoid merge conflicts

How people build software

@CaptainEyesight tomkrouper@github.com

Questions / Thanks

We’re Hiring:

Platform Data Engineer

https://bit.ly/platform-data

PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...

Documents