PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...

How people build software

!

"

Automating Schema Changes using

Percona Live Open Source Database Conference 2017-04-27 | 12:50 PM - 1:40 PM | Ballroom D

How people build software!

!

2

Tom Krouper

• @CaptainEyesight • ! @tomkrouper

• Database Infrastructure Engineer


MySQL at GitHub• GitHub stores repositories in git, and uses MySQL

as the backend database for all related metadata: • Repository metadata, users, issues, pull

requests, comments etc. • Website/API/Auth/more all use MySQL.

• We run a few (growing number of) clusters, totaling

around 100 MySQL servers. • The setup isn’t very large but very busy. • Our MySQL service must be highly available.

3

!


!

4

GitHub Online Schema

Transmogrifier


gh-ost --host=replica.with.rbr.com --database="my_schema" --table="my_table" --alter="engine=innodb" --max-load=Threads_running=25 --critical-load=Threads_running=1000 --critical-load-interval-millis=3000 --throttle-http="http://freno/${cluster}"

—throttle-control-replicas=“replica1,replica2" --switch-to-rbr --exact-rowcount --concurrent-rowcount --panic-flag-file=/tmp/ghost.panic.flag --postpone-cut-over-flag-file=/tmp/ghost.postpone.flag --hooks-path=/path/to/hooks/ --hooks-hint=“@$(whoami)" ... [--execute]

!gh-ost

5

!"!

"!

"

https://github.com/github/gh-ost

https://github.com/github/gh-ost


Migrationswithout the locking ALTER

6

!


!Rails

7

$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true

class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end

def down remove_column :example_table, :some_id end end

gh-ost ...

—table="example_table" —alter=“ADD COLUMN some_id int(11) DEFAULT NULL” ...


!

8

Hubot Chatops automation


!Chatops commands

9

.migration

.mysql

.qmtd

.queue for github

.deploy

.nagios

.xtrabackup

.mysqlproxy

.truck me

.where is


Putting it all together

automate all the things… well, at least some of the things

10

!

credit: https://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html

https://hyperboleandahalf.blogspot.com/2010/06/this-is-why-ill-never-be-adult.html


!Schema Migrations

11

$


!Schema Migrations

12

$$ cat db/migrate/20170427125019_add_some_id_to_example_table.rb # frozen_string_literal: true

class AddSomeIdToExampleTable < GitHub::Migration def up add_column :example_table, :some_id, :integer end

def down remove_column :example_table, :some_id end end


%!Schema Migration: Automation

13

Usage: .migration-queue <command> [<args>...] Workflow chatops for database migrations

show [needs_review|reviewed|scheduled] - Show all migrations in queue. add <pr-number> - Add a migration to the queue. schedule <pr-number> - Schedule a migration to be run. completed <pr-number> <migration-version> - Mark a migration as completed.

.migration-queue


!Schema Migrations

14

$


!Schema Migrations

15

$


%!.migration queue show

16


%!.migration queue add

17


%!.migration queue schedule

18


%!Schema Migrations

19

•CREATE

•DROP


%!Schema Migrations

20

•CREATE

•DROP


!Schema Migrations

21

%•CREATE

• Scheduling migrations outputs the CREATE TABLE statement

•DROP • Outputs chatops command to “RENAME TABLE“


%!Schema Migrations

22

•ALTER


%!Schema Migrations

23

•ALTER • Runs a script that calls gh-ost


!gh-migrate-ghost

24

gh-ost \ --conf=/etc/mysql/gh-ost.cnf \ --host=$ghost_replica \ --database="$database_name" --table="$table_name" --alter="$ddl" \ --max-load=Threads_running=25 --critical-load=Threads_running=1000 \ --critical-load-interval-millis=3000 --chunk-size=$chunk_size \ --throttle-http="http://${freno}:8111/check/gh-ost/mysql/${cluster}" \ --max-lag-millis=500 --heartbeat-interval-millis=100 \ --switch-to-rbr --allow-master-master --cut-over=default \ --timestamp-old-table \ --exact-rowcount --concurrent-rowcount --default-retries=120 \ --panic-flag-file=/tmp/ghost.panic.flag \ --postpone-cut-over-flag-file=/etc/github/ghost.postpone.flag \ --serve-socket-file="$socket_file" \ --hooks-path=${hooks_directory} --hooks-hint="${hooks_hint_user}" \ --verbose \ $execute $extra_args


%!Running Migration Automation

25

Manage running gh-ost migrations

Usage: .migration sup show brief status of running migrations .migration status show full status of running migrations .migration local show status of running migrations on localhost (useful from SSH logins)

.migration nice-ratio <ratio> set new nice-ratio for active migration .migration max-lag-millis <maxlag> set new max-lag milliseconds for active migration .migration max-load <status=val> set new status check for gh-ost to pause on, e.g., threads_running=30 .migration critical-load <status=val> set new status check for gh-ost to stop on, e.g., threads_running=1000 .migration throttle-control-replicas <hosts> set the replicas using commma delimited list of hosts. .migration throttle-http <url> change freno URL for running migrations. Empty string to disable throttling via freno.

.migration throttle|pause|suspend force throttling of active migrations .migration no-throttle|continue|resume terminate forced throttling (other throttling reasons may still apply)

.migration unpostpone|cut-over <table_name> cease to actively postpone; proceed to cut-over and completion .migration panic kill the running migration (requires magic_word)

.migration



26

.migration sup



27

.migration status



28

.migration nice-ratio <ratio>

.migration max-lag-millis <maxlag>

.migration max-load <status=val>

.migration critical-load <status=val>

.migration throttle-control-replicas <hosts>

.migration throttle-http <url>



29

.migration throttle

.migration resume

.migration cut-over <table_name>

.migration panic



30

.migration cut-over <table_name>


%!Finishing the Migration

31

.migration-queue completed <pr>


&!Merging the PR

32

•qmtd <url for pr> •… wait •update branch •wait for ci tests •merge (deploy)


$ … &!Next Pull Request

33


#!Automation Issues

34

•Migration Scheduler •Automatic gh-ost runs •Automatic pull request merge


$!Lessons Learned

35

•Smaller changes •Ask for help •Dig in as deep as you can first •You can’t automate everything


!Opportunity for improvements

36

•Everything is iterative •More automation ideas

• PR approval could include `.migration-queue add`. • merge `.migration-queue add` & `schedule` • rails level changes to avoid merge conflicts

How people build software

!

"

@CaptainEyesight [email protected]

Questions / Thanks

We’re Hiring:

Platform Data Engineer

https://bit.ly/platform-data

mailto:[email protected]

https://bit.ly/platform-data

Date post:	22-May-2020
Category:	Documents
Upload:	others
View:	5 times
Download:	0 times

PL2017 Automating Schema Changes - Percona · How people build so!ware MySQL at GitHub • GitHub...

Documents