MariaDB - Percona · mysvc start --rid container#03 ... HpUx ip fs vg sync hpvm srp openha FreeBSD...

transcript

MariaDB

Gcloud - Maxscale - MariaDB NoSQL - DockerGcloud - axscale - MariaDB ColumnStore - Docker

Gcloud - Maxscale - MariaDB Replication Manager - DockerGcloud - axscale - MariaDB Cluster - Docker

Agent Stack••••••

Collector Stack••

••

Step 1 : User account bootstrap•

Step 2 : Agent bootstrap•

wget -O/tmp/opensvc.deb http://repo.opensvc.com/deb/currentsudo dpkg -i /tmp/opensvc.deb

•sudo nodemgr set --param node.repopkg --value http://repo.opensvc.com/

sudo nodemgr set --param node.repocomp --value http://repo.opensvc.com/compliance/

sudo nodemgr set --param node.dbopensvc --value https://collector.opensvc.com

Step 2 : Agent bootstrap is generating a node conf file

•[node]repopkg = http://repo.opensvc.com/repocomp = http://repo.opensvc.com/compliance/dbopensvc = https://collector.opensvc.com/feed/default/call/xmlrpcdbcompliance = https://collector.opensvc.com/init/compliance/call/xmlrpchost_mode = PRDuuid = d5bccb78-a2b2-4809-b036-ac7c7bfa7101

[compliance]auto_update = trueschedule = @1440

[stats]schedule = @60

[gcedisks]scheduler = @120

Step 2 : Agent bootstrap•

sudo nodemgr register --user my.self@corp.com

sudo nodemgr set --param node.uuid --value d5bccb78-a2b2-4809-b036-ac7c7bfa7101

sudo nodemgr pushasset sudo nodemgr pushdiskssudo nodemgr pushpkgsudo nodemgr pushpatchsudo nodemgr checkssudo nodemgr sysreport

Step 3 : Node configuration

sudo nodemgr compliance fix --attach --moduleset mariadb.node

sudo nodemgr compliance attach --moduleset mariadb.node

sudo nodemgr compliance check --moduleset mariadb.node

sudo nodemgr compliance fix --moduleset mariadb.node

Step 4 : Service deployment•

sudo svcmgr -s mysvc pull

•sudo svcmgr -s mysvc provision

Run ops : Reconfigure MariaDB•

mysvc --cluster compliance fix

•mysvc --cluster stopcontainer; mysvc --cluster startcontainer;

Trouble Shooting Services •

mysvc --cluster print status

•svcmon -cv

•mysvc --cluster docker ps -a

•mysvc docker logs mysvc.container.03

•mysvc docker exec -ti mysvc.container.03 /bin/bash

•mysvc start --rid container#03

•mysvc logs or /var/log/opensvc/<mysvc>.log*

•nodemgr collector cli -- get /services/spdsvc01/resources --filter "res_type ip%" --filter "res_desc %10.0.0.33%" --filter "res_status up" --props res_desc,res_status,nodes.nodename --meta false

GCloud how to •

• gcloud compute instances set-machine-type mariadb01 --machine-type custom-1-4096 --zone europe-west1-b

iploopfsbind mountslvm2 vgveritas vgsync

lxcopenvzxenkvmvboxovmesxdocker

openhaheartbeatdKeepalived lvm2

Snaps Dds Netapp snapmirrorSymmetrix SRDF-SHP 3Par Remote CopyDatacore replicationDrbdBtrfsZfsJfsSymmetrix cloneHP EVA snapDatacore snapIBM DS8xxx snapNEC ISM, Bull Optima snapCeph RADOS snap

Solaris

iplofszfs poolveritas vgsync

zoneldomxenvbox

openha

ipfsvgsync

hpvmsrp openha

FreeBSD

ipmdfsnullfszfs poolsync

jail UNTESTED

GCloud Docker ✔ ✔ ✔ FUTURE ✔

Amazon Docker FUTURE FUTURE FUTURE FUTURE FUTURE

Azure Docker FUTURE FUTURE FUTURE FUTURE FUTURE

Openstack Docker FUTURE FUTURE FUTURE FUTURE FUTURE

••

○○○○

•••

○○○○○

MariaDBReplication

Manager

•• •

• 18

MariaDB & Maxscale Fix size 66 dockers pack in minimum 4 VMShare Disk (POD) for a pair of 1 spider + 1 backend

•front

Spider Sharding Proxy•

•front

Shard single table in spdc schema••

+------------+---------------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+------------+---------------------+------+-----+---------+-------+| id | bigint(20) unsigned | NO | PRI | NULL | || collection | bigint(20) unsigned | NO | PRI | NULL | || content | mediumblob | YES | | NULL | || inum | bigint(20) | YES | MUL | NULL | || istr | varbinary(40) | YES | MUL | NULL | |+------------+---------------------+------+-----+---------+-------+

‘t’ linked to local tables per collection •••

| t_c00 || t_c01 || t_c02 || t_c03 |

Each collection table linked to local shard tables in a schema per VM (blackhole)

| spdc34 || spdc36 || spdc38 || spdc40 || spdc42 || spdc44 |. . . . . . . . . . . . . . . . . . . . . . | spdc90 || spdc92 || spdc94 || spdc96 |

Each collection is exposed as a viewCREATE VIEW `dim_pays` AS SELECT column_get(t_ro.content, 'A2CODE' as char) AS A2CODE, spdc.t_ro.istr AS `A3CODE`, /*PK*/ column_get(t_ro.content, 'N3CODE' as char) AS N3CODE, column_get(t_ro.content, 'LIB' as char) AS LIB, column_get(t_ro.content, 'CONTINENT' as char) AS CONTINENT FROM t_ro WHERE t_ro.collection = cast(conv(left(md5('bigdata.dim_pays'),16),16,10) as unsigned)

Multi level spider layer for read to divide number of partitions to manage •

CREATE TABLE `t_ro` ENGINE=SPIDER DEFAULT CHARSET=utf8 COMMENT='mbk "1", mkd "1", alc "1", msi "144" '

(PARTITION p00 VALUES IN (0) COMMENT = 'srv "t_r1d00c00 t_l1d00c00", table "t_c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p01 VALUES IN (1) COMMENT = 'srv "t_r1d00c01 t_l1d00c01", table "t_c01", lst "0 0" ' ENGINE = SPIDER, PARTITION p02 VALUES IN (2) COMMENT = 'srv "t_r1d00c02 t_l1d00c02", table "t_c02", lst "0 0" ' ENGINE = SPIDER, PARTITION p03 VALUES IN (3) COMMENT = 'srv "t_r1d00c03 t_l1d00c03", table "t_c03", lst "0 0" ' ENGINE = SPIDER

CREATE TABLE `t_c00` ENGINE=SPIDER DEFAULT CHARSET=utf8 COMMENT='mbk "1", mkd "1", alc "1", msi "49" '

(PARTITION p00 VALUES IN (0) COMMENT = 'srv "t_r1d00c00 t_l1d00c00", table "t_d00c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p01 VALUES IN (1) COMMENT = 'srv "t_r1d01c00 t_l1d01c00", table "t_d01c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p02 VALUES IN (2) COMMENT = 'srv "t_r1d02c00 t_l1d02c00", table "t_d02c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p03 VALUES IN (3) COMMENT = 'srv "t_r1d03c00 t_l1d03c00", table "t_d03c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p04 VALUES IN (4) COMMENT = 'srv "t_r1d04c00 t_l1d04c00", table "t_d04c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p05 VALUES IN (5) COMMENT = 'srv "t_r1d05c00 t_l1d05c00", table "t_d05c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p06 VALUES IN (6) COMMENT = 'srv "t_r1d06c00 t_l1d06c00", table "t_d06c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p07 VALUES IN (7) COMMENT = 'srv "t_r1d07c00 t_l1d07c00", table "t_d07c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p08 VALUES IN (8) COMMENT = 'srv "t_r1d08c00 t_l1d08c00", table "t_d08c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p09 VALUES IN (9) COMMENT = 'srv "t_r1d09c00 t_l1d09c00", table "t_d09c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p10 VALUES IN (10) COMMENT = 'srv "t_r1d10c00 t_l1d10c00", table "t_d10c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p11 VALUES IN (11) COMMENT = 'srv "t_r1d11c00 t_l1d11c00", table "t_d11c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p12 VALUES IN (12) COMMENT = 'srv "t_r1d12c00 t_l1d12c00", table "t_d12c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p13 VALUES IN (13) COMMENT = 'srv "t_r1d13c00 t_l1d13c00", table "t_d13c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p14 VALUES IN (14) COMMENT = 'srv "t_r1d14c00 t_l1d14c00", table "t_d14c00", lst "0 0" ' ENGINE = SPIDER, PARTITION p15 VALUES IN (15) COMMENT = 'srv "t_r1d15c00 t_l1d15c00", table "t_d15c00", lst "0 0" ' ENGINE = SPIDER)

On service front • DO_NOT_DELETE_ME

/spdsvc02/bootstrap# lsbootstrap.sh init_backend init_repli.sh run_middle08.sh sp_frontwrite.sqlDO_NOT_DELETE_ME init_front my_change_master_to run_middle16.sh sp_frontwritetable.sqlfront_file_bootstrap_table.cnf init_middle08 my_ping sp_backend.sql sp_middle08.sqlfront_file_bootstrap_table.kv.cnf init_middle16 run_backend.sh sp_frontread.sql sp_middle16.sqlfront_file_bootstrap_table.sql init_repli run_front.sh sp_frontwritegroup.sql

••

cat front_file_bootstrap_table.kv.cnf ddl=(id bigint unsigned not null primary key auto_increment, collection bigint unsigned , content mediumblob, inum bigint, istr varbinary(40), key idx1(inum ), key idx2(istr))ddlfront=(id bigint unsigned not null, collection bigint unsigned , content mediumblob, inum bigint, istr varbinary(40), key idx1(inum ), key idx2(istr))collection=collectionid=id tbl=tdb=spdcengine=tokudb

•./cluster_check.sh

Boosting •••

set transaction isolation level read uncommitted ; MariaDB [xxx]> select count(*) from xxx ;+----------+| 19598363 |+----------+

Disable background statistics to fetch once every 3600 queries

•| spider_sts_bg_mode | 0 || spider_sts_interval | 3600 || spider_sts_mode | 1 || spider_sts_sync | 1 |

•| spider_crd_bg_mode | 0 || spider_crd_interval | 3600 || spider_crd_mode | 1 || spider_crd_sync | 1 |

spider_casual_read=1

Load data••

⅙ to ⅛ compression

••

Small Downtime1.2.

No Downtime with rolling upgrade1.2.

Speed ••

––

Memory ••

Storage Engine •

GCloud Sizing••

Thank You

MariaDB - Percona · mysvc start --rid container#03 ... HpUx ip fs vg sync hpvm srp openha FreeBSD...

Documents