Post on 29-Jul-2020
transcript
Monitoring MySQL with Prometheus
Prometheus
Ben Kochie - Prometheus Lead - GitLab
Prometheus
About Prometheus● Metrics collection● Time-series database● Graphing● Alerting
Prometheus
Performance
Prometheus
Performance● Millions of Timeseries● 800k samples per second● 1.3 bytes per sample (varbit)● Thousands of targets
Prometheus
In Production● 1500+ Tables● 160+ Databases● 30+ Clusters● 250+ Servers● 6000+ Events Digests
Prometheus
Real-World Load● 1.5 million timeseries● 80k samples per second● 400GB (28 days, old encoding)● 10 cores (2.0ghz Ivy-Bridge)
Prometheus
Prometheus Basics
Prometheus
Prometheus
mysqld_exporter Basics
Prometheus
https://github.com/prometheus/mysqld_exporter
CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'pass';GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'exporter'@'localhost';GRANT SELECT ON performance_schema.* TO 'exporter'@'localhost';
DATA_SOURCE_NAME=’exporter:pass@(localhost:3306)/’
Prometheus
Start mysqld_exporter./mysqld_exporter \ -collect.info_schema.innodb_metrics \ -collect.info_schema.userstats \ -collect.perf_schema.eventsstatements \ -collect.perf_schema.indexiowaits \ -collect.perf_schema.tableiowaits
Prometheus
Exporter Output (http://localhost:9104/metrics)
# HELP mysql_global_status_commands_total Total number of executed MySQL commands.# TYPE mysql_global_status_commands_total countermysql_global_status_commands_total{command="admin_commands"} 795205mysql_global_status_commands_total{command="alter_db"} 0
# HELP mysql_global_variables_read_only Generic gauge metric from SHOW GLOBAL VARIABLES.# TYPE mysql_global_variables_read_only gaugemysql_global_variables_read_only 1
# HELP mysql_slave_status_seconds_behind_master Generic metric from SHOW SLAVE STATUS.# TYPE mysql_slave_status_seconds_behind_master untypedmysql_slave_status_seconds_behind_master 0
Prometheus
Configuration
Prometheus
global: scrape_interval: 15s evaluation_interval: 15s
scrape_configs: - job_name: 'mysql' target_groups: - targets: - localhost:9104
Prometheus
PromQL
Prometheus
rate(mysql_global_status_connections[2m])
sum( rate(mysql_global_status_commands_total[2m]))
Prometheus
sum( rate(mysql_global_status_commands_total[2m]))
Prometheus
sum by (instance) ( rate(mysql_global_status_commands_total[2m]))
Prometheus
sum by (command) ( rate(mysql_global_status_commands_total[2m]))
Prometheus
sum by (command) ( rate(mysql_global_status_commands_total{ command=~"(insert|update|delete)" }[2m]))
Prometheus
mysql_slave_status_seconds_behind_master
Prometheus
mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
Rules Files
Prometheus
mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
Prometheus
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
ALERT MySQLSlaveLag IF mysql_slave_lag_seconds > 300 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }
Prometheus
mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay
ALERT MySQLSlaveLag IF (mysql_slave_lag_seconds > 60 AND predict_linear(mysql_slave_lag_seconds[2m],60*60)) > 0 FOR 2m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }
Prometheus
Examples/Graphs
Prometheus
Basic Traffic Overview
Prometheus
sum by (digest_text) (rate( mysql_perf_schema_events_statements_tmp_disk_tables_total[2m]))
Prometheus
topk(5, sum by (schema,digest_text) (rate(mysql_perf_schema_events_statements_total[5m])))
Prometheus
The Future
Prometheus
Demo Time
Prometheus