Monitoring MySQL with Prometheus

Prometheus

Ben Kochie - Prometheus Lead - GitLab

Prometheus

About Prometheus● Metrics collection● Time-series database● Graphing● Alerting

Prometheus

Performance

Prometheus

Performance● Millions of Timeseries● 800k samples per second● 1.3 bytes per sample (varbit)● Thousands of targets

Prometheus

In Production● 1500+ Tables● 160+ Databases● 30+ Clusters● 250+ Servers● 6000+ Events Digests

Prometheus

Real-World Load● 1.5 million timeseries● 80k samples per second● 400GB (28 days, old encoding)● 10 cores (2.0ghz Ivy-Bridge)

Prometheus

Prometheus Basics

Prometheus

Prometheus

mysqld_exporter Basics

Prometheus

https://github.com/prometheus/mysqld_exporter

CREATE USER 'exporter'@'localhost' IDENTIFIED BY 'pass';GRANT PROCESS, REPLICATION CLIENT ON *.* TO 'exporter'@'localhost';GRANT SELECT ON performance_schema.* TO 'exporter'@'localhost';

DATA_SOURCE_NAME=’exporter:pass@(localhost:3306)/’

Prometheus

Start mysqld_exporter./mysqld_exporter \ -collect.info_schema.innodb_metrics \ -collect.info_schema.userstats \ -collect.perf_schema.eventsstatements \ -collect.perf_schema.indexiowaits \ -collect.perf_schema.tableiowaits

Prometheus

Exporter Output (http://localhost:9104/metrics)

# HELP mysql_global_status_commands_total Total number of executed MySQL commands.# TYPE mysql_global_status_commands_total countermysql_global_status_commands_total{command="admin_commands"} 795205mysql_global_status_commands_total{command="alter_db"} 0

# HELP mysql_global_variables_read_only Generic gauge metric from SHOW GLOBAL VARIABLES.# TYPE mysql_global_variables_read_only gaugemysql_global_variables_read_only 1

# HELP mysql_slave_status_seconds_behind_master Generic metric from SHOW SLAVE STATUS.# TYPE mysql_slave_status_seconds_behind_master untypedmysql_slave_status_seconds_behind_master 0

Prometheus

Configuration

Prometheus

global: scrape_interval: 15s evaluation_interval: 15s

scrape_configs: - job_name: 'mysql' target_groups: - targets: - localhost:9104

Prometheus

PromQL

Prometheus

rate(mysql_global_status_connections[2m])

sum( rate(mysql_global_status_commands_total[2m]))

Prometheus

sum( rate(mysql_global_status_commands_total[2m]))

Prometheus

sum by (instance) ( rate(mysql_global_status_commands_total[2m]))

Prometheus

sum by (command) ( rate(mysql_global_status_commands_total[2m]))

Prometheus

sum by (command) ( rate(mysql_global_status_commands_total{ command=~"(insert|update|delete)" }[2m]))

Prometheus

mysql_slave_status_seconds_behind_master

Prometheus

mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

Prometheus

Rules Files

Prometheus

mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

Prometheus

mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

Prometheus

mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

ALERT MySQLSlaveLag IF mysql_slave_lag_seconds > 300 FOR 1m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }

Prometheus

mysql_slave_lag_seconds = mysql_slave_status_seconds_behind_master - mysql_slave_status_sql_delay

ALERT MySQLSlaveLag IF (mysql_slave_lag_seconds > 60 AND predict_linear(mysql_slave_lag_seconds[2m],60*60)) > 0 FOR 2m LABELS { severity = "warning" } ANNOTATIONS { summary = "Slave lag is too damn high." }

Prometheus

Examples/Graphs

Prometheus

Basic Traffic Overview

Prometheus

sum by (digest_text) (rate( mysql_perf_schema_events_statements_tmp_disk_tables_total[2m]))

Prometheus

topk(5, sum by (schema,digest_text) (rate(mysql_perf_schema_events_statements_total[5m])))

Prometheus

The Future

Prometheus

Demo Time

Prometheus