promgen - prometheus management tool
tokuhirom
Web Application Engineer
• Server side engineer
• Java
• Perl
• Server Admin
www1.example.com
node_exporter (httpd)
jmx_exporter (httpd)
Periodic GET request
It’s good for lazy web programmers
Get data → Get stats
Async and lazy
Easy to integrate with grafana
But there’s no server list management tool
official method to manage servers, is writing YAML.
With consul or something
YAML file contains: exporter port number
alerting rules
Server admins need to write BIG YAML FILE.
Editing YAML via github pull-request
We want to manage it via web app.
We wrote web application to manage server list.
We are managing the server list in our deployment tool
promgen can integrate with our deployment tool
(Of course, it’s pluggable)
Note: We haven’t use consul or similar
tool(now).
Promgen
• Simple prometheus manager app
• exporter port management
• alert management
ProjectService
Project
Project
Host
Host
Host
Data structure
Managed by Deploy tool
Farm
Farm
Farm
Alerting
Alert Manager Hipchatpromgen-alerting
Web hook
Mail
MySQL
Architecture
• Ruby
• Sequel
• MySQL
• erubis
• sinatra
I want to run it on prometheus host.
We shouldn’t use much memory
Part 2. Monitoring Java
processes via Prometheus
There are 3 patterns
1. Collect data from MBean in target JVM
App JVM (Runs jmx_exporter jetty) prom
HTTP
• Pros
• Easy to run
• Cons
• Hard to maintain YAML file
- pattern: '^com.zaxxer.hikari<type=Pool \((\w+)\)><>ThreadsAwaitingConnection'
name: hikaricp_pool_threads_awaiting_connection
labels:
source: "$1"
help: "HikariCP threads awaiting connection(pending threads) $1"
type: GAUGE
2. Collect data from JMX outside from target JVM
App JVM
jmx_exporter JVM prom
HTTPJMX
• Pros
• Easy to deploy
• Safety
• Cons
• Memory overhead
• Hard to maintain YAML file
3. Collect data by simpleclient_java
App JVM (Run simpleclient_java jetty) prom
HTTP
• Pros
• Clean
• lightweight
• no YAML required
• Cons
• Requires coding
simpleclient_java
simpleclient_dropwizard
Drop wizard metrics
MetricRegistryclient_java
simpleclient_hotspot
JVM core client_java
Default
simpleclient_log4j/logback
log4j/logback client_java
Error/Warn rate
simpleclient_springboot
Spring Boot Metrics client_java
NEW!
simpleclient_jetty
Jetty requests
status etc.
client_java
NEW!
simpleclient_hikaricp
HikariCP client_java
REJECTED!