Confidential

Become a MongoDB DBA

How to Automate, Monitor and Manage your MongoDB Servers

Art van Scheppingen, Senior Support Engineer

Confidential

Agenda

☐ MongoDB for DBAs☐ Deploying MongoDB☐ Automation☐ Monitoring☐ Managing☐ Orchestration systems

Confidential

MongoDB for DBAs

Confidential

MongoDB: why should I install it?

☐ Not everyone is happy with (My)SQL☐ Some storage needs differ from what (My)SQL can offer☐ Complex problems require different storage systems

☐ Alternative data storage☐ Polyglot persistence

Confidential

Polyglot Persistence?

Does a carpenter only use one tool?

Confidential

Polyglot Persistence?

Carpenters use many tools

Confidential

Monoglot Persistence

Confidential

Polyglot Persistence

Confidential

How different is MongoDB from MySQL?

☐ Document store (JSON)☐ Flexible schemas☐ Out of the box read scaling (replicaSets)☐ Out of the box write scaling (sharding)☐ Enthusiastic development community

Confidential

Eventual consistency

Confidential

How similar is MongoDB to MySQL?

☐ You still need to do the database basics☐ Deployment☐ Management☐ Monitoring☐ Backups☐ Scaling/Sharding

☐ It is less different than you think!

Confidential

Deploying MongoDB

Confidential

Install MongoDB

☐ Install MongoDB ☐ Download packages☐ Set up repositories

☐ Watch log file☐ /var/log/mongodb/mongod.log

[initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 },

Confidential

Configure MongoDB

☐ MongoDB Configuration file is in YAML# network interfacesnet: port: 27017# bindIp: 127.0.0.1 # Listen to local interface only, comment to listen on all interfaces.

☐ Restart mongod

Confidential

Install a second node

☐ Install packages☐ Change configuration

Confidential

Install a third node

☐ Install packages☐ Change configuration☐ This is getting boring…☐ And what if we deploy a sharded cluster with five shards?

Confidential

Automation

Confidential

Deployment Automation

☐ Deployment automation tools☐ Chef☐ Puppet☐ Ansible☐ Saltstack

Confidential

Chef

☐ Use mongodb3 cookbook☐ https://supermarket.chef.io/cookbooks/mongodb3☐ Others are not well maintained

☐ Handles configuration☐ Supports AWS☐ Supports replication and sharding☐ Supports mms_automation_agent

Confidential

Puppet

☐ Puppetlabs MongoDB module☐ https://github.com/puppetlabs/puppetlabs-mongodb☐ Still in beta: may change in the future☐ Well documented

☐ Handles☐ Installation☐ Configuration☐ Replication & sharding☐ Crude user management (admin user)

Confidential

Ansible

☐ Ansible MongoDB example☐ https://github.com/ansible/ansible-examples/tree/master/mongodb

☐ Installs a sharded MongoDB cluster☐ Expects RHEL/Centos 6 hosts☐ Covers only one use case

☐ You can alter a playbook, right?

Confidential

Ansible (cont.)

☐ Stouts.mongo Ansible role☐ https://github.com/Stouts/Stouts.mongodb

☐ Handles☐ Installation☐ Configuration☐ Replication

☐ Supports mms_automation_agent

Confidential

Saltstack

☐ MongoDB Formula☐ https://github.com/saltstack-formulas/mongodb-formula

☐ Handles☐ Installation☐ Configuration☐ Replication & Sharding

Confidential

Monitoring & Trending

Confidential

Confidential

Monitoring vs Trending

☐ Monitoring☐ Keeps an eye on your systems☐ Will alert if a threshold is met

☐ Trending☐ Insight into the system internals☐ Trends can warn you before anything has happened yet☐ Keep historical state/data

Confidential

Monitoring: Availability

☐ Do more than just opening a connection☐ Measure true status of nodes and cluster☐ Test read/write☐ Open essential databases and collections☐ Keep an eye on the replication lag☐ Increase oplog size?

☐ Check the full topology

Confidential

Monitoring Tools

☐ Open Source☐ Nagios☐ Zabbix

☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl

Confidential

Nagios

☐ Nagios-MongoDB☐ https://github.com/mzupan/nagios-plugin-mongodb/☐ Performs some very important checks☐ Replication lag☐ Lock time percentage☐ Index miss ratio

Confidential

Zabbix

☐ MongoDB Zabbix monitoring plugin☐ https://github.com/nightw/mikoomi-zabbix-mongodb-monitoring☐ All the necessary metrics and more☐ Entries in oplog

☐ Pre-canned triggers

Confidential

Trending: Why do we need trends?

☐ Trending☐ Plot trends of key (performance) metrics☐ Find problems before they arise☐ Pre-emptive problem management

☐ Trending tools☐ Granularity of sampling☐ More datapoints = better

Confidential

Trending: What metrics to store?

☐ Resource monitoring☐ CPU☐ Memory☐ IO capacity☐ Diskspace

☐ Replication monitoring☐ Replication status☐ Replication lag☐ Oplog size and usage

Confidential

Trending: Solutions

☐ Trending tools☐ Statsd/Grafana☐ Cacti☐ Zabbix

☐ Subscription based☐ MongoDB Cloud Manager☐ VividCortex☐ ClusterControl

Confidential

Cacti

☐ Percona MongoDB Monitoring Templates☐ https://www.percona.com/doc/percona-monitoring-plugins/1.1

/cacti/mongodb-templates.html

Confidential

Cacti

Confidential

Managing

Confidential

Types of backups

☐ Logical backups☐ Dump of your data

☐ Physical backups☐ File(system) copy of your data

Confidential

Logical backups

☐ Mongodump☐ MongoDB Backup☐ Mongob☐ Hotbackup (only available in Percona Server and in beta)

Confidential

Logical backups: mongodump

☐ Mongodump☐ BSON dump of the data☐ OEM tool☐ Works great but needs some wrapping

Confidential

Logical backups: MongoDB Backup

☐ MongoDB Backup☐ https://www.npmjs.com/package/mongodb-backup☐ Nodejs backup solution☐ CLI and API☐ Can stream backups

Confidential

Logical backups: Mongob

☐ Mongob☐ https://github.com/cmpitg/mongob☐ Python based CLI tool☐ MongoDB instance or bz2 target☐ Can copy data between collections☐ Incremental backups☐ Rate limiting

Confidential

Physical backups: Filesystem snapshots

☐ Filesystem snapshots☐ LVM☐ ZFS☐ XFS (xfs_freeze)☐ EBS

Confidential

Physical backups: Strata

☐ MongoRocks Strata☐ https://github.com/facebookgo/rocks-strata☐ Backs up on file level☐ Supports incremental backups☐ Queryable backups

Confidential

Percona MongoDB Consistent Backup

☐ Currently it is impossible to make a consistent backup of a sharded cluster☐ You can only make backups of individual hosts☐ Not all backups finish at the same time

Confidential

Noteworthy: Percona MongoDB Consistent Backup

Confidential

Percona MongoDB Consistent Backup

☐ Percona is working on a MongoDB consistent backup tool☐ Make a consistent backup of a sharded cluster☐ Snapshots individual hosts/replicasets in a sharded cluster☐ Oplog tailing☐ Point in time recovery possible☐ Still in development/beta

☐ Get it here:☐ https://github.com/Percona-Lab/mongodb_consistent_backup☐ Requires Python 2.7

Confidential

Why do schemas need changes?

☐ Application development☐ New features to existing modules☐ New modules☐ Data management and archiving

(partitions)☐ Schema optimization☐ Performance optimization

Confidential

Schema changes for MySQL

Confidential

Schema changes for MongoDB

Confidential

Schema Validation

☐ Meteor2 collection☐ https://github.com/aldeed/meteor-collection2☐ Ensures data gets written according to schema☐ Automatic: both client and server validate schema☐ Reactive: returns useful error messages

Confidential

Schema Validation

☐ Mongoose☐ http://mongoosejs.com/index.html☐ Node.js middleware for schema modelling and validation

Confidential

Schema Analyzer

☐ MongoDB Schema☐ https://github.com/mongodb-js/mongodb-schema☐ Schema analyzer

Confidential

Admin UIs

☐ Mongoclient☐ http://www.mongoclient.com/☐ Management / Backups

☐ PHPMoAdmin☐ http://www.phpmoadmin.com/

☐ Mongo Express☐ https://www.npmjs.com/package/mongo-express☐ Overview and data browser

☐ Robomongo☐ https://robomongo.org/download☐ Admin / data browser

Confidential

Orchestration Systems

Confidential

Orchestration systems: Mongoctl

☐ Mongoctl☐ https://github.com/mongolab/mongoctl☐ Python based CLI tool☐ Manages nodes, clusters and shards☐ Deployment☐ Configuration management☐ Scaling

Confidential

Orchestration systems: Percona Monitoring & Management

☐ PMM☐ https://www.percona.com/doc/percona-monitoring-and-management/

☐ Open Source Monitoring & Management framework☐ Can deploy, manage and monitor MySQL & MongoDB☐ Uses Prometheus and Grafana☐ Still in Beta

Confidential

Orchestration Systems: ClusterControl

☐ ClusterControl☐ http://www.severalnines.com☐ Deploy Mongo shards & replicasets☐ Monitor and trend☐ Manage configuration and backups☐ Scale

☐ Community edition

Confidential

Q & A

Confidential

Additional Resources

☐ Blog series: Become a MongoDB DBA☐ http://severalnines.com/blog-categories/mongodb

☐ Webinar series: Become a MongoDB DBA☐ http://severalnines.com/upcoming-webinars

☐ Visit our website for more resources!☐ http://www.severalnines.com