Post on 07-Jan-2017
transcript
Asya Kamsky, Lead Product Manager MongoDB
Diagnostics and Debugging 3.4
Asya Kamsky, Lead Product Manager MongoDB
Diagnostics and Debugging 3.4
“ I thought of my old teacher Joe Bell, ... of his eerie trick of spotting details. If he were a detective he would surely reduce this ... business to something nearer an exact science. —Arthur Conan Doyle
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
The image cannot be displayed. Your computer may not have enough memory to open the image, or the image may have been corrupted. Restart your computer, and then open the file again. If the red x still appears, you may have to delete the image and then insert it again.
Understanding The Patient
#MDBW16
Understanding The Patient
Understand the system
Knowledge
Monitor trends over time
Trends
Record all metrics "at rest"
Baseline
Gathering Data “Data! Data! Data! I can't make bricks without clay.”
-- Sherlock Holmes, The Adventure of the Copper Beeches
“There is nothing like first-hand evidence.” � -- Sherlock Holmes, A Study in Scarlet
Available Tools
#MDBW16
Available Tools: Say "AH"
#MDBW16
Available Tools: Say "AH"
#MDBW16
Available Tools: Say "AH"
#MDBW16
Available Tools: Say "AH"
> db.isMaster( ) > rs.conf( ) > rs.status( ) > sh.status( )
> db.version( ) > db.serverCmdLineOpts( )
#MDBW16
Available Tools: Say "AH"
#MDBW16
> show dbs > show collections > db.getCollectionNames().forEach(function(c) { printjson(db.getCollection(c).getIndexes()); });
Available Tools: Say "AH"
#MDBW16
Available Tools: Say "AH"
#MDBW16
Available Tools: Compass
#MDBW16
Available Tools: OS
% free % blockdev --report % dmesg % ulimit -a % ifconfig, ip <...>, iptables % iostat % netstat % top; htop % perf % iperf3
#MDBW16
Available Tools: OS for MongoDB % mongostat % mongotop
mongod logs
2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
2014-09-01T15:50:03.184-0004 [conn208] query
data.activity query: { $query: { _updated_at: { $gt: new
Date(1396459946346) }, _to: "_UserA" } }, $orderby:
{ _updated_at: -1 } } planSummary: IXSCAN { _to: 1,
_created_at: -1 } ntoreturn:100 ntoskip:0 nscanned:19692
nscannedObjects:19692 numYields:170 locks(micros) r:
283957 nreturned:65 reslen:24939 342ms
mongod logs
2015-05-26T23:33:32.774-0500 I
2015-05-28T12:37:11.440-0500 I
2015-05-28T12:38:35.839-0500 I
D, I, W, E, F
3.0 mongod logs
2015-05-26T23:33:32.774-0500 I COMMAND
2015-05-28T12:37:11.440-0500 I NETWORK
2015-05-28T12:38:35.839-0500 I QUERY
3.0
COMMAND, NETWORK, QUERY, REPL, ACCESS, INDEX, JOURNAL, SHARDING, WRITE, -
mongod logs
2015-05-26T23:33:32.774-0500 I COMMAND [conn45] command admin.$cmd command: listDatabases { listDatabases: 1.0 } ntoskip:0 keyUpdates:0 writeConflicts:0 numYields:0 reslen:393 locks:{ Global: { acquireCount: { r: 12 } }, Database: { acquireCount: { r: 6 } } } 321ms
2015-05-28T12:37:11.440-0500 I NETWORK [initandlisten] connection accepted from 127.0.0.1:48625 #183 (21 connections now open)
2015-05-28T12:38:35.839-0500 I QUERY [conn183] getmore tableau.flights201406 query: { origin_city_market_id: 31703.0 } cursorid:61957110347 ntoreturn:0 cursorExhausted:1 keyUpdates:0 writeConflicts:0 numYields:176 nreturned:22579 reslen:1332181 locks:{ Global: { acquireCount: { r: 354 } }, Database: { acquireCount: { r: 177 } }, Collection: { acquireCount: { r: 177 } } } 114ms
3.0 mongod logs
3.2 mongod logs
2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c
command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt: 4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }
keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1
keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187
locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms
3.2 mongod logs
2016-06-23T10:34:41.559-0700 I COMMAND [conn26] command test.c
command: find { find: "c", filter: { a: { $elemMatch: { v: { $gte: 3.0, $lt: 4.0 } } } } } planSummary: IXSCAN { a.v: 1.0 }
keysExamined:2 docsExamined:1 fromMultiPlanner:1 cursorExhausted:1
keyUpdates:0 writeConflicts:0 numYields:1 nreturned:1 reslen:187
locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_command 110ms
3.2 mongod logs
3.4 mongod logs
3.4 mongod logs
2016-06-01T15:30:04.373-0700 I COMMAND [conn99] command socialite.following command: aggregate { aggregate: "following", pipeline: [ { $match: { _f: "45705" } }, { $group: { _id: null, followees: { $addToSet: "$_t" } } }, { $lookup: { from: "following", localField: "followees", foreignField: "_f", as: "fofollowees" } }, { $project: { fofs: { $setUnion: [ "$followees", "$fofollowees._t" ] } } } ] } planSummary: IXSCAN { _f: 1, _t: 1 } keysExamined:1 docsExamined:0 numYields:11 reslen:316214 locks:{ Global: { acquireCount: { r: 3800 } }, Database: { acquireCount: { r: 1900 } }, Collection: { acquireCount: { r: 1900 } } } protocol:op_query 218ms
3.4 mongod logs
2016-06-25T23:38:27.346-0500 I WRITE [conn128] update ycsb.usertable query: { _id: "user7074965863272626663" } planSummary: IDHACK update: { $set: { field1: BinData(0, 23...) } } keysExamined:1 docsExamined:1 nMatched:1 nModified:1 numYields:1 locks:{ Global: { acquireCount: { r: 3, w: 3 } }, Database: { acquireCount: { w: 3 } }, Collection: { acquireCount: { w: 2 } }, Metadata: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } 11ms
> db.getLogComponents()
> db.getLogComponents(){ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},
"index" : { "verbosity" : -1},"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},
"replication" : { "verbosity" : -1},"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,
"journal" : { "verbosity" : -1}},"write" : { "verbosity" : -1}
}
> db.getLogComponents(){ "verbosity" : 1,
"accessControl" : { "verbosity" : -1},
"command" : { "verbosity" : -1},"control" : { "verbosity" : -1},
"executor" : { "verbosity" : -1},
"geo" : { "verbosity" : -1},"index" : { "verbosity" : -1},
"network" : { "verbosity" : -1},
"query" : { "verbosity" : -1},"replication" : { "verbosity" : -1},
"sharding" : { "verbosity" : -1},
"storage" : { "verbosity" : -1,"journal" : { "verbosity" : -1}},
"write" : { "verbosity" : -1},
"ftdc" : { "verbosity" : -1}}
> db.setLogLevel( logLevel, component )
> db.setLogLevel( 1, "sharding" )
> db.setLogLevel( 2, "query" )
"It is of the highest importance ... to be able to recognize, �out of a number of facts, which are incidental and which vital. " �� Sherlock Holmes, The Reigate Puzzle
https://github.com/rueckstiess/mtools
by Thomas Rückstieß
https://github.com/rueckstiess/mtools
by Thomas Rückstieß
#MDBW16
mplotqueries $ mplotqueries --help usage: mplotqueries [OPTIONS] [logfile [logfile ...]] A script to plot various information from logfiles. ...
#MDBW16
mplotqueries $ mplotqueries --help usage: mplotqueries [OPTIONS] [logfile [logfile ...]] A script to plot various information from logfiles. ...
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration)
#MDBW16
mplotqueries $ mplotqueries --help usage: mplotqueries [OPTIONS] [logfile [logfile ...]] A script to plot various information from logfiles. ...
optional arguments:
--type {nscanned/n,rsstate,connchurn,durline,histogram,range,scatter,event} type of plot (default=scatter with --yaxis duration) --group GROUP specify value to group on. All basic plot types can group on 'namespace', 'operation', 'thread', 'pattern' ...
"... what is out of the common is usually a guide� rather than a hindrance." � — Sherlock Holmes, A Study in Scarlet
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms 2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172 nreturned:56 reslen:18745 13086ms
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms 2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172 nreturned:56 reslen:18745 13086ms
mplotqueries
% mplotqueries firstmongo.log --type nscanned/n 2014-06-17T17:13:34.235 [conn1569841] query db1.coll query: { time: { $lt: "2014-06-17 17:13:31", $gte: "2014-06-17 17:04:31" } } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:5169727 nscannedObjects:5169727 keyUpdates:0 numYields:12492 locks(micros) r:37736571 nreturned:72 reslen:32707 30129ms 2014-06-17T17:13:47.607 [conn1569990] query db2.coll query: { time: { $lt: "2014-06-17 17:14:05", $gte: "2014-06-17 17:05:05" }, status: 8 } planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:2057564 nscannedObjects:2057564 keyUpdates:0 numYields:5008 locks(micros) r:11557172 nreturned:56 reslen:18745 13086ms
mplotqueries
% mplotqueries updates?.log
% mplotqueries updates?.log
% mplotqueries updates?.log
% mplotqueries updates?.log
% mplotqueries updates?.log
"Eliminate all other factors, and the one which remains must be the truth." Sherlock Holmes -The Sign of Four
#MDBW16
Available Tools
% mongostat
#MDBW16
mongostat
#MDBW16
--discover
mongostat
#MDBW16
mongostat insert query update delete getmore command % dirty % used flushes vsize res faults qr|qw ar|aw netIn netOut conn ReplSetName role ts
#MDBW16
Trends
#MDBW16
Available Tools
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
#MDBW16
Available Tools: in the Cloud
% mplotqueries –type connchurn
#MDBW16
db.currentOp()
#MDBW16
db.currentOp()
> db.currentOp({secs_running:{$gt:10}})
{ "desc" : "conn3482",
"threadId" : "140512575297280",
"connectionId" : 3482,
"client_s" : "10.0.149.179:55295",
"active" : true,
"opid" : "SocialiteAtlas-shard-2:55547103",
"secs_running" : 13,
"microsecs_running" : NumberLong(13483812),
"op" : "query",
"ns" : "db1.coll1",
"numYields" : 48815,
"query" : {
"field1" : 5,
#MDBW16
db.currentOp()
> db.currentOp({desc:/^conn/,secs_running:{$gt:0}}).inprog.forEach(function(op) { print(op.opid, op.op, op.ns, op.secs_running); });
SocialiteAtlas-shard-0:68479014 query socialite.content 3
SocialiteAtlas-shard-0:68479730 query socialite.content 2
SocialiteAtlas-shard-0:68480174 query socialite.content 1
SocialiteAtlas-shard-0:68479234 query socialite.content 2
SocialiteAtlas-shard-0:68480136 query socialite.content 1
SocialiteAtlas-shard-0:68479529 query socialite.content 2
SocialiteAtlas-shard-0:68480182 query socialite.content 1
SocialiteAtlas-shard-0:68480261 query socialite.content 1
>
db.killOp ( opid )
#MDBW16
Asya Kamsky, Lead Product Manager MongoDB
Diagnostics and Debugging 3.4
Title Slide Option 1 Click here to add speaker name and title
Title Slide Option 2 Click here to add speaker name and title
This is divider slide option 2
#MDBW16
Click to add title: keep it to 56 characters w/spaces
This is a typical content slide with full width body.
• First level bullet list • Second level bullet list
• Third level bullet list
#MDBW16
Two content
Click to add text.
• First level bullet list • Second level bullet list
• Third level bullet list
Click to add text.
• First level bullet list • Second level bullet list
• Third level bullet list
#MDBW16
Left content
Click to add text.
• First level bullet list • Second level bullet list
#MDBW16
Left content
Click to add text.
• First level bullet list • Second level bullet list
#MDBW16
Pie Chart
64%
25%
11%
1st Qtr 2nd Qtr 3rd Qtr
1st Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .
2nd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .
3rd Quarter Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique .
#MDBW16
Bar Graph
0
1
2
3
4
5
6
Category 1 Category 2 Category 3 Category 4
Chart Title
Series 1 Series 2 Series 3
#MDBW16
Column Header 1 Column Header 2 Column Header 3 Column Header 4 Column Header 5
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet
Table
#MDBW16
Title only
#MDBW16
Coding Example – Light Background
// Retrieve var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});
#MDBW16
Coding Example – Dark Background
// Retrieve var MongoClient = require('mongodb').MongoClient;
// Connect to the db
MongoClient.connect("mongodb://localhost:27017/exampleDb", function(err, db) {
if(err) { return console.dir(err); }
db.collection('test', function(err, collection) {});
db.collection('test', {w:1}, function(err, collection) {});
db.createCollection('test', function(err, collection) {});
db.createCollection('test', {w:1}, function(err, collection) {});
});
#MDBW16
List slide – can also be used for agenda
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 01 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 03 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 02
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 05 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 06 Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem Ipsum 05
#MDBW16
Columns and icons with copy (option 1)
Lorem ipsum dolor sit amet, onsectetur
adipiscing elit. Praesent sodales odio sit amet odio tristique.
Linked Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique.
Planning Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique.
Writing Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique.
Research
#MDBW16
Columns and icons with copy (option 2)
Lorem ipsum dolor sit amet, onsectetur
adipiscing elit. Praesent sodales odio
sit amet odio tristique .
Linked Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique .
Planning Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique .
Writing Lorem ipsum dolor sit
amet, onsectetur adipiscing elit.
Praesent sodales odio sit amet odio tristique .
Research
#MDBW16
Timeline or progress
2013
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
2014
2015
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique sit elit.
2016
“ Quote sample. Lorem ipsum dolor sit amet, onsectetur adipiscing elit amet sodales. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit. Praesent sodales odio sit amet odio tristique. Lorem ipsum dolor sit amet, onsectetur adipiscing elit.”