Date post: | 15-Jan-2015 |
Category: |
Technology |
Upload: | will-button |
View: | 360 times |
Download: | 1 times |
@wfbutton
Phoenix MUGSharding: A Case Study
Overview
• What is sharding• How we knew it was time to
shard• What to shard• Choosing a shard key• Building servers
• Integrating sharding into a production environment
• Monitoring for success/failure
• Lessons learned• Things you can do today
About Me
• DevOps/IT/DBA for myList.com
• Extensive background in both development and ops, specifically in scalability and sustainability
Will Button | @wfbutton | google.com/+WillButton
What is sharding?
Config Servers Mongos
• Stores metadata for the cluster• Not a replica set• Metadata consists of:
• collections• shards• chunks• mongos instances
• routing service for mongo shards• likely to run on application server• apps talk to mongos in sharded • environment
When Should I Shard?
When Should I Shard?{"ts" : ISODate("2013-11-01T01:34:30.683Z"),"op" : "query","ns" : "MyListContent.RepoThing","query" : {"query" : {"provId" : "cae56942-5c9c-776c-0506-2c9f4092e107","provUnifiedId" : "40233411"},"$readPreference" : {"mode" : "primary"}},"ntoreturn" : 0,"ntoskip" : 0,"nscanned" : 58,"keyUpdates" : 0,"numYield" : 16,"lockStats" : {"timeLockedMicros" : {"r" : NumberLong(1260461),"w" : NumberLong(0)},"timeAcquiringMicros" : {"r" : NumberLong(1275073),"w" : NumberLong(2369)}},"nreturned" : 57,"responseLength" : 91643,"millis" : 1200,"client" : "10.110.1.27","user" : ""}
58 records scanned57 documents returned
1200 milliseconds
YIKES!
What To Shardsocialcatalog03:SECONDARY> db.system.profile.aggregate( ... { $group: ... { _id: "$ns", count: ... { $sum: 1 } ... }... } ... ){"result" : [{"_id" : "admin.$cmd","count" : 1},{"_id" : "MyListContent.BrandPage","count" : 97},{"_id" : "MyListContent.RepoThingUpdate","count" : 50},{"_id" : "MyListContent.RepoThing","count" : 1824}],"ok" : 1}
system.profile collection is your friend!
Choosing A Shard Key
• Next to getting married, the most important decision you’ll ever make
Choosing A Shard Key
Collection:stuffShard key: _id
0…………..100…………..200…………..300…………..400…………..500…………..600
Shard 1 Shard 2 Shard 3
Adding New Servers
• Expanding production• Using Amazon EC2• Updating production• Does dev match prod?
Build EC2
Image
Clone Instances
Update conf rs.init()
Shard: Actual Steps
mongos> db.BrandPage.ensureIndex( { "_id": "hashed" } )
mongos> sh.shardCollection("MyListContent.BrandPage", { "_id": "hashed" })
Monitoring Shard Status
Monitoring Shard Statusmongos> db.BrandPage.getShardDistribution()
Shard socialcatalog03 at socialcatalog03/10.110.1.148:27018,10.110.3.215:27018,10.110.4.142:27018 data : 1.26GiB docs : 3334394 chunks : 41 estimated data per chunk : 31.49MiB estimated docs per chunk : 81326
Totals data : 1.26GiB docs : 3334394 chunks : 41 Shard socialcatalog03 contains 100% data, 100% docs in cluster, avg obj size on shard : 406B
mongos> db.BrandPage.getShardDistribution()
Shard rs210 at rs210/10.110.1.10:27018,10.110.1.110:27018,10.110.1.147:27018 data : 54.48MiB docs : 122774 chunks : 7 estimated data per chunk : 7.78MiB estimated docs per chunk : 17539
Shard rs220 at rs220/10.110.1.117:27018,10.110.1.149:27018,10.110.1.252:27018 data : 54.09MiB docs : 122151 chunks : 7 estimated data per chunk : 7.72MiB estimated docs per chunk : 17450
Shard rs310 at rs310/10.110.1.146:27018,10.110.1.197:27018,10.110.1.220:27018 data : 54.65MiB docs : 123138 chunks : 7 estimated data per chunk : 7.8MiB estimated docs per chunk : 17591
Shard rs320 at rs320/10.110.1.112:27018,10.110.1.150:27018,10.110.1.26:27018 data : 54.63MiB docs : 123163 chunks : 7 estimated data per chunk : 7.8MiB estimated docs per chunk : 17594
Shard socialcatalog02 at socialcatalog02/10.110.1.184:27018,10.110.1.222:27018,10.110.1.84:27018 data : 46.54MiB docs : 105031 chunks : 6 estimated data per chunk : 7.75MiB estimated docs per chunk : 17505
Shard socialcatalog03 at socialcatalog03/10.110.1.148:27018,10.110.1.16:27018,10.110.1.53:27018 data : 99.9MiB docs : 242755 chunks : 7 estimated data per chunk : 14.27MiB estimated docs per chunk : 34679
Totals data : 364.31MiB docs : 839012 chunks : 41 Shard rs210 contains 14.95% data, 14.63% docs in cluster, avg obj size on shard : 465B Shard rs220 contains 14.84% data, 14.55% docs in cluster, avg obj size on shard : 464B Shard rs310 contains 15% data, 14.67% docs in cluster, avg obj size on shard : 465B Shard rs320 contains 14.99% data, 14.67% docs in cluster, avg obj size on shard : 465B Shard socialcatalog02 contains 12.77% data, 12.51% docs in cluster, avg obj size on shard : 464B Shard socialcatalog03 contains 27.42% data, 28.93% docs in cluster, avg obj size on shard : 431B
Sharding takes time…
• But check the logs
Thu Nov 21 00:19:35.964 [Balancer] caught exception while doing balance: error checking clock skew of cluster 10.110.0.251:27019,10.110.3.87:27019,10.110.4.225:27019 :: caused by :: 13650 clock skew of the cluster 10.110.0.251:27019,10.110.3.87:27019,10.110.4.225:27019 is too far out of bounds to allow distributed locking.
Thu Nov 21 21:25:16.249 [conn84709] about to log metadata event: { _id: "aws-prod-mongo301-2013-11-21T21:25:16-528e7a3c374ed2e78b6298e4", server: "aws-prod-mongo301", clientAddr: "10.110.1.71:43357", time: new Date(1385069116248), what: "moveChunk.from", ns: "MyListContent.BrandPage", details: { min: { _id: -7394546541005003026 }, max: { _id: -6937685518831975781 }, step1 of 6: 0, note: "aborted" } }
Tips, Tricks, Gotchas
• Always use 3 config servers• Always use NTP• Always use CNAMES• Always specify configdb servers in the same
order• Shard early, shard often
Things You Can Do Today
• Enable/analyze system.profile• Identify long running queries• Review indexes, queries and performance• Verify replica sets are in sync• Setup alerting for replica set sync• Replica sets are not backups• Schedule a data review with the devs to plan
sharding strategies
Sharding MongoDB:A Case Study
www.two4seven.me/shardingSay hi! @wfbutton