Post on 16-Mar-2018
transcript
Jenni Snyder@jcsuperstar
WE'RE NOT YELLING AT YOUAvoiding perceived conflict
Yelp’s MissionConnecting people with great
local businesses.
Yelp StatsAs of Q1 2016
90M 3270%102M
● MySQL DBA since 2004● At Yelp 5 years● CS + Sociology = <3● Mom
About Me
OPS ROCKS!
About Yelp Operations
DEPUTIES ROCK!
"Ops deputies are for life, not just for your birthday."
Yelp Ops Deputies
So, What's the Problem?
Paging people to deliver bad news
It's Easy to Fall Into Old Stereotypes
It's Easy to Jump to Conclusions
"Devs Just Want to Change Stuff"
Icons by Freepik CC 3.0
"Ops Wants to Keep Things Frozen"
Icons by Freepik CC 3.0
Change is Good
Icons by Freepik CC 3.0
Stability is Good
Icons by Freepik CC 3.0
People Want to Do the Right Thing!
● Limited developer disk space, servers● Ops would get paged when it filled up
Problemtunity #1 - Disk Space
Devs Be Like
Icons by Freepik CC 3.0
This will rock!
Logging, FTW!
Ops Be Like
Icons by Freepik CC 3.0
Ops Feel Like
Especially If
Devs Feel Like
● Long-running queries and txns consume resources● DBAs gets paged● Had to manually track down, kill & ticket queries
Problemtunity #2 - Long Queries/Txns
Devs Be Like
Icons by Freepik CC 3.0
This will rock!
select * from rad
DBAs Be Like
Icons by Freepik CC 3.0
DBAs Feel Like
Especially If
Devs Feel Like
Dev vs Ops
Dev + Ops = <3
1. Set expectations2. Automation3. Include the right people4. Metrics
Solution
Icons by Freepik CC 3.0
1. Fixed tools 2. Auto backup/cleanup3. motd early warning4. Easy to move to shiny new machines5. Interns work on seasonal hosts
Disk Space
1. GrossQueryChecker rocks2. Anemometer3. Query Annotations4. Pt-kill, long-running transaction killah5. Monitors - IRC, tickets6. Hecka logs7. Query Annotations
DREAM: automatic tix in right projects for queries IN STAGE
Long Queries/txns
● People want to do the right thing● Make the right thing to do the easiest● Fix the problem at the right place
Remember
Icons by Freepik CC 3.0
"The only surprise people like is a birthday cake."- me
No One Likes Bad News
Icons by Freepik CC 3.0
● Failed tests● motd● IRC● Email● Jira tickets● Pages
Send Signals to the Right Place!
● Tests!● Monitors!● Graphs!
Make Tools to Discover Info
Icons by Freepik CC 3.0
Did I Say Metrics Yet?
● Encourage participation● We make it easy to find code● Metrics make it easier to make decisions
Make Feedback/Contributions Easy
Icons by Freepik CC 3.0
● I'm out of icons● Make it easiest to do the right thing● Use the right processes & tooling● Keep track of how it's working
Conclusion
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp