Jenni Snyder@jcsuperstar

WE'RE NOT YELLING AT YOUAvoiding perceived conflict

Yelp’s MissionConnecting people with great

local businesses.

Yelp StatsAs of Q1 2016

90M 3270%102M

● MySQL DBA since 2004● At Yelp 5 years● CS + Sociology = <3● Mom

About Me

OPS ROCKS!

About Yelp Operations

DEPUTIES ROCK!

"Ops deputies are for life, not just for your birthday."

Yelp Ops Deputies

So, What's the Problem?

Paging people to deliver bad news

It's Easy to Fall Into Old Stereotypes

It's Easy to Jump to Conclusions

"Devs Just Want to Change Stuff"

Icons by Freepik CC 3.0

"Ops Wants to Keep Things Frozen"

Icons by Freepik CC 3.0

Change is Good

Icons by Freepik CC 3.0

Stability is Good

Icons by Freepik CC 3.0

People Want to Do the Right Thing!

● Limited developer disk space, servers● Ops would get paged when it filled up

Problemtunity #1 - Disk Space

Devs Be Like

Icons by Freepik CC 3.0

This will rock!

Logging, FTW!

Ops Be Like

Icons by Freepik CC 3.0

Ops Feel Like

Especially If

Devs Feel Like

● Long-running queries and txns consume resources● DBAs gets paged● Had to manually track down, kill & ticket queries

Problemtunity #2 - Long Queries/Txns

Devs Be Like

Icons by Freepik CC 3.0

This will rock!

select * from rad

DBAs Be Like

Icons by Freepik CC 3.0

DBAs Feel Like

Especially If

Devs Feel Like

Dev vs Ops

Dev + Ops = <3

1. Set expectations2. Automation3. Include the right people4. Metrics

Solution

Icons by Freepik CC 3.0

1. Fixed tools 2. Auto backup/cleanup3. motd early warning4. Easy to move to shiny new machines5. Interns work on seasonal hosts

Disk Space

1. GrossQueryChecker rocks2. Anemometer3. Query Annotations4. Pt-kill, long-running transaction killah5. Monitors - IRC, tickets6. Hecka logs7. Query Annotations

DREAM: automatic tix in right projects for queries IN STAGE

Long Queries/txns

● People want to do the right thing● Make the right thing to do the easiest● Fix the problem at the right place

Remember

Icons by Freepik CC 3.0

"The only surprise people like is a birthday cake."- me

No One Likes Bad News

Icons by Freepik CC 3.0

● Failed tests● motd● IRC● Email● Jira tickets● Pages

Send Signals to the Right Place!

● Tests!● Monitors!● Graphs!

Make Tools to Discover Info

Icons by Freepik CC 3.0

Did I Say Metrics Yet?

● Encourage participation● We make it easy to find code● Metrics make it easier to make decisions

Make Feedback/Contributions Easy

Icons by Freepik CC 3.0

● I'm out of icons● Make it easiest to do the right thing● Use the right processes & tooling● Keep track of how it's working

Conclusion

@YelpEngineering

fb.com/YelpEngineers

engineeringblog.yelp.com

github.com/yelp