Date post: | 15-Jan-2015 |
Category: |
Technology |
Upload: | lookout |
View: | 790 times |
Download: | 1 times |
LOOKOUT!GROWING PAINS
KYLE BARTON, LOOKOUT, INC.
THEN AND NOW
• Getting a solid product off of the ground
• Dealing with a growing user base
• Where we are today
GETTING OFF THE GROUNDMVP
• Try to reproduce locally with a test
• Production debugging
• Eye balling change sets
• Synclogs
HOW WE DEALT
WE STARTED GROWING
• More Users
• More Traffic
• More Developers
• More Problems
• Ran out of integer space for primary keys
• Load spikes
• Clients checked in at the same time
• No way to tell clients to back off
PROBLEMS
• More code going out in each deploy
• Difficult to diagnose issues after deploy
• Pretty much guaranteed rollbacks on deploy
PROBLEMS CONT...
• Implemented “load shedding”
• Under high load respond to client with “come back later” message
• Automate with ganglia checks
• Randomized client check in
• Picture Backup disabled by default
LOAD SPIKES
INTEGER OVERFLOW
• Created new tables
• Big int primary and foreign keys
• Application code to use new and old tables
• New data goes to new table
• Old data moved on login
• Background Process to update untouched records
GIANT DEPLOYS
• Feature rollouts
• Slowly enable a new features
• Configuration
• Ability to turn off new code without rolling back
WHERE WE ARE TODAY
PROBLEMS
• Syncml
• One endpoint to service all types of requests
• Bloated request and responses
• Team Structure
• Communication
• Too many meetings
• Versioned RESTful APIs on existing architecture
• Eventually end of life old clients that haven’t upgraded
DEPRECATE SYNCML
• Persisted chat
• Smaller teams
• Better documentation
TEAM STRUCTURE AND COMMUNICATION
Keep in touch with
@lookout
/mylookout
blog.lookout.com
http://bit.ly/scaling-for-mobile