Date post: | 13-Jan-2015 |
Category: |
Documents |
Upload: | devops-israel |
View: | 3,334 times |
Download: | 3 times |
Continuous Delivery @ Wix
DevOpsCon
2013
Yaniv Even-Haim
VP R&D
Wix Intro
Wix in numbers • 29,000,000 registered users from 233 countries
• ~1,200,000 new users every month
• ~ 30,000 new websites every day
• 26,000,000 web sites
• 800,000 mobile sites
• 1Tbyte of media uploaded every day
• More than 1 Billion users media files
• 250+ servers in 3 data centers
• 350 employees
• 100 R&D people
A story on Wix time machine
CD is business concern
Traditional “Economies of scale” threaten by cloud technology
Be faster than small startups
CD “Economies of Agility “ = Velocity X Size
20
10
-8
20
10
-9
20
10
-…
20
10
-…
20
10
-…
20
11
-1
20
11
-2
20
11
-3
20
11
-4
20
11
-5
20
11
-6
20
11
-7
20
11
-8
20
11
-9
20
11
-…
20
11
-…
20
11
-…
20
12
-1
20
12
-2
20
12
-3
20
12
-4
Revenue per month
CD is culture & mindset • Trust the developers
– Empower developers to change production
– Developer knows his system best
CD is culture & mindset • Automation as a default choice
– no more “is it worth to automate ? ”
– Everything should be automated
• Product/Dev/QA/Ops boundaries are going down
– Everyone responsible for everything
Welcome to the twilight zone
• Move fast - Less formality
– Corridor - IN , Meeting Room - Out
Welcome to the twilight zone
Get out of thought land
• The Law of Failure
– Most new “its” will fail even if they are flawlessly executed
• Invest less, in-touch less , better ability to admit it fail
– Data beats opinions - Let the market decide
make sure you building the right it before build it right
Some Best Practices
• Everyone develops on the Trunk/Master
– Release frequent – small pieces of functionality
– It’s the developer responsibility not to break anything
• Code can get to production at anytime
– TDD & integration test
• Use Feature Toggle
– Unbaked new code can go to production – no harm done
– New code goes with a guard – use new or old code
Some Best Practices
• Backwards and Forwards compatible
– Each component has to function with latest, next or prior version of other components (including DBs)
• Gradual Deployment & Self Test
– Deploy new version to one server and perform self test. If it passes, continue deployment to other servers
• A/B Testing
– Open a new feature to a percent of your users. Is it better?
– Deployment is an engineering decision
CD – prepare to invest….. • Dev infrastructure - Refactor , Refactor, Refactor
• Testing infrastructure & know how
• Deployment infrastructure & tools
• Automation , Automation , Automation
• Monitoring (business and technical)
– hundreds of aspects
– thresholds use is a Must
– Monitor business KPIs
– Internal & external
– Endless Tuning & learning
• Deployment complexity
– More services
– More dependencies
– Multi data centers
• Development complexity
– Very hard to work on master/trunk with 30+ dev on same code base
– backward and forward comptability is not easy
• Wider set of technologies ,tools & languages
– Higher level of infrastructure investments
– Harder to have all around players
– More communication/sync required
Scaling challenges
Scaling challenges – Quality
– Testing coverage challenge
• Use cases increases
• Browsers wild west
• Tablets and mobile variety
– Feature toggle combinations
• Dozens of parallel feature toggles
– Production frequent changes
• Services are updated daily
• Toggles switch on/off on daily basis
Scaling challenges - Availability • Service Availability
• Cost of fault/downtime increasing as business grow
• Service downtime during deployment is unacceptable
• Human mistakes are inventible when changing production daily
Scaling challenges – Monitoring
• Monitoring in multi technology env
• Unified dashboard require major effort
• Reduce alert noises resulted by deployments
• Identify root cause is not simple
– Some faults does not have immediate impacts
– Cascading effects
• Reliable business KPI without false positive
–Marketing changes
– Special events (holidays, sport event etc)
Scaling challenges – Product • Product MVP practices
– Does MVP meet your product standards?
• What about tooltip,help,first time ux, etc.. ?
– How to define a product that can be developed in a day ?
– And that can win in a/b test …
To Be
Implemented
Scaling challenges – Org • Complex Matrix Management
- People moving between teams frequently
- One can have several managers on different tasks
- Hard both to managers and employees
- Everyone should master the matrix management skills
Wix Developer Lifecycle
Wix Developer Lifecycle
New Relic - Monitoring
Application dashboard
Chix – Staging Deployment
It is not all or nothing game - it is
continuous journey