Date post: | 15-Apr-2017 |
Category: |
Technology |
Upload: | luca-milanesio |
View: | 433 times |
Download: | 2 times |
#GerritUserSummit
Zero Downtime GerritHow to upgrade your Gerrit master while keeping the balls rolling.
Luca [email protected]
@gitenterprise @gerritreview #GerritUserSummit
• Founded in the UK• HQ in London, Offices in the UK and Italy• Committed to OpenSource
About GerritForge
@gitenterprise @gerritreview #GerritUserSummit
• Co-founder and Director of GerritForge • over 20 years in Agile
Development and ALM• Contributor to many
OpenSource projects• Mind and hands of
@gitenterprise and @gerritreview Twitter feeds
About Luca Milanesio
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
•Gerrit is quite stable
•GerritHub uptime to date: 226 days
• Extensive unit & integration tests suite
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
BUT
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
"Sometimes" we need to upgrade
– Feb 2 gerrit-2.11.6– Feb 12 gerrit-2.11.7– Mar 3 gerrit-2.12.1– Mar 9 gerrit-2.11.8– Mar 11 gerrit-2.12.2– May 17 gerrit-2.11.9
– Jul 7 gerrit-2.12.3– Aug 24 gerrit-2.12.4– Aug 26 gerrit-2.11.10– Sep 24 gerrit-2.12.5– Oct 21 gerrit-2.13– Oct 21 gerrit-2.13.1– Oct 21 gerrit-2.13.2
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
How others are managing upgrades
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
Three flavors up upgrades1. Code-only : Typically patch-releases2. Code + DB Schema upgrade: Major releases3. Code + DB Schema upgrade + Repos upgrade: Painful
Major release
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
How Google is managing upgrades
@gitenterprise @gerritreview #GerritUserSummit
Life outside Google Data-Centers
Level of pain in the OpenSource Gerrit implementations1. Code-only
5 mins of fear2. Code + DB Schema upgrade
1/2 days of planning + 15/20' outage3. Code + DB Schema upgrade + Repos upgrade:
1/2 months of planning + leap of faith
@gitenterprise @gerritreview #GerritUserSummit
Levels of Zero-Downtime
0. Outage1. Server always-on and reachable2. Gerrit UX visible and responsive3. Git/HTTP and Git/SSH accessible4. Most of the functions operational5. All functions fully operational
@gitenterprise @gerritreview #GerritUserSummit
Levels of Zero-Downtime: where are we?
0. Outage1. Server always-on and reachable2. Gerrit UX visible and responsive3. Git/HTTP and Git/SSH accessible4. Most of the functions operational5. All functions fully operational
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 1
Client traffic
Ver. X (empty)RSync
DB Export / Import
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 2
Client traffic
Ver. X Ver. XGerrit replication
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 3 upgrade
Client traffic
Ver. X Ver. X + N
Upgrade Gerrit war to Ver. X+NInit + Off-line reindex
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 4 – read-only
Client traffic
Ver. X Ver. X + N
Install Read-only plugin
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 5 – delta replication
Client traffic
Ver. X Ver. X + N
Install Read-only plugin
Delta replication
DB Export / Re-Import
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 6 – Reinit
Client traffic
Ver. X Ver. X + N
READ-ONLY Init + Startup
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 7 – delta reindex
Client traffic
Ver. X Ver. X + N
Delta reindexREAD-ONLY
@gitenterprise @gerritreview #GerritUserSummit
HA Proxy
Zero-Downtime Upgrade – Stage 8 – traffic redirect
Client traffic
Ver. X Ver. X + N
Cache warm-up
@gitenterprise @gerritreview #GerritUserSummit
Timing and service level
1Stages 2 3 4 5 6 7
Times: days hours 10s 5' 5' 1s
ServiceLevel 100% up on Ver. X Read-only Ver. X 100% up on Ver. X
+ N
8
@gitenterprise @gerritreview #GerritUserSummit
DEMO
@gitenterprise @gerritreview #GerritUserSummit
Wants to know more?
www.gerritforge.com/contact