Implementing Rolling Upgrades
Containerized Environment powered by Mesos, Marathon, Docker and HAProxy
1. Deploy NEW application (with a new App ID) with the minimum set of instances (eg: 1 in the beginning)
as needed to manage the traffic
2. Wait for the NEW instance(s) to go healthy
3. Add the NEW Instance(s) to the HAProxy Pool
4. Load gets balanced between OLD and NEW Pool
5. DISABLE one of the OLD instances in the HAProxy Pool
6. Wait for the HAProxy Connection to drain on the Disabled OLD Instance
7. After waiting period is over, Remove the Disabled OLD Instance by issuing Docker STOP on the
Container
8. Scale the NEW Instances and let them add to the HAProxy Pool
9. Repeat the steps 4 - 8 till all OLD instances are removed, and all the NEW instances are added to
HAProxy
10. Delete the OLD Application
ALGORITHM: Rolling Upgrades
OLD OLD OLD OLD OLD OLD
HAProxy
CURRENT STATUSOld Instances are running and load balanced by HAProxy
OLD OLD OLD OLD OLD OLD
HAProxy
L4
DNS
Customer
Service Port 1
CURRENT STATUSOld Instances are running and load balanced by HAProxy
NEWOLD OLD OLD OLD OLD OLD
HAProxy
Minimum required to serve just
enough traffic
NEW INSTANCE DEPLOYED1 Instance of the New Version of Application is deployed
NEWOLD OLD OLD OLD OLD OLD
HAProxy
NEW INSTANCE DEPLOYEDHealth Check Passed. Added to Load Balancer.One Old Instance is DISABLED from Load Balancer
OLD OLD OLD OLD OLD OLD
HAProxy
L4
DNS
Customer
Service Port 1
NEW
NEW INSTANCE DEPLOYEDAdded to Load Balancer. 16% new requests goes to NEW Instance. No New Request will go to Old Instance #1
disabled
NEWOLD OLD OLD OLD OLD OLD
HAProxy
ZDD
Control
Marathon
Mesos
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDZDD requests HAProxy to DISABLE one of the OLD Instance
ZDD is a Python Script part of the Marathon-lb Project from Mesosphere
disabled
NEWOLD OLD OLD OLD OLD OLD
HAProxy
ZDD
Control
Marathon
Mesos
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDZDD now waits for N seconds to drain any existing connections
N seconds
NEWOLD OLD OLD OLD OLD OLD
HAProxyMesos
STOP
ZDD
Control
Marathon
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDAfter N seconds, ZDD requests Marathon to STOP one of the Old Instances
NEWOLD OLD OLD OLD OLD OLD
HAProxyMesos
STOP
M seconds
ZDD
Control
Marathon
Control
HOW IS ROLLING UPGRADE IMPLEMENTEDMesos invokes Docker STOP on the Old Container, and waits for M seconds
NEW NEW
HAProxy
NEWNEWNEWNEW
L4
DNS
Customer
HOW IS ROLLING UPGRADE IMPLEMENTEDNow all NEW Instances have been added to HAProxy Pool