WSO2 Elastic Load Balancer

Chintana Wilamuna chintana@wso2.com

Load balancer

• Distribute load among a set of nodes • Static (fixed set of nodes)

• Dynamic (add/remove nodes at runtime)

• Load balancing algorithm (Round robin / weighted RR / weighted RR least connection)

Session affinity / sticky sessions

• All subsequent connections within a session is served by the worker node which processed the first request of the session

• Alternate solution to session replicated backend services

Session replication in AppServer

• AppServer clustering – Apache Tribes • Session data storage,

1. ServiceContext (data that should only be available to the service)

2. ServiceGroupContext (common data for all the services in a service group)

3. ConfigurationContext (common data for all service groups)

• AppServer replicates session data in all 3 levels (costly compared to sticky sessions)

LB – Membership mgt. schemes

• Static • Dynamic Join by specifying the group name

• Hybrid (WKA) Set of well known members – static Join by specifying the group name

Evolution of WSO2 ELB

• Started as a solution to scale PaaS – (WSO2 Stratos)

• Stratos is a multi-tenanted PaaS • Tenant – An organization (a domain)

• Isolated from other tenants • Support multiple PaaS services • A service = WSO2 product (DSS, ESB

etc…) offered as a service

Evolution of WSO2 ELB cont.

• A tenant can deploy their own services, mediations, workflows etc…

• Multi-tenant services (ESB, DS, AppServer etc…)

• Stratos can provision resources on demand

• Problem: making a scalable PaaS transparent to the user

Solution 1

• Stratos have multi-tenant services (ESB, AppServer, DS etc…)

• User talks to Identity Server, gets an SAML token, logs into any service

• Store tenant data in Registry • Each server loads all tenant data at

startup • Can support any tenant

Solution 1

Solution 1 - Problems

• Doesn’t scale with increasing number of tenants

Solution 2

• Started running multiple service instances

• Put a LB in front • When the load is high LB will spin up

a new instance and shuts it off when the load decreases (autoscale)

Solution 2

Solution 2 - Problems

• When we had a large number of tenants with tens of services, took a long time to load at startup

• Some tenants stayed inactive (consumed resources)

Solution 3

• Add lazy loading • Tenant info in the central registry • Tenant info loaded only when needed • More info - http://blog.afkham.org/2011/11/lazy-loading-

deployment-artifacts-in.html

Solution 3 - Problems

• When a tenant has many artifacts loading those takes time

• Initial requests for an artifact of a tenant timed out

Solution 4

• Introduce the GhostDeployer • Loads only the metadata of a tenant • Actual artifacts are loaded on demand • More info - http://blog.afkham.org/2011/11/lazy-loading-

deployment-artifacts-in.html

Solution 4 - Problems

• LB didn’t scale to handle large number of requests

Solution 5

• Replace LB with multiple LBs • HW LB or DNS RR between LBs

Solution 5

Solution 5 - Problems

• Many to many relationship between LBs for synchronizing metadata

• LB is not aware of tenants • A single node might have too many

tenants

Solution 6

• Tenant aware load balancer

WSO2 ELB architecture

WSO2 ELB

• Autoscaling component keeps track of traffic for each cloud service cluster

• Calls EC2 API • Implementation: 2 mediators and a

task (autoscale-in and autoscale-out)

• An ESB endpoint routes requests to the correct node

WSO2 ELB

• Service cluster identification through Host HTTP header

• According to LB algo. select a node • If the node is down it’ll fail over to

another member • Keeps track of sticky sessions (identified by JSESSIONID)

WSO2 ELB

• BinaryRelay is used as a message pass through mechanism

(no building or processing the message)

• Membership discovery & mgt. – Axis2 clustering. Membership handler for each clustering domain

• Use Apache Tribes Group Mgt. Framework

WSO2 ELB

WSO2 ELB – Port mapping

WSO2 ELB – Port mapping

• Advertise the mapped ports when joining the cluster

• LB retrieves these as properties of the member

• When dispatching requests, get mapped real port from the member (as advertised)

WSO2 ELB – Autoscaling

WSO2 ELB – Autoscaling

• AutoscaleInMediator creates a unique token when a request comes, put into a list

• Removes this when a response is sent • LoadAnalyzerTaks periodically checks

the list. Autoscale according to config params

WSO2 ELB – Autoscaling

• Autoscaling algorithm - http://nirmalfdo.blogspot.com/2012/07/autoscaling-algorithm-used-in-wso2.html

WSO2 ELB – Configuration

• How to configure the ELB - http://nirmalfdo.blogspot.com/2012/06/fronting-wso2-application-server-50.html

Questions?