Inside Sqale’s BackendRubyConf Taiwan 2012

Gosuke Miyashitapaperboy&co., Inc.， Japan

cpan:mizzygithub.com/mizzy

mizzy.org@gosukenator

Kentaro KuribayashiLiving on the Edge Rails

Kensuke NagaeIntroducing nonopaste-cli

Shinya TsunematsuBuilding production server

environment of ruby in modern way

Hiroshi SHIBATAHow to discover the Ruby's defects

with web application

Gosuke MiyashitaInside Sqale’s Backend

We are from paperboy&co., Japan

What is Sqale?

Cloud Application Platform like Heroku

Architecture Overview

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

Containers

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

Virtual Environments Assigned To Users

Similar to Dynos of Heroku

Containers made by LXC (Linux Containers)

EC2 Instance (1 Virtual Machine)

Container for

user B

Container for

user A

Container for

user A

Container for

user B

Container for

user B

Container for

user D

Container for

user D

Container for

user C

Container for

user E

Container for

user E

Container for

user F

Container for

user F

Container for

user E

Container for

user F

Container for

user F

NginxUnicorn

sshdsupervisrod

on each container

Amazon Linux+

Patched kernel(3.2.16)

grsecurity kernel patchfor various restrictions

original kernel patchesto restrict tcp port

bind and fork bomb

Anti fork bomb patch makes some changes to cgroup and fork process

Seepaperboy-sqale/sqale-patches

on GitHub

Web Proxy

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

nginx

Container for

user A

Container for

user B

Container for

user B

Container for

user C

Container for

user C

Container for

user C

ELB

nginx

HTTP/HTTPS

nginxlua-nginx-module

redis2-nginx-module

Container for

lokka-mizzy

Container for

lokka-mizzy

Container for

i4pc-mizzy

Container for

i4pc-mizzy

nginx

http://www.i4pc.jp/

Redis

nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084

Which containers?

host001:8083, host001:8084

host001

or

location / { set $container "";    set $next_containers "";

error_page 502 = @failover;

    rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

nginx.conf (excerpt)

local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)end

local containers, type = parser.parse_reply(reply.body)

dynamic-proxy.lua (excerpt)

while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

dynamic-proxy.lua (excerpt)

ngx.var.container = containerngx.var.next_containers = luabins.save(containers)

dynamic-proxy.lua (excerpt)

location / { set $container "";    set $next_containers "";

error_page 502 = @failover;

    rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

nginx.conf (again)

location @failover { error_page 502 = @failover;

    rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

nginx.conf (excerpt)

failover.lua (excerpt)

local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end

failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

if not container then ngx.exit(503)end

ngx.var.container = containerngx.var.next_containers = luabins.save(containers)

failover.lua (excerpt)

location @failover { error_page 502 = @failover;

    rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

nginx.conf (agin)

SSH Router

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

SSH Router

File Repositories(Git Server)

Git SSH Login

File Repositories(File Server)

Containers

SFTP

How implement this routing?

OpenSSH with script authentication patch

Seemizzy/openssh-script-auth

on GitHub

Change routes by SSH_ORIGNAL_COMMAND

In case of SSH_ORIGINAL_COMMAND

is “git-*”

SSH Router

File Repository(Git Server)

git push(ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’)

MySQL

Run AuthorizedKeys Script

Verify the public keyand get the user’s git server

command=“ssh sqale@git001.sqale.lan git-recieve-pack ‘/var/repos/mizzy/lokka.git’”

In case of SSH_ORIGINAL_COMMAND

is “sftp-server”

SSH Router

File Repository

(File Server)

sftp sqale@gateway.sqale.jp(ssh sqale@gateway.sqale.jp sftp-server)

MySQL

File Repository(Git Server)

git push

Run AuthorizedKeys Script

Verify the public keyand get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server”

In case of SSH_ORIGINAL_COMMAND

is empty

SSH Router

Container

ssh sqale@gateway.sqale.jp

MySQL

Run AuthorizedKeys Script

Verify the public keyand get the user’s cotainers list

command=“ssh sqale@ users001.sqale.lan -p 8081”

Display the user’s containers list and wait the user’s selection

Ruby On RailsPuppet/Chef

UnicornResque

And many gems

多謝