europython 2014

1 / 26

How we switched our800+ projects fromApache to uWSGI

Max Tepkeev23 July 2014Berlin, Germany

europython 2014

2 / 26

Who We ?!

• Ailove Group• Ailove• Aitarget• iCom

• 120+ people• 800+ projects

europython 2014

3 / 26

Who I ?!

Max TepkeevRussia, Moscow

• python-redmine• architect

https://www.github.com/maxtepkeev

europython 2014

4 / 26

Previous Technology Stack

• PHP• 5.3• 5.4• 5.5

• Python• 2.7• 3.3

• nginx• Apache• mod_php• mod_wsgi

europython 2014

5 / 26

Problems

Separate Apache for each version of PHP and Python listening on different ports:

• :8080 – Apache/PHP5.3• :8081 – Apache/PHP5.4• :8082 – Apache/PHP5.5• :8083 – Apache/Python2.7• :8084 – Apache/Python3.3

europython 2014

6 / 26

Problems

Monitoring for code changes during development:

• embedded-mode – full restart• daemon-mode – 100+ lines script

mod_wsgi >= 4.1.0 (Django only):

• runmodwsgi --reload-on-changes

europython 2014

7 / 26

Problems

mod_wsgi WSGIDaemonProcess name can’t be dynamic, e.g. separate daemon process per GIT branch:

• dev.project.com – project-dev.conf• dev2.project.com – project-dev2.conf

europython 2014

8 / 26

Problems

Apache can’t load configuration files dynamically:

• reload – not so graceful restart• total crash if there are errors

europython 2014

9 / 26

Problems

• Apache configuration files are ugly (subjective)

• Apache is hard to configure properly (subjective)

• Apache is old (2.4 fixed a lot)• mod_wsgi seemed to be abandoned

(is actively developed again)

europython 2014

10 / 26

uWSGI

• Modern project• Fast development cycle• Multi-language (Python, PHP, Lua,

PERL, Ruby, Erlang, Go, Java and more)• Supports > 20 OS• nginx speaks uwsgi protocol• Ton more features

europython 2014

11 / 26

Installationhttp://projects.unbit.it/downloads/uwsgi-latest.tar.gz

To change install location add to buildconf/core.ini:

bin_name = /usr/local/uwsgi/bin/uwsgiplugin_dir = /usr/local/uwsgi/plugins

mkdir /usr/local/uwsgimkdir /usr/local/uwsgi/binmkdir /usr/local/uwsgi/plugins

europython 2014

12 / 26

InstallationCore:

python uwsgiconfig.py --build core

Plugins:

python2.7 uwsgiconfig.py --plugin plugins/python core python27python3.3 uwsgiconfig.py --plugin plugins/python core python33

UWSGICONFIG_PHPDIR=/usr/local/php5.4 python uwsgiconfig.py --plugin plugins/php core php54UWSGICONFIG_PHPDIR=/usr/local/php5.5 python uwsgiconfig.py --plugin plugins/php core php55

europython 2014

13 / 26

Solutions

Multi-version plugins:

• php53_plugin.so – PHP5.3• php54_plugin.so – PHP5.4• php55_plugin.so – PHP5.5• python27_plugin.so – Python2.7• python33_plugin.so – Python3.3

europython 2014

14 / 26

Solutions

Monitoring for code changes during development:

• --py-auto-reload N (seconds)• --touch-reload=django.wsgi

europython 2014

15 / 26

Solutions

Emperor mode - event based dynamic handling of applications (vassals):

• Monitor config files in directories• More plugins available (postgres,

mongodb, amqp, zeromq and more)

europython 2014

16 / 26

Solutions

Monitor config files in directories:uwsgi --emperor “/usr/local/uwsgi/apps/*/*.ini”

• New file “/usr/local/uwsgi/apps/app/app.ini”• Spawn vassal

• File modified• Restart vassal

• File removed• Kill vassal

europython 2014

17 / 26

Solutions

Template config files:

ln -s /usr/local/uwsgi/templates/app1_tpl/usr/local/uwsgi/apps/app1/main.ini

europython 2014

18 / 26

Solutions[uwsgi]project_dir = /srv/projects/my_project

plugins = python27virtualenv = %(project_dir)/pythonpythonpath = %(project_dir)socket = %(project_dir)/tmp/%n.sockwsgi-file = %(project_dir)/repo/%n/wsgi/django.wsgitouch-reload = %(project_dir)/repo/%n/wsgi/django.wsgilogto = %(project_dir)/logs/uwsgi-%n.log

europython 2014

19 / 26

Solutions

Create symlink for each GIT branch:

• ln -s /usr/local/uwsgi/templates/app1_template/usr/local/uwsgi/apps/app1/dev.ini

• ln -s /usr/local/uwsgi/templates/app1_template/usr/local/uwsgi/apps/app1/master.ini

europython 2014

20 / 26

Cool Features

Auto-scaling:

• broodlord mode• zerg mode• emperor• idle/die-on-idle

europython 2014

21 / 26

Cool Features

Alarm subsystem:In configuration[uwsgi]alarm = jabber xmpp:foo@jabber.xxx;mypass;admin@jabber.xxxlog-alarm = jabber ^TERRIBLE ALARM

In applicationprint "TERRIBLE ALARM!"

europython 2014

22 / 26

Cool Features

Aliasing Python modules:

[uwsgi]# some configuration herepymodule-alias = foo=/opt/proj/experimental_foo.pypymodule-alias = bar=/opt/proj/experimental_bar.py

europython 2014

23 / 26

Cool Features

• built-in crontab• built-in load-balancer• clustering subsystem• offload subsystem• plugins (geoip etc.)• integration with web-servers• django admin integration

europython 2014

24 / 26

How to switch

• no fully automatic way• write scripts that generate basic

config files for you• tune them by hand afterwards

europython 2014

25 / 26

Conclusion

• apache is not bad• benchmarks are pointless• web servers perform similarly if

configured properly• it’s applications that have

problems, not web servers• choose a right tool for a right job

europython 2014

26 / 26

Questions

slides: http://slideshare.net/maxtepkeev

github: https://github.com/maxtepkeevemail: tepkeev@gmail.comskype: max.tepkeev