Learning by Experience, Deploying pyxbackup

Jervin RealPyCon.PH 2015

pyxbackup

https://github.com/dotmanila/pyxbackup

There was a need

• to manage backups • for non-dbas • portable (GNU/Linux) • set and forget (kind of)

.. then a solution

• xbackup.sh • bash shell script • yes, portable (GNU/Linux) • yes, for non-dbas • yes, it can manage backups • yes, set and forget (kind of)

But!

• becoming difficult to maintain • .. even debug at times • not very flexible i.e. strings

Python!

How do we begin?

How do we begin

• understood, but have not written Python code

• Opportunity! • Contributing to open source • docs.python.org • StackOverflow, really, really :-)

We all set then!

Debugging, containing errors

• catching Exceptions, or throwing one

Debugging, containing errors

• logging and emailing

Debugging, containing errors

• --debug

Options, parameters, config files

• OptionParser • ConfigParser - also used for metadata

Interacting with the environment

• subprocess.Popen

Interacting with the environment

• os.environ, et. al

http://stackoverflow.com/questions/842059/is-there-a-portable-way-to-get-the-current-username-in-python

Interacting with the environment

• lock/pid files

Interacting with the environment

• signal handling

Interacting with the environment

• logging

Interacting with the environment

• exit codes

Interacting with the environment

• subprocess.Popen • os.environ • lock/pid files • signal handling • logging • exit codes

Aiming for portability

• standard modules, except MySQLdb • MySQLdb instead of Connector/Python • limited external dependencies

pyxbackup

https://github.com/dotmanila/pyxbackup

TODOs

• switch to logging module • discard shell=True • using classes (BackupProcess, Storage, Config) • improve documentation • use Multiprocessing async work • unit tests (aside from functional tests) • daemonize from python (binlog-stream) • port to Python 3

Summary

• portable - standard modules, predictable behaviour through signals and exit codes

• easy to use - useful errors and logs, debuggable

• maintainability - in progress :)

.. and yours truly

• Senior MySQL Consultant at Percona

•@dotmanila

• https://github.com/dotmanila

• http://dotmanila.com/blog/

• http://www.percona.com/blog/author/jervin

Questions