5 years of Daiquiri · needs to be maintained along MySQL/MariaDB versions now: Celery and RabbitMQ...

transcript

5 years of DaiquiriLessons learned and further developments

Jochen Klar

Leibniz-Institute for Astrophysics Potsdam (AIP)

jklar@aip.de / @jochenklar

1 / 15

DaiquiriA framework for the publication of scientific databases

Allows for highly customizable web applications

Based on a common easily maintainable code base

Separated into an app and the daiquiri library

Features:

SQL web interface to relational databases

User management and user registration work flow

Metadata management and access control

First impression in-browser plotting

Table download and file service

Employs VO protocols and standards

2 / 15

Public databases in astronomySloan Digital Sky Survey (SDSS) and Millennium simulation

Image credit: David Kirkby

3 / 15

Databases hosted @ AIPRAVE database query interface for DR3 (Siebert et al., 2011)

MultiDark Database (Riebe et al., 2011)

RAVE (2013), CosmoSim (2014), APPLAUSE (2015), Gaia@AIP (2016)

4 / 15

Daiquiri v1aka legacy-daiquiri (2012 - 2016)

Written in PHP using the Zend framework

Front-end first written using jQuery, later AngularJS 1 and Bootstrap 2

Using:

MySQL 5.5 as database server

PHP-SQL-Parser to parse queries

MySQL Query Job Queue for asynchronous jobs

paqu for parallel access to sharded databases (Cosmology, Gaia)

mysqldump-vo to create download files

Gearman to create download files asynchronous

5 / 15

Language and Frameworkthen: PHP and the Zend framework version 1.12

PHP: fast to get into, easy to deploy, very popular, but [insert rant here]!

Zend: insufficient ORM, no templating beyond PHP, security pitfalls, noecosystem

6 / 15

Language and Frameworkthen: PHP and the Zend framework version 1.12

PHP: fast to get into, easy to deploy, very popular, but [insert rant here]!

Zend: insufficient ORM, no templating beyond PHP, security pitfalls, noecosystem

now: Python and Django

Python: todays preferred scripting language, widely used in astronomy

Django: full MVC framework with everything included, huge community

Django REST framework: de-facto standard for REST interfaces in Django

django-allauth: local and social authentication, registration work flows

astropy: community python library for Astronomy

6 / 15

Front-endthen: Angular 1 and Bootstrap 2

AngularJS 1: JavaScript front-end framework, templates, 2-way binding

Bootstrap 2: CSS and JS for layout, forms, modals, buttons, navbars, ...

7 / 15

now: Angular 1 and Bootstrap 3

AngularJS 1: still awesome

Bootstrap 3: responsive layout, mobile friendly

7 / 15

now: Angular 1 and Bootstrap 3

AngularJS 1: still awesome

Bootstrap 3: responsive layout, mobile friendly

Front-end web development is evolving fast

While the number of ways to organize JavaScript is almost infinite, here are some toolsthat help you build single-page applications.

https://github.com/showcases/front-end-javascript-frameworks

7 / 15

Understanding queriesthen: PHP-SQL-Parser

a lot of custom code needed to get the necessary information from the query

no support for ADQL, therefor no support for TAP

8 / 15

Understanding queriesthen: PHP-SQL-Parser

a lot of custom code needed to get the necessary information from the query

no support for ADQL, therefor no support for TAP

now: queryparser based on Antlr

Antlr: parser generator for structured text or binary files

queryparser: Antlr generated python code to parse query strings

ADQL translator using ADQL grammar to translate to MySQL syntax

MySQL parser using MySQL grammar for parse a MySQL query

open source and available on GitHub and PyPI (python2, python3)

Using mysql_sphere to translate ADQL functions into MySQL

developed and maintained by Gal Matijevic (AIP)

8 / 15

Asynchronous Jobsthen: MySQL Query Job Queue

MySQL plugin implementing a job queue inside the database

using UDF like qqueue_addJob() and tables like mysql.qqueue_jobs

needs to be maintained along MySQL/MariaDB versions

9 / 15

Asynchronous Jobsthen: MySQL Query Job Queue

MySQL plugin implementing a job queue inside the database

using UDF like qqueue_addJob() and tables like mysql.qqueue_jobs

needs to be maintained along MySQL/MariaDB versions

now: Celery and RabbitMQ

Celery: asynchronous task queue queue in Python, widely adopted

RabbitMQ: message broker in Erlang

redis: in-memory data structure store for task results, can also be used forcaching

systemd: new init system for Linux, make it easy to deploy daemons

9 / 15

Downloading tablesthen: Gearman and mysqldump-vo

Gearman: to create download files asynchronous

mysqldump-vo: to create custom download formats:

MySQL dumps

VOTable, VOTable Binary, VOTable Binary2

10 / 15

Downloading tablesthen: Gearman and mysqldump-vo

Gearman: to create download files asynchronous

mysqldump-vo: to create custom download formats:

MySQL dumps

VOTable, VOTable Binary, VOTable Binary2

now: Celery and RabbitMQ and old-school unix pipes

mysqldump database_name table_name | some_magic > table_name.csv

10 / 15

Daiquiri v2aka django-daiquiri (since 2016)

Python (>= 2.7 and >= 3.4) using the Django framework (1.11)

Responsive front-end written in AngularJS 1 and Bootstrap 3

Using:

MariaDB 10.1

queryparser and Antlr to parse and translate queries

Celery, RabbitMQ, redis and systemd for asynchronous tasks (not only queries)

old-school unix pipes to create files using mysqldump

open source and available on GitHub and PyPI

11 / 15

Featuresimplemented

SQL query interface (with examples, job list, plotting, ...)

Customizable data query forms

Contact messages to the support staff

DOI integration and landing pages for databases and tables

File service and zip-archive creation

Registration and log-in using Oauth2 (facebook, twitter, GitHub, Google, ORCID)

WordPress as CMS for project presentation and documentation

VO protocols:

Data Access Layer Interface (DALI)

Table Access Protocol (TAP)

Universal Worker Service Pattern (UWS)

12 / 15

Featuresupcoming

File access, filtering and download for observatory archives

Cut-out service for images and datacubes

FITS tables download

Use of sharded databases (paqu v2)

Management of collaboration meetings

VO protocols:

Simple Image Access (SIA)

Simple Spectral Access (SSA)

Provenance Data Model

13 / 15

Try django-daiquiri!as a user

Development version of the Gaia@AIP Services: https://gaia.aip.de/dev

as a provider

README: https://github.com/aipescience/django-daiquiri/blob/master/README.rst

git clone https://github.com/aipescience/django-daiquiri-app appcd app; python3 -m venv env; source env/bin/activatepip install django-daiquiri mysqlclientcp config/settings/sample.local.py config/settings/local.pymkdir log download

./manage.py sqlcreate # shows commands for MariaDB

./manage.py migrate # creates database and tables

./manage.py migrate --database=tap # creates TAP_SCHEMA

./manage.py createsuperuser # creates admin user

./manage.py runserver # runs a development server

14 / 15

Thanks!

Jochen Klar

jklar@aip.de / @jochenklar

github.com/aipescience

escience.aip.de

15 / 15

5 years of Daiquiri · needs to be maintained along MySQL/MariaDB versions now: Celery and RabbitMQ...

Documents