LHCbPR V2

Sasha Mazurov, Amine Ben Hammou, Ben Couturier

5th LHCb Computing Workshop2015-05-19

5th LHCb Computing Workshop 2

Content

• Objectives– LHCbPR V1 Experience– Why V2?

• New architecture and tools• RESTful service• Web frontend

2015-05-19

5th LHCb Computing Workshop 3

LHCbPR

• “LHCb Performance and Regression(PR) is a service designed to record important measures about releases of the version of the LHCb software”

• Current version (V1): https://lhcb-pr.web.cern.ch/lhcb-pr/

2015-05-19

5th LHCb Computing Workshop 4

LHCbPR V1 experience (1) Web frontend

Main form for jobs searching:lack of dynamic filtering for options, platforms, ...

2015-05-19

5th LHCb Computing Workshop 5

LHCbPR V1 experience (2) Admin backend

Automatic admin site scaffolding is not usable for creating a new job description – too many unobvious steps. A wizard for such task is neede to be written

2015-05-19

5th LHCb Computing Workshop 6

LHCbPR V1 experience (3) New handlers and analysis development

• Lack of python and javascript libraries for common LHCbPR programming tasks. A lot of copy-paste code.

• Unreadable raw sql in the source code.

To get a readable code weneed to use standard Djangodatabase model classes

2015-05-19

5th LHCb Computing Workshop 7

Why V2?• First version of LHCbPR is a good program for

understanding the real needs for regression analysis. • We have collected a list of features that hard to

implement in V1 (thresholds, attributes management, separate versions and applications,…).

• We need to have a common javascript libraries and components specific for LHCbPR that can be shared among developers (now we have a spaghetti code)

• Decouple data access application and presentation sites.• Avoid using raw SQL.

2015-05-19

5th LHCb Computing Workshop 8

New database entities and changes• Versions table – separate application and its version entities. Has

timestamp field that is inserted from its first job data, so we can arrange releases’ and nightly builds’ versions.

• Nightly slots table – store name of nightly slot (“lhcb-head”, “lhcb-gaudi-head”,…). Versions table can have a link to this table.

• Thresholds table for attributes – we can check attributes ranges• Attributes’ group table – attribute can belong to several groups

We have a tool that import data from V1 database.

2015-05-19

5th LHCb Computing Workshop

New architecture

• LHCbPR as a RESTful data service:– Separate data access and data representation.

Interoperate using JSON format– Web site is only one of the clients of the service– Easy to write command line scripts (for nightlies,

reports,…)

92015-05-19

5th LHCb Computing Workshop

Tools

• RESTful service– Django 1.8 with rest framework application.

• Web site– AngularJS JavaScript MVC framework– Bootstrap based CSS framework– No backend

102015-05-19

5th LHCb Computing Workshop

REST API

11

http://amazurov.web.cern.ch/amazurov/lhcbpr-api - outputs all possible service endpoints

2015-05-19

5th LHCb Computing Workshop 12

REST API features

• We can use it for all CRUD operations (create, read, update, delete)

• Can output arbitrary JSON (examples later)• Authentication plugins: e.g. shibboleth for

web and tokens for console applications• Output pagination, throttling and requests’

cache allow to avoid service overloading.

2015-05-19

5th LHCb Computing Workshop 13

Examples

• Jobs list - /jobs• Job details - /jobs/178– We can expand any object’s relationships.

• Nested requests - /jobs/178/results/– Output job attributes’ values

• Output filtering - /compare/?ids=32,33&contains=Event– Show and group attributes of jobs with id 32 and

32. Attribute name should contain “Event” string2015-05-19

Web Frontend

5th LHCb Computing Workshop 14

Only http requests from browser to REST server, no backend behind

http://amazurov.web.cern.ch/amazurov/lhcbpr-dev - DEMO

2015-05-19

5th LHCb Computing Workshop 15

Reusable Web Components= <div search-jobs="search-jobs" on-found="onJobsFound(searchParams)”></div>

onJobsFound – client’s custom callback function thatget search form results: ids of application, options,Versions

More components in development

2015-05-19

5th LHCb Computing Workshop

Command line / Scripts

16

Can be easily parsed with standard python libraries

$> curl http://amazurov.web.cern.ch/amazurov/lhcbpr-api/jobs/

2015-05-19

5th LHCb Computing Workshop 17

For Developers

• We have prepared a set of reusable JavaScript components that simplifies work with REST service and introduce common UI elements for LHCbPR

• Well organized modules’ directory structure

2015-05-19

5th LHCb Computing Workshop 18

Modules• Controllers are written in

JavaScript• Views use Jedi template

engine• Modules support lazy

JavaScript loading, so only required libraries are used in modules.

2015-05-19

5th LHCb Computing Workshop 19

Summary• API

– Updated database schema for proper support of application versions, build slots, thresholds, attribute groups. Import script from V1 database

– Ready for production RESTful service that can be used from any application (currently in readonly mode)

– https://github.com/LHCbDev/lhcbpr-api• Frontend

– Fixed a structure of modules in web frontend– Prepared a set of web components and services.– Compare jobs’ results module. Framework is ready for

developing (migration) of the rest of analysis modules.– https://github.com/LHCbDev/lhcbpr-www

2015-05-19