WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers .

IT-SDC : Support for Distributed Computing

WLCG Transfers Dashboard A unified monitoring tool for

heterogeneous data transfers.

Alexandre Beche<[email protected]>

2IT-SDC

Outlines Mission History of the data transfers monitoring

tools Generic monitoring workflow New WLCG Transfers Dashboard Summary

18 December 2013Alexandre Beche – White Area

3IT-SDC

Mission

ATLASFTS + XRootD

CMSFTS + XRootD

LHCbFTS

ALICEXRootD


4IT-SDC

ATLASFTS + XRootD

CMSFTS + XRootD

LHCbFTS

ALICEXRootD

Mission


5IT-SDC

ATLASFTS + XRootD

CMSFTS + XRootD

LHCbFTS

ALICEXRootD

Mission


6IT-SDC

History of data transfers monitoring tools

WEBAPI / UI

WLCG

FTS

WLCGTransfers

Dashboard


7IT-SDC


WEBAPI / UI

WLCG

FAX

AAA

FTS

WLCGTransfers

Dashboard

XRootDintegration


8IT-SDC


WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

WLCG FAX AAA

FAX

AAA

FTS

WLCGTransfers

Dashboard

XRootDintegration

XRootDDetailled

Dashboard


9IT-SDC


WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

WLCG FAX AAA

FAX

EOS

AAA

EOS

FTS

WLCGTransfers

Dashboard

XRootDintegration

XRootDDetailled

Dashboard

EOSintegration


10IT-SDC


WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

WLCG FAX AAA

FAX

EOS

AAA

EOS

FTS

ALIC

EWLCG

TransfersDashboard

XRootDintegration

XRootDDetailled

Dashboard

EOSintegration

ALICEintegration


11IT-SDC

Generic monitoring workflow

AMQAMQ

Raw&

Statistics

Consumer WEB API WEB UI


12IT-SDC

From ActiveMQ to Oracle Load balanced collector

Horizontal scaling

Consumer

AMQ DB

Consumer


13IT-SDC


Horizontal scaling

Consumer

AMQ DB

Stompclt*

* Developed by the CERN messaging team


14IT-SDC


Horizontal scaling

Consumer

AMQ DB

Stompclt*

Disk queue*



15IT-SDC


Horizontal scaling

Consumer

AMQ DB

Stompclt*

Disk queue*

DB inserter Customizable

inserter Data enhancement Filtering



16IT-SDC

From ActiveMQ to Oracle

Modular architecture Common building blocks (EPEL) Reliable

Load balanced collector Horizontal scaling

Consumer

AMQ

Simplevisor*

DB

Stompclt*

Disk queue*

DB inserter Customizable

inserter Data enhancement Filtering



17IT-SDC

Raw data aggregation Well optimized storage

Partitioned table No global indexes (including the PK)


18IT-SDC

Raw data aggregation Well optimized storage

Partitioned table No global indexes (including the PK)

Aggregation of unordered events Stateless: Full re-computation of touched

bins each time Compute stats from raw data in 10 min bins Aggregate 10 min stats in daily bins


IT-SDC

Aggregation methods

19

2pm 3pm 4pm 5pm 6pm 7pmTran

sfer

s


IT-SDC

Aggregation methods

20


sfer

s

Easy method


IT-SDC

Aggregation methods

21


sfer

s

Easy method

Transfers 1 0 0 2 1Bytes 10 0 0 15 20


IT-SDC

Aggregation methods

22

2pm 3pm 4pm 5pm 6pm 7pm

Transfers 1 0 0 2 1Bytes 10 0 0 15 20

Transfers 1 (1) 1 (0) 2 (0) 3 (2) 1 (1)

Bytes 8 1 14 (9+6) 15 (1+9+5) 5

Easy method

Tran

sfer

s


IT-SDC

Aggregation methods

23

2pm 3pm 4pm 5pm 6pm 7pm

Transfers 1 0 0 2 1Bytes 10 0 0 15 20

Transfers 1 (1) 1 (0) 2 (0) 3 (2) 1 (1)

Bytes 8 1 14 (9+6) 15 (1+9+5) 5

Easy method

Tran

sfer

s

Adopted method

• Both method equivalent if:• Small transfers or many transfers


24IT-SDC

Raw&

Statistics

DAO

A well defined web API Legacy dashboard framework:

Use the MVC design pattern (based on Struts)

Application specific

Collection of SQL queries

Generic


25IT-SDC

Raw&

Statistics

DAO Action





Generic


Run a parameterized query,Return the results in a dict

26IT-SDC

Raw&

Statistics

DAO Action

ViewsJson, xml, …





Python dict to #format# converter


Generic


27IT-SDC

Raw&

Statistics

DAO Action

ViewsJson, xml, …

Cont

rolle

r





Router of url+mimeTo action + view



Generic


28IT-SDC

Raw&

Statistics

DAO Action

ViewsJson, xml, …

Cont

rolle

r

Configuration







List of actions


Generic


29IT-SDC

Raw&

Statistics

DAO Action

ViewsJson, xml, …

Cont

rolle

r

Configuration







List of actions


Generic


Django under investigation

30IT-SDC

Visualization layerDashboard UI

Rich web single-page user interface AJAX+JSON communication


31IT-SDC


Rich web single-page user interface AJAX+JSON communication jQuery + popular plugin technology cocktail


32IT-SDC

xbrowse


Rich web single-page user interface AJAX+JSON communication jQuery + popular plugin technology cocktail Modular code reuse with xbrowse framework

Lightweight in-house URL-driven MVC (<300 LOC) Views shared with several other applications Common look and feel


33IT-SDC


WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

WLCG FAX AAA

FAX

EOS

AAA

EOS

FTS

ALIC

EWLCG

TransfersDashboard

XRootDintegration

XRootDDetailled

Dashboard

EOSintegration

ALICEintegration


34IT-SDC

WLCG Transfers Dashboardfederated approach

WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

FTS FAX AAA

FAX

EOS

AAA

EOS

FTS

ALIC

E

WEBAPI

ALICE No duplication anymore

EOS could be included


35IT-SDC

WLCG Transfers Dashboardfederated approach

WEBAPI / UI

WEBAPI/UI

WEBAPI/UI

FTS FAX AAA

FAX

EOS

AAA

EOS

FTS

ALIC

E

WEBAPI

ALICE

WLCG Transfers DashboardAPI / UI

No duplication anymore

EOS could be included


Alexandre Beche 36IT-SDC

Some plots

27 May 2013

37IT-SDC

New WLCG Transfers Dashboard

Multi-threaded HTTP Call Thread pool Async request (std::future c++11)

Main thread

Async call 1Async call 2

Get results

Aggregate

Return


38IT-SDC

Not just a re-writing

Possible re-think of UI

WEBAPI / UI

WEBAPI/UI

FTS FAX

FAX

EOS

FTS


39IT-SDC

Not just a re-writing

Possible re-think of UI VO specific UIWEB

API / UIWEB

API/UI

FTS FAX

FAX

EOS

FTS

WEBAPI

WEBAPI

FTS FAX

FAX

EOS

FTS

ATLASMonitoring UI


40IT-SDC

Summary

More flexible architecture Database-less Based on common building blocks UI code fully re-used

Thanks for your attention


41IT-SDC

Useful links AAA Dashboard

http://dashb-cms-xrootd-transfers.cern.ch FAX Dashboard:

http://dashb-atlas-xrootd-transfers.cern.ch FTS Dashboard:

http://dashb-fts-transfers/ui/ WLG Transfers Dashboard:

http://dashb-wlcg-transfers-new/ui/#

Xbrowse framework: https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework

29 November 2013Alexandre Beche - ITTF

http://dashb-cms-xrootd-transfers.cern.ch/

http://dashb-cms-xrootd-transfers.cern.ch/

http://dashb-atlas-xrootd-transfers.cern.ch/

http://dashb-atlas-xrootd-transfers.cern.ch/

http://dashb-fts-transfers/ui/



http://dashb-wlcg-transfers-new/ui/

http://dashb-wlcg-transfers-new/ui/

https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework

https://twiki.cern.ch/twiki/bin/view/ArdaGrid/XbrowseFramework

Date post:	24-Feb-2016
Category:	Documents
Upload:	taran
View:	31 times
Download:	0 times

WLCG Transfers Dashboard A unified monitoring tool for heterogeneous data transfers .

Documents