Delivery System Design Document Number ................................................. SKA-TEL-SDP-0000025 Document Type ........................................................................ DRE Revision ................................................................................. 02 Author .... R. Simmonds, D. Aikema, I. Emsley, S. Gaudet, S. Goliath, Y. Grange, R. Lakhoo, S. S ´ anchez Esp ´ osito, D. Wallom Release Date ................................................................... 2017-07-15 Document Classification ........................................................ Unrestricted Status ............................................................................ Released
AAAI Authentication, Authorisation, Allocation and Identity ManagementAPI Application Programming InterfaceAuthN AuthenticationAuthZ Authorisation (based on US spelling)CA Certificate AuthorityGUI Graphical User InterfaceHPC High Performance ComputingHSM Hierarchical Storage ManagementIaaS Infrastructure as a ServiceICD Interface Control DocumentIdP Identity ProviderIVOA International Virtual Observatory AllianceLAN Limited Area NetworkLMC Local Monitor and ControlMPL Multi-Protocol Label SwitchingNREN National Research and Education NetworkOST Observatory Support ToolsSaDT Signal and Data TransportSDP Science Data ProcessorSIA Simple Image Access (IVOA protocol)SKA Square Kilometre ArraySKAO SKA OfficeSP Science ProductSRC SKA Regional CentreSSA Simple Spectral Access (IVOA protocol)SSD Solid State DriveTAP Table Access ProtocolTBC To Be ConfirmedTBD To Be DecidedTM Telescope MangerURI Uniform Resource IdentifierVLBI Very Long Baseline InterferometryVO Virtual ObservatoryWAN Wide Area Network
The Delivery System provides managed user access to the Square Kilometre Array (SKA)Science Catalogue and Science Products through Software as a Service tools. These toolsinclude International Virtual Observatory Alliance (IVOA) compliant web services. The UserPortal provides a graphical web interface enabling searching of Science Catalogues and re-questing transfer of Science Products. Requests will be managed by a bulk data transferservice to support managed movement of data to SKA approved sites.
To support integration with the SKA more generally, Delivery System services must beintegrated with central services supported within the observatory, including approved authen-tication and authorisation systems, and system health monitoring.
Delivery System middleware and tools will enable the use of SKA Regional Centres (SRC)[RD-01] in several ways. It provides services to facilitate efficient transfer of Science Productsto SRCs and the replication of the content of the Science Catalogue to SRCs. The transferof Science Products needs to be scheduled from the Science Data Processor (SDP) sites toprovide the most effective use of the available network capacity and prioritise the transfer ofScience Products (SPs) based on SKA defined policies. Also the ability to locate SPs thathave already been replicated to other sites could remove the need to transfer those same SPsagain. The ability to replicate the Science Catalogue is needed so that queries to this can beperformed against a database close to where the researchers are located thus providing fast,reliable searches.
The following documents are applicable to the extent stated herein. In the event of conflict be-tween the contents of the applicable documents and this document, the applicable documentsshall take precedence.
The following documents are referenced in this document. In the event of conflict betweenthe contents of the referenced documents and this document, this document shall take prece-dence.
Reference ReferenceNumber[RD-01] SKA-TEL-SDP-0000060 Regional Centres[RD-02] SKA Actor List, https://confluence.ska-sdp.org/download/
attachments/159907885/SKA.actors.pdf?api=v2
[RD-03] SKA-TEL-SDP-0000079 SDP Observatory Support Tools Design
This document presents the requirements and preliminary design for the SDP Delivery Sys-tem. The Delivery System addresses those elements of the SDP concerned with making data,which originates in the SDP Preserve Data function, available to users.
1.2 Scope of this Document
This document only presents information about components relevant to making data originat-ing from the SDP Preserve Data function available to telescope users. The scope of datadelivery includes querying and requesting of data by users and how data is distributed to othertrusted computing centres for further scientific use. These trusted centres are referred to asSKA Regional Centres (SRCs). Tools to support data movement to SRCs are described. Tele-scope users may query and request data at SDPs and at SRCs. How Science Products arecreated and placed into the Preserve Data store at each of the SDP sites is not discussed.
The two main components are 1) the Delivery Software, which is a software stack whosepurpose is to enable end user queries, user requests for data and management of data transferto SRCs, and 2) the Delivery Platform, which is the hardware for the software stack.
1.3 Assumptions made in this Document
This document assumes that there will be some form of SRC that will be responsible for datahosting for specific geographical regions. It is assumed that SRCs will allow users to runanalysis tools for post-processing activities. However, the actual analysis tools used at thesesites are assumed to be created by third parties.
Who hosts an SRC is not relevant to this document. An SRC could be provided by aregional High Performance Computing (HPC) hosting centre, by a University, or by commercialhosting facilities, for example Infrastructure as a Service (IaaS) Cloud computing providers.
The details of how Authentication and Authorisation information will be distributed withinthe SKA are not finalised, but in this document we have assumed that this information will beavailable from the Telescope Manager (TM). If that is not the case in the final design, it simplymeans that the same information should be obtainable from another source.
1.4 Layout of the Document
The functional description of the Delivery System is provided in Section 2, including decom-position of high-level functions. Section 3 provides a dynamic overview of the Delivery Systemarchitecture, providing details of how different components of the architecture interact. Sec-tion 4 identifies traceability for the allocation of functions to products, and the internal interfacesbetween products. Section 5 describes the allocation of requirements to functions. The ex-
ternal interfaces to the Delivery System are referenced in Section 6, and the risks associatedwith the implementation of the system are identified in Section 8.
The functional context for the Delivery System is presented in Figure 1.
Figure 1: Delivery System Functional Context.
The Deliver Data function provides location-independent access to the SKA Science Cat-alogue and Science Products for users. The function must be capable of operation at boththe SDP sites accessing data via the Preserve Data function. Having the function running atthe SDP sites will be critical during commissioning and to provide data access to users thatdon’t have access to SRCs. To ensure optimal utilisation of all allocated resources, includingnetwork bandwidth, data transfers to remote sites should be scheduled from each of the SDPsites.
The relationships with other SKA elements and SDP functions are depicted in Figure 1.This identifies the functional context for Deliver Data in the SDP and SRC environments,including all the interfaces to Deliver Data, with respective inputs and outputs. Details ofthe interfaces are provided in Section 6.
The Deliver Data function decomposes into the following functions, pictured in Figure 2:
1. Authentication, Authorisation, Allocation and Identity (AAAI) Management
2.1 Authentication, Authorisation, Allocation and Identity ManagementFunction
The Deliver Data function should support at least two authentication mechanisms: federatedidentity management and X509 proxy certificate based interaction. The AAAI function willinterface with SKA to retrieve lists of trusted identity providers (IdPs) and of trusted CertificateAuthorities (CAs).
The Deliver Data function will enforce authorised access to the SKA Science Catalogue,Science Products, and Delivery System functions. Orthogonal to the SKA AAAI, the usermay also have SKA Regional Centre authorisation to enable delivery of Science Products tothe SKA Regional Centre. The functions enabled by the combination of the two AAAI systemsare described in Figure 3.
Figure 3: Functions enabled by combination of Authentication and AuthorisationUpdated
SKA authorisation information is an identification of which groups may access which Sci-ence Catalogue entries and which Science Products. Groups with access to Science Cata-logue entries and Science Products are captured in the Science Catalogue data models.
Generally, groups correspond to observing projects, so that the members of one group willhave access to all the Science Catalogue entries and Science Products that are producedduring the execution of the project. Special role-based groups may exist at the SKA for Oper-ations Staff to represent access to all Science Catalogue entries and Science Products.
Figure 4: AAAI work-flow.
The AAAI work-flow is pictured in Figure 4.
The AAAI function decomposes into the following functions:
1. Authorise User - will interface to TM to request a list of groups of which a user is amember. It is assumed TM will manage user role information as inputs to group cre-ation and membership maintenance. It is assumed that the Preserve Data function willmanage the association of groups to proprietary Science Catalogue entries and ScienceProducts.
2. Authenticate User - will interface to TM to retrieve authentication information as a list oftrusted IdPs, as well as a list of trusted CAs. If a user provides X509 credentials, Au-thenticate User will evaluate the list of trusted CAs for authentication. If a user providesthird-party credentials, Authenticate User will issue an IdP Authentication request to theappropriate IdP, and accept the IdP authentication response. Failure to authenticate willrequire anonymous query access. The policy for anonymous access at the SDPs will bedefined in the future, by the SKA board / project office.
Prepare and Deliver will always require a credential for an SRC identity SRC’s are theonly delivery destinations for SKA data and are assumed to require user authorisationfor reception and storage of data.
The Query and Request work-flow is pictured in Figure 5.
The Query and Request function decomposes into the following functions:
1. Query Science Catalogue - Allows users to query the SKA Science Catalogue for Sci-ence Product discovery. A Science Catalogue query is looking at data from both SDPs,as catalogue content is replicated between the SDPs by the Preserve Data Function.
2. Filter On Authorisation - Access to proprietary Science Catalogue entries and ScienceProducts by any user is dependent on their authorisation. This function ensures thegroup listing for a user is part of Science Catalogue query criteria, provides authorisationinformation as part of displaying query results to all users, and ensures the authorisa-tion associated with the group listing for a user is maintained during Science Productselection and delivery.
3. Select Science Product - The SDP will provide a web-based graphical user interface(GUI) which will allow users to select Science Products from the Science Catalogueentries that are the query response, and submit User Requests.
4. Subscribe - Subscribe is pre-selecting Science Catalogue query constraints, format,logical Science Product types, destination, and the necessary credentials to then allowautomated discovery of new Science Products and submission of a fully formed UserRequest.
5. Request - Provides monitoring of delivery activity for a User Request, resulting in Sci-ence Product delivery.
6. Provide Science Product Usage Statistics - Aggregates Delivery Request completionsand forwards this information to an interface with the Control SDP function. User Re-quest completion may be the successful transfer of all Science Products, the successfultransfer of some Science Products, or the failure to transfer any Science Products.
2.2.1 Query Science Catalogue
The Query Science Catalogue function provides web-based query support of the local copyof replicated Science Catalogue entries, querying the Science Catalogue and its associatedlogical Science Products. Users (may be SKA or Observatory Support Tools (OST) Users) willquery either via a browser-based solution, or via direct access to query web services.
Users may request previews (thumbnails, snapshots) of Science Catalogue entries aspart of the Science Product result. The thumbnails could be from both continuum images orspectral lines.
As much as practical, Query Science Catalogue will be compliant with the following IVOAstandards:
• TAP (Table Access Protocol) – a service protocol for accessing general table data, in-cluding astronomical catalogues as well at general database tables. See [AD-01].
• SIA (Simple Image Access) – provides capabilities for the discovery, description, ac-cess, and retrieval of multi-dimensional image datasets, including 2-D images as well asdatacubes of three or more dimensions. See [AD-02].
• SSA (Simple Spectral Access) – defines a uniform interface to remotely discover andaccess one dimensional spectra. See [AD-04].
• DataLink – provides the linking of Science Catalogue entries to give access to the Sci-ence Products, further detailed metadata, related resources, and to services that per-form operations on the Science Products. See [AD-03].
The Query Science Catalogue function decomposes into the following:
1. VO-compatible general purpose querying of science catalogues (TAP) - To query alltypes of Science Catalogues: observation, transient source, local sky model, proposal,publication, and events. The mechanism for query will be the TAP standard from IVOA.
2. VO-compatible querying of multi-dimensional observations (SIA) - The IVOA Simple Im-age Access (SIA) protocol streamlines querying of Science Catalogue entries by restrict-ing possible query parameters to the support of multi-dimensional image querying. TheSIA service will rely on TAP for query execution.
3. VO-compatible querying of spectral observations (SSA) - The IVOA Simple SpectralAccess (SSA) protocol streamlines querying of Science Catalogue entries by restrictingpossible query parameters to the support of spectral querying. The SSA service will relyon TAP for query execution.
4. Query using a GUI web interface - The SDP shall provide a web-based GUI throughwhich to query the Science Catalogue. The GUI query parameters available to userswill include criteria for all available Science Catalogues. Query using a GUI will rely onthe TAP service for query execution.
5. Query for logical science products (DataLink) - The IVOA DataLink protocol will be usedto identify the relationship between Science Catalogue entries and the Science Productsassociated with those catalogue entries.
2.2.2 Filter On Authorisation
The Delivery System will filter query results and allow product selection using authorisationbased on group membership. See Figure 3.
In addition to querying public Science Catalogue entries and selecting public ScienceProducts, authorised users will also be seeing proprietary entries and selecting proprietaryproducts that their authorisation allows them to access. Only authenticated users are able totransfer data to SRCs.
The policy for anonymous access for public data sets at the SDPs will be defined in thefuture by the SKA board / project office.
2.2.3 Select Science Product
Upon receiving the results from querying the Science Catalogue, a user (may be an SKAor OST User) selects some subset of the list of Science Catalogue entries and submits aquery for the list of the associated logical Science Products. From the resulting list of logicalScience Products, a user then selects some subset of those products to request, formingthe basis of a User Request. A User Request is used to track Science Product delivery untilrequest completion. A User Request is the collection of information gathered from the userand augmented by the Prepare and Deliver function with the state of the associated DeliveryRequests. The User Request includes:
• Selected logical Science Products
• Science Product format(s)
• Transfer destination
• The necessary credentials to identify the SKA user and to allow access to proprietaryScience Products. These credentials are checked at each point of reference to the UserRequest content, avoiding User Request forgeries.
• The necessary credentials for authorising the destination of the delivery
• User to notify upon completion of the request. This defaults to the user making therequest.
This transfer request eventually results in Science Product delivery to the destination iden-tified by the user. This destination must be an SKA approved site (e.g. to an SRC).
The Select Science Product function provides immediate acknowledgement to the user ofreceipt of the User Request.
This function will gather the necessary credentials for authorising the transfer of the Sci-ence Products to the destination.
The Select Science Product function decomposes into the following:
1. Select Science Catalogue entries - Selection from Science Catalogue query results willbe made by users. The DataLink protocol will take as input the selected Science Cata-logue entries to produce a list of the associated logical Science Products.
2. Select logical Science Products - The DataLink service will provide catalogue entry-product relationship information for evaluation in the GUI. Logical Science Product se-lection will form the basis of the User Request content.
3. Select destination - The GUI will enable users to specify the SRC to which their ScienceProduct(s) will be delivered. This information will be added to the User Request.
4. Select format - The GUI will enable users to specify the formats in which the ScienceProduct(s) are to be delivered. This information will be added to the User Request.
5. Specify user to notify - Identify a user to be notified upon completion or failure of adelivery request. User identification will be consistent with authentication informationfrom TM. The user information will be added to the User Request.
2.2.4 Subscribe
A subscription results in regular, automated transfer of newly observed Science Products asthey are created in Preservation. The Subscribe function provides periodic execution ofstored Science Catalogue query parameters. Each subscription execution will generate aUser Request for each query that returns a list of Science Products that have yet to be deliv-ered, that is forwarded to the Request function. The User Requests and subsequent DeliveryRequests generated via subscription are subject to the same Data Distribution policies asdescribed in Section 2.2.5, Item 2.
The Subscribe function includes subscription maintenance, including creation and modifi-cation of the following constituents:
• The necessary credentials to identify the SKA user and to allow access to proprietaryScience Catalogue entries and proprietary Science Products if necessary.
• The necessary credentials for authorising the destination of the delivery
• User to notify upon completion of the User Request. This defaults to the user makingthe subscription.
Subscriptions may be deleted but the policy defining when and by whom if yet to be de-fined.
The Subscribe function decomposes into the following:
1. Persist query constraints and user selections - Store predefined queries, identifying alluser-specified elements from the Query and Select functions, as well as informationrequired to execute authentication and authorisation.
2. Modify persisted subscription request - Users may wish to modify existing subscriptionquery criteria. This function provides for the retrieval and update of subscription content,changing any of the user-specified elements for Query and Request. The user may alsomodify authentication and authorisation information associated with the subscription.
3. Manage subscriptions - Manage subscription persistence, including replication as pro-tection against system failure, authorisation for creation and modification of subscrip-tions, and clean-up of expired subscriptions. Initiate periodic execution of persisted sub-scriptions, including ensuring existing subscription queries are executed and that anyquery result differentials initiate a User Request.
2.2.5 Request
The Request function manages the life cycle of a User Request and its associated DeliveryRequests. It takes as input a User Request and generates one or more Delivery Requests,one per geographical location of the Science Products. The function consists of asynchronousmanagement of a queue of User Requests to be executed.
The Request function decomposes into the following:
1. Convert logical to physical Science Product - A query/response interaction with IndexScience Products to translate the logical Science Products into the collection of DROPinputs needed to create those physical Science Products. The information returned fromIndex Science Products will be appended to the User Request. Failure to identify asso-ciated Drop Uniform Resource Identifiers (URIs) will result in physical Science Productsbeing undeliverable. The User Request completion status will reflect this failure.
2. Discover physical Science Product locations - Interfaces to Respond to location query ateach SDP to retrieve the list of Drop URIs (TBC). Discover will append the URI list to theUser Request. Once each Science Product from a User Request has an identified loca-tion, this function calculates an optimum transfer breakdown based on science productlocation. The User Request will be updated with this breakdown.
The Data Distribution Policy for queueing Delivery Requests includes priority and bud-get to determine the ordering of the Delivery Request queue for each science productsource. Priority is set by Science Product type, Science Product, and user. Transfer
Budget is limited by Science Product and user. Transfer Count limit controls how manytimes the transfer of the same Science Product to the same destination is retried.
3. Manage Delivery Request Life Cycle – The function submits a Delivery Request to theappropriate Prepare and Deliver. Data delivery occurs asynchronously, and this functionmonitors the transaction. If all Delivery Requests for a User Request are successful, theUser Request terminates successfully. If a Delivery Request fails partially or completely,this function will attempt to locate an alternate source of the Science Product inputs, andif found, will generate a Delivery Request for that source. If all attempts to execute anyof the Delivery Requests fail, the User Request is terminated as a partial or completefailure. Manage Delivery Request Life Cycle will log the successful or failed terminationof a Delivery Request. Once all Delivery Requests associated with a User Request areterminated, Manage Delivery Request Life Cycle will notify the user identified in the UserRequest. The notification mechanism will be chosen later. It could be email, IM, or ateam collaboration application.
4. Monitor User Request – A GUI to allow users to track a User Request as it is beingfulfilled.
2.3 Prepare and Deliver Function
The Prepare and Deliver function enables Delivery Requests to be queued and prioritised. Itmanages the delivery of Science Products to the requested destination. The Delivery Systemfunctions need local storage for working space and to buffer Science Products being trans-ferred.
A Delivery Request can complete in the following states:
• all Science Products delivered to destination.
• some Science Products not delivered due to partial failure of any of the Drop accumula-tion, Science Product Preparation, or Bulk Data Transfer functions.
• no Science Products delivered to destination.
The work-flow for Prepare and Deliver is pictured in Figure 6.
The Prepare and Deliver function decomposes into the following:
1. Respond to location query - Interfaces to Index Science Products at the local SDP toprovide the existing locations for the Drop inputs to the physical Science Products. Thisresponse is a list of Drop URIs (TBC). Respond to location query will forward all infor-mation it receives. An Index Science Products failure will be reflected in the response.
2. Manage data delivery - Consists of asynchronous management of a queue of DeliveryRequests to be executed. Entries in this queue are considered against the amount ofstaging space available to Prepare Science Product, the amount of network bandwidthavailable to Bulk Data Transfer, and the distribution policy. Manage data delivery will in-terface to Persist Science Products to request Drop retrieval when conditions for Prepa-ration are met, starting the Persist Science Products push of Drops to Transfer Staging.
Manage data delivery will enforce authorised Drop retrieval. A Delivery Request mayrequire more staging space than is available, in which case the request will be dividedinto a list of sub-requests to fit the available Transfer Staging space. The staging spacewill be monitored for Drop arrival, triggering Prepare Science Product. Manage datadelivery will interface with Prepare science product to know when preparations are com-plete. Following preparation completion, Bulk Data Transfer will be scheduled. Bulk DataTransfer will be monitored for transfer completion, and Manage data delivery will recordthe status of the delivery in the Delivery Request.
3. Operate data delivery - Provides manual override of Prepare and Deliver. OperationsUsers may manually start, pause, continue, and terminate Delivery Requests. Oper-ations Users may also interrupt or terminate a Delivery Request during Prepare andDeliver.
4. Prepare science product - Create deliverable Science Products from a list of Dropspresent on Transfer Staging.
5. Bulk data transfer - As Science Products accumulate on Transfer Staging, copies Sci-ence Products to the destination over the network (a LAN for OST). The Science Prod-ucts are removed from Transfer Staging upon successful completion of a transfer.
6. Replicate Catalogue - start Science Catalogue replication to a specified approved site.
7. Science Product Receive - SRCs will interface to Bulk Data Transfer, as part of send-ing/receiving a data transfer. The accepting endpoint for the transfer is subject to thesame network capacity management and interruption recovery configuration.
2.3.1 Respond To Location Query
This function interfaces with Index Science Products to provide the translation from physicalscience product identifiers to Drops with identifiers for physical retrieval.
2.3.2 Manage Data Delivery
Due to resource limitations between the SDP sites and SRCs imposed by the cost of largebandwidth WAN links and given the volume of data that will be generated by the SKA tele-scopes, it will be necessary to optimise transfers. This will include scheduled rather thanon-demand data transfer as the normal mode of operation.
2.3.3 Operate Data Delivery
The Operate Data Delivery function allows Operations Staff to start, pause, resume, andterminate the Bulk Data Transfer function. This is to control Delivery System bandwidth con-sumption.
2.3.4 Prepare Science Product
The Prepare Science Product function will create a Science Product from a list of Dropspresent on Transfer Staging.
2.3.5 Bulk Data Transfer
Bulk Data Transfer can occur from SDPs to SRCs. This function provides support for:
• network capacity management
• interruption recovery
• efficient WAN use
The Bulk Data Transfer function will copy science products to a destination over a WideArea Network (WAN) (LAN for OST) as Science Products accumulate on Transfer Staging.The Science Products are then removed from Transfer Staging upon successful completion.
See Section 3.2 for a description of the hardware capacity available for Bulk Data Transfer.
The Delivery System provides services that maintain replica databases of the Science Cata-logue search information used by the services.
Replicate Catalogues represents the Application Programming Interface (API) required toreplicate Science Catalogues to SRCs.
2.3.7 Science Product Receive
Science Product Receive represents an API for receiving and storing Science Products forOST or at SRCs, outside of the Preserve Data function in SDP. It provides two functions:
• to answer whether the requesting user has authorised access to the receiving locationfor initiation of a User Request.
• to receive Science Products from Bulk Data Transfer.
The Delivery System sub-elements will interact with the other sub-elements of the SKA SDP.These include the Preserve Data sub-element that is responsible for long term Science Cata-logue and Science Product indexing and persistence, and Control SDP that handles interac-tions with and monitoring of the SDP.
3.1 Delivery System Dynamic View
Nominal Deliver Data behaviour includes the following:
• Authentication, Authorisation, Access, and Identity - these user actions are common tosequences which access Science Catalogues and Science Products.
• Query and Request - anonymous and authenticate users search Science Cataloguesfor Science Products. The query parameters for the search may extend to any ScienceCatalogue parameters. Once a user has discovered the Science Products available, theuser submits a User Request for Science Products be transferred to storage accessibleby the Observatory Support Tools, or at an SRC.
• Prepare and Deliver - manage the transfer of selected Science Products.
3.1.1 AAAI
The AAAI work-flow is pictured in Figure 4.
1. The Authenticate User function will carry out the following for authentication via:
(a) X509:
• verify user-provided proxy certificate is valid.• verify that certificate used to create this proxy was issued by an SKA trusted
CA.• retrieve identity from user-provided proxy certificate.• given certificate verification success, proceed to AuthZ (authorisation).
(b) Identity Provider (IdP):
• forward AuthN (authentication) connection to appropriate IdP from the SKAOffice (SKAO)-maintained list for it to perform AuthN operation
• given match success at the approved IdP, carry on to AuthZ
2. The Authorise User function, given User Credentials, will query for the set of SKA groupsof which that identity is a member. The set of groups identifies the Science Catalogues,Science Products, and services the requesting user may access.
The Query and Request work-flow is pictured in Figure 5. Where additional decomposition isinformative, lower level work-flows are provided in the following section.
Query and Request will support external users of the SDP as SKA Users, and internalusers of the SDP as OST users.
1. The Query Science Catalogue function will provide both SKA and OST users with fourmethods to query the catalogue: a web client, an IVOA SSA service, an IVOA SIAservice and an IVOA TAP service. The three former methods will access the cataloguethrough the latter one. The IVOA TAP service results will be filtered for authorisationbased on group membership.
Eventually, the queries made through the web client could ask for further detailed Sci-ence Catalogue metadata. This information will be provided by the IVOA DataLink ser-vice.
Figure 7: Query Science Catalogue work-flow.
The work-flow for Query Science Catalogue is pictured in Figure 7.
2. The Request function will manage the lifecycle of the User Request and its associatedDelivery Requests. The User Request may result in multiple Delivery Requests beingcreated and queued for execution, and the Request function makes that decision.
The User Request Lifecyle, pictured in Figure 8, requires a backing data store for create,read, update, and delete operations. Visibility into the state of the User Request Lifecycleis provided by the Monitor User Request function.
The Request function collates the completion of Delivery Requests against their origi-nating User Request, capturing the overall completion state (success or failure) of theUser Request.
The Provide Science Product Usage Statistics function will collate the information avail-able from User Requests to provide information on which users access which scienceproducts. This information will be provided to the Control SDP function. This enablestracking of access to proprietary data.
The Delivery Request Lifecycle, pictured in Figure 9, requires a backing data store forcreate, read, update, and delete operations. The Delivery Request relies on the Re-ceived Authorisation from the User Request to ensure a transfer has write capability atthe destination. The information available from Delivery Requests may also be added tothe usage statistics to provide a log of transferred Science Products.
Figure 10: Request work-flow.
The work-flow for Request is pictured in Figure 10.
3. The Subscribe work-flow is pictured in Figure 11. Subscriptions require a backing datastore for create, read, update, and delete operations. Subscriptions store user authori-sation information for forwarding to User Request creation.
3.1.3 Prepare and Deliver
The work-flow for Prepare and Deliver is pictured in Figure 6. Where additional decompositionis informative, lower level work-flows are provided in the following section.
1. Manage data delivery will queue Delivery Requests for execution. Data DistributionPolicy will inform the order of queueing, and includes how many times a Science Producthas already been transferred, as well as budget by project ID.
Figure 12: Manage data delivery work-flow.
The work-flow for Manage data delivery is pictured in Figure 12.
Manage data delivery will update Delivery Requests during transfer execution.
Manage data delivery will evaluate the amount of staging space needed for a Deliv-ery Request, and will initiate the retrieval from Persist Science Products when there issufficient space available.
Operate data delivery may be used manually to control (start, pause, continue, termi-nate) any Delivery Request. This will interact with the queueing and request transfer.
Manage data delivery will initiate the Bulk Data Transfer step for Science Product trans-fer.
2. The Prepare science product will operate on the staged drops prior to the Bulk DataTransfer step. This will require access to the Transfer Staging. It will be triggered byManage data delivery when the set of drops necessary to create a science product areavailable on Transfer Staging.
The work-flow for Prepare science product is pictured in Figure 13.
The Delivery Platform is the hardware that will host the Delivery Software at each SDP site.This software includes (1) web based services that need to be responsive for real-time inter-action with users and, (2) data transfer services that will write data to the Wide Area Network(WAN) or to the LAN for OST.
For (1), these services should be highly available, but they have modest compute and I/Orequirements. It would be best to host these on more than one physical server to providefor fail-over of services if there is a hardware failure or when maintenance is required. Twodual socket nodes, each with 256GB RAM have been costed for this. These will each haveat least 2TB of local disk to manage databases used by the local services. The expectedload on these nodes is not sufficient to require the use of a dedicated load-balancer. This isan outward facing platform, which runs the risk of external attack. Denial of service attacks(either deliberate or due to unexpected high public interest) on this platform must not interferewith the rest of the SDP system. Breach of this platform must not lead to access to the rest ofthe SDP system.
For (2), systems with high I/O capabilities will be required. Based on technologies thatare currently available this will require multiple compute servers hosting Solid State Drives(SSDs). These will need to connect to an Ethernet switch in the preservation and deliverynetwork to combine their output to fill the WAN connection that carries data out of the SDPsite. This WAN connection is currently specified to run at 100 Gb/s.
The preservation and delivery network design, that this part of the system will be integratedin, is currently expected to be based on a 100 GbE core, with 25 GbE connectivity to the nodes.The staging hosts therefore need at least one 25 GbE network interface per node. In totalthe staging hosts will need at least 4 of these interfaces to fully utilise the external 100 GbEconnection. The staging hosts need to simultaneously receive data from the PreservationComponent to the staging media. This may require additional 25 Gb/s Ethernet interfaces,although the unidirectional nature of both transfers may allow sharing of hardware resources.
In order to keep the network connection full, the staging system will need to be able toreceive data from the Preservation component at least 100 Gb/s and write it out to the WANconnection at 100 Gb/s. The aim is to not limit performance so both the Preservation com-ponent and the staging system will require fast media, and tuned network endpoints. Forcosting we have assumed 6 (six) staging nodes will be used with each host having a single25 Gb/s Ethernet adaptor.
Assuming that each SSD can receive data at 500MB/s each node will need a minimum of
8 SSDs. To provide a safety factor we have costed each node to have 12 SSDs.
We are assuming very limited resilience for the staging nodes, since there is currentlyno resiliency requirement on this part of the system. Having individual nodes shut down willdowngrade performance, but delivery and staging should still be possible while at least onestaging node is still operational. We will not duplicate data over staging nodes, data on a failednode will have to be re-staged.
The Delivery Platform represents the dedicated hardware and network connectivity for Sci-ence Product transfer.
Delivery node - at least two physical nodes providing the Delivery services with some faulttolerance, an interface to the SaDT WAN, and a Storage Access interface.
Delivery staging storage - data staging and scratch space for use by the Delivery Plat-form. Delivery staging storage includes the Storage interconnection system, which is a fastconnection to Delivery nodes.
4.2 Delivery Software View
Figure 15 shows the decomposition of the Delivery System element into its products.
All software products interface to LMC [AD-10] for monitoring and logging while deployedat the SDPs.
Tiered Data Delivery Manager represents the software needed to move science productsfrom SDPs to SRCs.
Science Catalogue Service represents the asynchronous replication of science cataloguesfrom SDPs to SRCs.
Delivery System User Interfaces represent all of the tools that execute within a web browser,or via existing command-line http clients (e.g. wget, curl). SKA Users and OST Users providethe inputs to this set of products.
Delivery System Service Manager represents all user accessible software and serviceswith which the User Interfaces communicate. The Services Manager software should be acommon software stack that can be deployed at both SDPs. As much as possible, the ServicesManager products will be exposed via a RESTful API.
Delivery System Interface Managers represents the interfaces to the Delivery System,both within the SDP, and external to the SDP.
Persistence Management represents the software needed to provide staging buffers forthe Prepare and Deliver function, and the User Request, Delivery Request, and Subscriptionstores for the Query and Request and Prepare and Deliver functions.
Prepare Science Product represents the software to execute the Prepare Science Productfunction.
4.2.1 Tiered Data Delivery Manager
Tiered Data Delivery Manager comprises a further set of services that are provided to helpsupport the efficient transfer of data between different locations running the Delivery Systemsoftware stack.
1. Data Transfer Scheduler
Data Transfer Scheduler is used to support observatory distribution policy for bulk dis-tribution. Science Products are transferred according to the Delivery Request queue.There is one queue per data source, where a data source is an SDP. Each queue con-tains source-specific prioritised entries for transfer.
Operations Staff may enforce observatory distribution policy with budget and bulk distri-bution configuration choices. Data Transfer Scheduler is subject to these constraints:
• information such as Science Product size,
• transfer priority based on previous bandwidth consumption and transfer, and
• a transfer budget is maintained to enable fair-share network consumption. It pro-vides options to lower the priority of transfers based on previous bandwidth con-sumption. The distribution will be based on project/proposal ids and other TBDcriteria.
4.2.2 Science Catalogue Service
An API that provides the specific service for the replication of Science Catalogue metadata. Acomplete copy of the Science Catalogue will be maintained at each SRC.
4.2.3 Delivery System User Interfaces
Delivery System User Interfaces comprises a set of Graphical User Interfaces (GUI) as webclients for specific Delivery System services.
The UIs are provided for the Astronomer, Telescope Operator, and Public Interfaces toaccess the Science Catalogues and Science Products. The same access is available to allusers, though access to them could be limited by policy.
1. Science Catalogue Query Interface - GUI that interacts with the TAP and DataLink ser-vices to provide user searchability for Science Catalogues and selection of ScienceProducts for transfer (i.e. the creation of a User Request).
2. User A&A Login Interface - GUI that interacts with the Authentication Service, the Au-thorisation Service, and the client for forwarding IdP enquiries.
3. Monitor User Request Interface - GUI that interacts with the User Request Service toenable tracking of the state of a User Request, which includes preparation and transfer.
4.2.4 Delivery System Service Manager
Delivery System Service Manager comprises the set of public-facing and internal web ser-vices that enable the Delivery System. Each public-facing service will ensure any access toproprietary Science Catalogue entries or Science Products is authorised. Private servicesmay use a trust relationship for optimisation.
1. Location Service - a private service that identifies the physical location of Science Prod-ucts. This service takes as input a list of Science Products and a destination, and deter-mines which Science Products have already been delivered to the given destination.
2. User Portal Framework - provides a common look and feel to all GUIs.
3. IVOA TAP Discovery Service - implements [AD-01] as a public-facing web service. Thisservice interfaces with the Delivery System Interface Managers for execution of queriesagainst the storage engine.
4. IVOA SIA Discovery Service - implements [AD-02] as a public-facing web service. Thisservice provides a restricted set of query criteria, and relies on the TAP Discovery Ser-vice for query execution.
5. IVOA SSA Discovery Service - implements [AD-04] as a public-facing web service. Thisservice provides a restricted set of query criteria, and relies on the TAP Discovery Ser-vice for query execution.
6. IVOA DataLink Service - implements [AD-03] as a public-facing web service. This ser-vice interfaces with the Delivery System Interface Managers for execution of queriesagainst the storage engine.
7. User Request Service - provides a public-facing web service to create, read, update,and cancel User Requests. It accesses the User Request Store.
8. Delivery Request Service - provides a private service to create, read, update, and cancelDelivery Requests. It accesses the Delivery Request Store.
9. Subscription Service - provides a public-facing service to create, read, update, and can-cel Subscriptions. It accesses the Subscription Store. Also provides for periodic execu-tion of subscriptions, and will interface with the User Request service as necessary forUser Request creation.
10. Authentication Service - public-facing service that provides user authentication informa-tion for all other services. This will support at least federated IdPs with an approved listprovided by TM, and X509 certificates with a list of trusted CA provided by TM. IdPs willreturn a third-party token for use elsewhere.
11. Authorisation Service - public-facing service that provides user authorisation informationfor all other services. This will return a group membership list, obtained from TM.
12. Receive Platform Storage API - provides the API for the destination endpoint when trans-ferring bulk data.
13. Tier 1 Receive Service - provides a private service implementation of the Receive Plat-form Storage API at an SRC.
4.2.5 Delivery System Interface Managers
Delivery System Interface Managers comprises the set of clients for interfaces external to theDelivery System.
1. Local Monitor and Control (LMC) Manager
The LMC Manager comprises the set of clients for interfacing with the LMC. The interfaceallows health, status, operational event, and alarm reporting by the Delivery System backto the common SDP mechanism through the LMC at an SDP site.
(a) Client of LMC Monitoring Service – interfaces to LMC Monitoring [AD-10]. Utilisedby all Delivery System software that reports for the purposes of LMC Monitoring.(Interface to C.3.2.4.4 EM Data Collector).
(b) Client of LMC Logging Service – interfaces to LMC Logging [AD-10]. Utilised byall Delivery System software that logs execution. (Interface to C.3.2.4.3 EM LogManager).
(c) Science Product Usage Statistics Reporting - interfaces to the QA Metric Objectinterface in LMC, for the purpose of identifying the history of user access to ScienceProducts. (Interface to C.3.2.1.1 QA Interface Library.).
2. TM Manager
The TM Manager comprises the set of clients for interfacing with the AAAI informationavailable from the TM.
We assume that AuthN will be provided using several mechanisms including federatedidentity management and X509 certificates. With federated identity management, sitesbeing accessed by users would call back to approved IdPs to verify user identities withthis list of IdPs being managed by the observatory. With X509 based AuthN, sites wouldcheck that user provided proxy certificates are valid and were created using certifi-cates signed by observatory-trusted CAs. AuthZ will need some other service, suchas Grouper [AD-09] to manage which users can access information associated with par-ticular project IDs.
(a) User Authentication IdP List Client - interfaces to TM to retrieve the list of trustedIdPs, for the purposes of third-party user authentication.
(b) User Authentication CA List Client - interfaces to TM to retrieve the list of trustedCAs, for the purposes of X509 user authentication.
(c) Group Membership Client - interfaces to TM to retrieve the list of groups that a useris a member of, for the purposes of user authorisation.
3. Preservation Manager
The Preservation Manager comprises the set of clients for the Preservation Softwarefrom the Preservation System.
(a) Storage Index Database Client - interfaces to Preservation Software to retrieve thedisk location of drops, for the purpose of initiating drop retrieval.
(b) Long-Term Storage Hierarchical Storage Management (HSM) software Client - in-terfaces to Preservation software to initiate drop retrieval.
(c) Science Product Catalogue Database Client - interfaces to Preservation Softwareto query the catalogue store, for Science Catalogue retrieval as a response to userqueries.
4. WAN Manager
The WAN Manager comprises the set of clients for interfacing with the network providedexternally.
The SKA project/board will identify the entity responsible for designing the WAN thatwill be used for distribution of data externally to the SDP sites at rates that satisfy SKArequirements. This will be defined in the Interface Control Document (ICD) betweenSDP and TBD. The Delivery System will provide information on the rates that satisfy thescience requirements and work with TBD to define other specific requirements for theWAN. The Delivery System will set up the transfers between pairs of sites involved inany transfer.
The majority of traffic should be moved over point-to-point connections directly to SRCsor other SDP sites. Given that these will often be routed over international WAN linksthe flow of traffic on these links should be closely controlled. Other traffic may go toNational Research and Education networks (NRENs), but care will be needed to notinterrupt the main WAN traffic. Other users may access data from the Internet, but thatwill require a firewall at the SDP site which will limit bandwidth. If these connectionsshare links with the main transfer paths, they could lead to reduced effective utilisationof the international WAN links.
In addition to the large bandwidths that will be required for moving Science Products,there also needs to be small additional bandwidth to distribute Science Catalogues.
Very Long Baseline Interferometry (VLBI) data distribution will be controlled by TM , andis therefore out-of-scope for the Delivery System. VLBI data will be moved using an un-reliable protocol, on the same network as the Science Product transfers. Bandwidth willneed to be guaranteed to ensure that VLBI data does not get dropped due to the num-ber of Science Products being transported to remote sites using protocols optimised touse all available bandwidth. This guarantee should be provided at the network layer, butcould be provided by controlling the data rates from the endpoints if that is not possible.
It is assumed that all data transfers between SDP sites and SRCs will be scheduled tomake best use of the bandwidth on the international WAN links. In the current design
the sending site invokes transfers, so that it has control over what is being placed onits outgoing network. This in itself is not enough to guarantee the high utilisation of theinternational WAN links if there is contention on another network segment in the path. Anexample of this would be where the receiver is also receiving data on the same incominglink from another SDP or from another SRC site.
There are several ways of overcoming the problem described above and it will be theTBD element’s role to engineer a solution to this problem. An example of how this couldbe resolved is the approach used by CERN. In this approach dedicated network pathsare used to distribute data from CERN (Tier-0) to the regional Tier-1 sites. These providededicated bandwidth end-to-end which is not shared with other traffic. A second networkcalled LHCONE connects all of the Tier-1 and Tier-2 processing sites. This network isimplemented using different technologies depending on what is available on a particularnetwork link (e.g. VLANS, MPLS paths). On most links LHCONE shares bandwidth withother traffic, but since network engineers can differentiate between LHCONE traffic andother traffic, they have the opportunity to add traffic management policies to providesuitable sharing of available bandwidth.
(a) LAN/WAN Capacity Client - interfaces to the Delivery Platform WAN Interface forthe purposes of determining capacity.
4.2.6 Persistence Management
Persistence Management comprises the APIs for data and metadata persistence within theDelivery System.
1. Staging Buffer - the software interface to staging space used by Prepare and Deliver,when staging Science Products prior to their movement via Tiered Data Transfer. Thisis currently expected to be the POSIX file system.
2. User Request Store - the software interface to, and the schematic definition of, the datastore for User Requests. This is currently expected to be an interface to C.3.2.6.1 LocalDatabase Services.
3. Delivery Request Store - the software interface to, and the schematic definition of,the data store for Delivery Requests. This is currently expected to be an interface toC.3.2.6.1 Local Database Services.
4. Subscription Store - the software interface to, and the schematic definition of, the datastore for Subscriptions. This is currently expected to be an interface to C.3.2.6.1 LocalDatabase Services.
4.2.7 Prepare Science Product
Prepare Science Product comprises the set of applications for transforming a drop into aScience Product. These applications utilise the Staging Buffer interface to carry out any Dropoperations.
1. Format Converter - convert the Drop to a Science Product in the format specified by theUser Request. If the User Request specifies the as-is file format for Science Products,for a Science Product this application will not be run.
2. Remove Drop Signature - remove any Drop metadata to create a Science Product in theas-is file format for the product type.
4.3 Product Tree Interaction View
The following sections describe the interactions of product tree leaves, to fulfil the require-ments specified in Delivery System use cases. The set of use cases presented is sufficient toshow the involvement of all Delivery System product tree leaves.
Figure 16 is an overview of the use cases used to describe the interactions of the DeliverySystem product tree elements.
4.3.1 Authenticate and Authorise User
Figure 17 describes the sequence between product tree items for User Authentication andAuthorisation.
4.3.2 User Queries Science Catalogue
Figure 18 describes the sequence between product tree items for the use case of a UserQueries Science Catalogue.
4.3.3 Request Data
Figure 19 describes the sequence between product tree items for the use case of RequestData.
4.3.4 Execute User Request
Figure 20 describes the sequence between product tree items for the Execute User Requestuse case.
4.3.5 Deliver Prepared Data to User
Figure 21 describes the sequence between product tree items for the Deliver Prepared Datato User use case.
The interface to LMC is described in Section 4.2.5, Item 1.
6.2 Interface To Preservation System
The interface to the Preservation System is described in Section 4.2.5, Item 3.
A simplified version of this interface would be deployed at SRCs to handle interactions withthe local storage system.
6.3 User Interfaces
The interface to the User Interfaces is described in Section 4.2.3.
6.4 WAN Requirements
The WAN Interface is described in Section 4.2.5, Item 4.
6.5 Interface To OST
The interface to OST consists of allowing observatory access to Query and Request producttree elements, described in Section 4.2.3, and Prepare and Deliver product tree elements,described in Section 4.2.4.
7 Next Steps
To progress the design further prototyping will be performed. It should be noted that there isvery little risk currently in this element with the size of the final Science Products that havebeen requested from the Key Science Projects. Prototyping will mainly allow for the finali-sation of interfaces and evolution of data transfer scheduling that we are recommending beperformed. There exists many operational portal instances capable of the SKA functional-ity and features (ALMA, CADC, GAVO, ASKAP, cyberSKA), so many of the functions in thedelivery system are already working in production systems and therefore don’t present anyrisks.
Prototyping of interfaces is required to define the interfaces between the Delivery subsystemand the other SKA subsystems (Preservation, LMC and TM).
• Preservation – This prototyping work in collaboration with the Preservation team willwork to understand the data products and to data engineer the CAOM data model insupport of the Science Catalogue Query UI and the IVOA data models which underpinthe IVOA services. This work will also help to identify any missing data models or missingelements in the baseline data models. We will use the CADC implementation of the IVOAservices for this work. These are already deployed at CADC and we will also deploythem on the IDIA systems in Cape Town. We will used data we receive from ASTRONand MeerKAT for testing until we have examples of SKA Science Data Projects.
• LMC – For LMC we will work with the LMC team from SKA SA to develop an interfaceprototype that will interoperate with their LMC prototype.
• TM – For TM, we will work with the AAAI team from TM in Italy to develop an interfaceprototype that will interoperate with their Authentication and Authorization prototype.
7.2 data transfer issues
Prototyping work with the data transfer and transfer scheduling will mainly use existing toolsutilised in other big data projects. Data items are treated as first class objects in the Condorsystem, so it would be a good candidate to use in an initial prototype with the recently clarifieddelivery platform scope. This can work together with with a number of GridFTP implemen-tations including the one provided in the Globus cloud based data transfer platform. We hadpreviously indicated that the NGAS data management system would be a good choice to ex-plore providing some of the functionality, and this is still likely true. However, with the newscope definition it may provide more functionality than is required.
Systems such as condor are often used with external tools to add functionality withoutneeding to modify the Condor code. We can do this to add the subscription service and toadd other more complex highly scalable schedulers, such as the Maui or Moab scheduler, toprovide the rich policy implementations and massive queuing capacity that a final productionversion of the Delivery system should provide. We will work with the the designers of thepreservation system to detail the interface to that element and insure it works with the datamovement prototype we are creating.
8 Risks
The known risks carried by this element are:
• Evolving requirements, especially from the Operations Plan and the Data Flow AdvisoryPanel.
• Lack of way to copy Science Products from SRCs currently. This is because the currentdesign does not have interface to do this. Without that objects already copied out ofthe host countries cannot then be transferred on to other SRCs that request the sameScience Product.
• Final data formats are not yet defined. This will limit prototyping to data formats used incurrent radio astronomy projects.
• Available bandwidth – this risk is carried by the SKA project / board.
![[published at AAAI-2013]](https://static.documents.pub/doc/80x56/568161f7550346895dd222ee/published-at-aaai-2013.jpg)
