SKA SDP :A snapshot of recent technical directions
and conclusions
Chris BroekemaASTRON
Netherlands Institute for Radio Astronomy
Highlights
A whirlwind overview of recent prototyping and design work done within the SDP consortium.
Note: these are invariably proofs-of-concept. While concepts are taking shape, unless otherwise stated, no down-selects have been made.
Algorithm Reference Library
https://github.com/SKA-ScienceDataProcessor/algorithm-reference-library
Illustrative flow chart
4
DRAFT
Describing the data relationships between components
• All major calibration and imaging algorithms expressed in Python unencumbered by practicalities of I/O, memory management, optimisation
• Establlishes reference e.g. for later implementation by non-experts
• Prototypes of functional components (referentially transparent)
• Synthesis components: Fourier transforms (predict, invert) using 2D, wprojection, faceting, wslicing, snapshots
• Model for LOW sky(S3) and LOW station beam (OSKAR)
• Testing is memory-limited: largest image made so far is 25K by 25K pixels
• Largely complete
• 97% coverage test suite
Algorithm Reference Library (ARL)A Python prototype of the algorithms
5
Algorithm Reference LibrarySynthesis framework allows experimenting with components
6
Dask graph for ICAL11 ingests. 5 major cycles.
7
Execution framework: DaLiuge
https://arxiv.org/abs/1702.07617
Astronomer
Component Parametersdefault parameter values of components
Pipeline Logic reduction components and sequence
Data Parallelisationhints about the potential of parallelism
Algorithms what’s the best algorithm to get the desired
answer
Parallel Executionwhat is executed where
Parallel Codingwriting parallel code
Code Optimisationoptimise parallel code
I/O Optimisationoptimise I/O on hardware
OS and hardware co-designoptimise hardware for code to be run
HPC Software Engineer
OS level S/W Engineer
Computer H/W Engineer
Logical Graph Template
Component Parametersdefault parameter values of components
Pipeline Logic reduction components and sequence
Data Parallelisationhints about the potential of parallelism
Algorithms best algorithm to get the desired answer
Parallel Executionwhat is executed where
Parallel Codingwriting parallel code
Code Optimisationoptimise parallel code
I/O Optimisationoptimise I/O on hardware
OS and hardware co-designoptimise hardware for code to be run
Physical Graph Template
Physical Graph
Logical Graph
Astronomer
HPC SoftwareEngineer
OS level S/W Engineer
Computer H/W Engineer
Operator
SDP Consortium meeting, SKA Engineering Meeting, Rotterdam, Friday 16 th June 2017
SDP Integration Prototype (SIP)
Ben Mort, Fred Dulwich (University of Oxford) Brian McIlwrath, Chris Pearson, David Terrett, Nijin Thykkathu (Rutherford Appleton Laboratory)
Arjen Tamerus, Vlad Stolyarov, Verity Allan, John Taylor (University of Cambridge)
SDP System Integration Prototype (SIP) - activities
14
Current activities
• TANGO interfaces to TM• CSP interfaces (SPEAD, Pulsar search)• Master Controller interface to tasks and
execution frameworks• Prototyping a number of test capabilities
(pipelines)• LMC services such as LTS using distributed
Redis• Deployment of SIP code onto AlaSKA-P3
Next Steps
• Prototyping of the SDP buffer• Integration of LMC services (eg. LTS, Sky
model, QA) with execution frameworks
Exploring high-risk areas
Example outcome: resilience in running LMC services. By moving towards a micro-services like architecture using container orchestration the risk of single key service failure for our control interfaces is largely mitigated
Implemented
Aims of SIP
End-to-end SDP system prototype:• Prototype of all major external and internal
interfaces• Verification and testing of SDP (software)
architecture• Focused testing and analysis of
technology choices• Link / overlap with execution framework
work and vertical prototyping• Link to hardware prototyping
(AlaSKA-P3)
TM
CSP
SDP
LMC services
Capabilities / Execution framework(s)
LFAA?
Master Controller: Platform as a Service
• Service that starts, monitors and stops tasks– Manages CPU resources– Restarts failed tasks– Redundancy / failover– Service discovery
• Abstract interface– Multiple implementations for different sorts
of task• First implementation based on Docker swarm
mode• Mesos + Marathon & Apache Spark have now
also been implemented
SaaSSaaS
PaaSPaaS
IaaS (OpenStack)IaaS (OpenStack)
Middleware
Containers, logging & monitoring, visualisation
Containerisationa modern software paradigm
* http://diginomica.com/2014/07/02/virtualization-dead-long-live-containerization/
Container: A choice of engines
● Docker
● Rkt
● LXC/LXD
● Others: Singularity, runC
● Docker currently dominates – greatest set of tools, largest
community and adoption
Container Orchestration: choices
● Container build & run and Orchestration
● Container orchestration
● Batch processing - Old School!
● Bespoke – DALiuGE, Dask, AirFlow
We must schedule the pipelines
SDP: Logging Architecture
SDP: Visualisation & Integration
Software-defined networks
http://dl.acm.org/citation.cfm?doid=3075564.3075594https://www.astron.nl/~broekemahttps://gitlab.com/broekema/SDP_Controller
SDN capable switch
Host A
Host X
Host Y
./generator
./analyzer
./analyzer
Network controller
OpenFlowryu
SDP_controller
The following OpenFlow features are used in our proof-of-concept application:
– Set Field action destination MAC address
– Set Field action destination IP address
– Group Type=ALL: apply action instructions
– Group Type=ALL: output to multiple ports
All of the investigated switches worked to some degree. Many of the features we rely on are optional and are not guaranteed to be implemented by the switch vendor.
– The concept is viable
– We rely on optional features that are not always implemented
– performance and quality of implemented features varies wildly
– A software-defined network adds robustness over a conventional network
– Adding the network to the processing platform is a potentially disruptive new paradigm that offers great additional features
Numerical precision