Home > News > SDP eNews - December 2017

SDP eNews - December 2017

SDP Consortium Progress since last eNews submission

M19 - All things Architecture

The next significant milestone for the SDP Consortium is M19 on the 30th November. M19 is a lightweight review of the current state of the SDP software architecture and forms a checkpoint with the SKAO to present the current state of the architecture and progress towards Critical Design Review (CDR).

All relevant SDP SEI style views and other supporting documentation for functionally close blocks of architecture will be reviewed. The goal is to present the information in sufficient detail to facilitate immediate use by a knowledgeable audience member.

In preparation for this review, a Stakeholders Overview will take place. The required participants will include key members of the SDP architectural design team and knowledgeable stakeholders from SKAO. This review is to ensure that the high-level architecture documentation, for the purposes of providing an introductory, high-level explanation of the design, is sufficiently complete. In preparation for this overview, key members of the SDP Architecture design team recently selected a collection of views for review and this information has been distributed to the SKAO.

Comments and or suggestions made during the Stakeholder Overview and the official M19 review will guide further architectural priorities ahead of M20 (Element pre-CDR).

SDP Architecture doc

Figure 1 - collection of SDP Architecture views ready for review with SKAO

SC17 – Denver, Colorado

SC17 picture
SC17 Phil and Rosie Banner

SuperComputing 2017 (SC17) was held in Denver, Colorado, during mid-November. It is the last such conference to be held before the SDP CDR. The keynote presentation of the conference was given by the SKA Director General, Professor Phil Diamond, and SKA Regional Centre and SDP Project Scientist, Dr Rosie Bolton. They explained the background and relevance of radio astronomy, the aspirations and developments of the SKA and in particular the computing challenges it faces.

The timing of the conference and the topic of the keynote was ideal as the SKA consortia CDRs begin in the coming months and the organisation gears up for construction. The conference provided opportunities for attending SDP members to test industry views on SKA, such as the desire to use Scaled Agile approaches for software construction (especially SAFe), gain insight into vendor roadmaps and where feasible pursue collaborative work on applicable prototyping activities.

SC 17 SKA Keynote

Keynoters Philip Diamond, Director General of SKA, and Rosie Bolton, SKA Regional Centre Project Scientist and SDP Project Scientist on stage at SC17.

The conference also provided an opportunity for SDP Consortium members to meet and get feedback from leading industry experts in Birds of a Feather (BOF) sessions related to SKA computing, and contribute to workshops on topics of direct interest to the community.

SDP Construction Planning

Since the last eNews submission significant progress has been made on the SDP Construction Plan and its supporting documents. This progress consists of:

  • Completion of the Rollout Plan that encapsulates the schedule for Construction.

  • Formulation of the framework and direction of the Construction Plan to ensure alignment with the SAFe framework.

  • Completion of an initial high level Construction Plan based on the framework described above.

  • Completion of an initial Work Breakdown Structure (WBS) in accordance with SAFe ideologies. The Product Breakdown Structure is not affected at this stage.

  • Delivery of a webcast to SDP Consortium members to present the current status and key points of the Construction Plan including the Rollout roadmap (refer to figure below). This was well received with additional discussion points raised for further investigation and consideration.

SDP Construction Roadmap - WIP

Figure 2 – SDP Construction Schedule and Roadmap. The SDP Commissioning needs to be in-step with other aspects of the telescope which is provided by an Assembly Integration and Verification (AIV) system.

Significant progress has been made, but the Construction Plan remains a work in progress. The final version will be ready for April 2018 ahead of the SKA System Review.

SIP – SDP Integration Prototype

The SDP Integration Prototype (SIP) is an active project to produce a lightweight horizontal prototype for the SDP system, providing a verification of the SDP architecture, and a test of all major internal and external SDP interfaces. SIP is under active development by a small distributed Agile team, working closely with the SDP architecture design group and the SDP hardware prototype, an OpenStack software-defined bare metal private cloud.

To take a practical approach towards delivering an initial working system, we have chosen to develop SDP components as a set of independent deployable (and testable) microservices, which make use of container orchestration (currently utilising Docker Swarm). These provide a modular, highly available, and horizontally scalable deployment of the SDP domain specific and platform services. In turn, this set of services support the core functions of the SDP, namely, the real-time and batch science pipeline workflows, and data ingest from Central Signal Processor (CSP).

The SIP team is currently at work developing a minimal set of services and a select number of science pipelines and data ingest workflows with the aim of deploying a system prototype on the SDP test hardware system towards the beginning of 2018. To this aim, we are currently focused on critical services such as the SDP execution control, logging and monitoring (including the Tango interfaces with the Telescope Manager - TM), and prototyping of science workflows which will provide a fairly complete test of the system: visibility ingest from CSP, and two different imaging and calibration pipelines. We then plan to then use the lessons learnt from this first working system to iterate on the design, and develop the prototype further.

SIP picture

Figure 3: SDP Integration Prototype schematic


Housed at Cambridge University, the ALaSKA SDP Performance Prototype Platform has been quick to take advantage of the Big Data Cooperation Agreement signed over the summer with CERN (https://skatelescope.org/news/ska-signs-big-data-cooperation-agreement-cern/). ALaSKA uses OpenStack to deliver a flexible but performant bare metal compute environment to enable SKA project scientists to experiment with and explore software technologies and make objective performance comparisons.

The ALaSKA system uses several OpenStack technologies that are already in full-scale production at CERN.  Conversely, to develop ALaSKA's capability some advanced technologies have been developed by the StackHPC team managing ALaSKA.  The CERN team have identified several areas where ALaSKA's experience can inform the ongoing development of CERN's compute infrastructure.

An informal collaboration has already borne fruit, and that collaboration was jointly presented by Belmiro Moreira from CERN and Stig Telfer from StackHPC at the recent OpenStack summit in Sydney (https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska).

OpenStack SDP P3 code submits

Figure 4: View of code submits into OpenStack coming largely from work within the SDP Performance Platform Prototype work.

OpenStack and Science

Figure 5: Cover of the latest edition of the OpenStack Cloud Computing for Scientific Workloads book.


Related news


Since the last eNews submission in April 2018, the SDP Consortium successfully submitted a documentation pack of roughly 30 documents for its Pre-CDR, M20, milestone to the SKAO. The documentation pack included key Systems Engineering documents, Interface Control Documents (ICDs) and an updated snapshot of the latest SDP Architecture views since the M19 submission and review in November 2017.

In late June, a two-step review process was undertaken between SDP Consortium representatives and the SKAO M20 review panel. The first step was a documentation review process where the objective was to understand and assess the suitability and risks associated with the SDP design before entering the CDR review process. The second step was a face to face meeting which first analysed the suitability of SDP software architecture to meet the needs of its stakeholders, conducted using the SEI ATAM (Architecture Trade-Off Analysis Method) process and based on scenarios generated previously and then discussed observations made against the documentation pack in the first step.


Since the last eNews submission the SDP has successfully submitted and certified its last milestone, M19, and now is busy preparing for the next milestone, M20, Pre-CDR. The M20 Release Readiness Notice (RRN) was submitted in early March and contains 19 documents within the submission pack. This list includes Systems Engineering documents, Interface documentation (ICDs) and an updated snapshot of the latest SDP Architecture views since the M19 submission.

This pre-CDR milestone is a stepping stone to a successful CDR delivery in October.  An incremental submission allows valuable key stakeholder feedback to be gained in particular on the architectural design to ensure continuous improvement and alignment of architectural priorities.

In addition to the finalisation of deliverables for Pre-CDR submission, other areas of focus for the Consortium are - further iterations of the SDP software architecture, advances in the SEI views of the high-level SDP architecture, continued understanding of the SDP interfaces, progress on the SDP functional model using the Algorithmic Reference Library (ARL), progress on the SDP prototyping testbed (P3) in Cambridge, progress on the SDP Integration Prototype (SIP) including end-to-end testing, consolidation of data models work, monitoring of the hardware costs evolution and updates to the corresponding predicted cost to name a few!

The sections below expand in further detail the recent work efforts and progress in the areas of platform and systems integration prototyping (P3, SIP) as well as the ARL. This work reduces risk and provides the rationale for design choices ahead of System CDR.