Home > News > SDP eNews - December 2017

SDP eNews - December 2017

SDP Consortium Progress since last eNews submission

M19 - All things Architecture

The next significant milestone for the SDP Consortium is M19 on the 30th November. M19 is a lightweight review of the current state of the SDP software architecture and forms a checkpoint with the SKAO to present the current state of the architecture and progress towards Critical Design Review (CDR).

All relevant SDP SEI style views and other supporting documentation for functionally close blocks of architecture will be reviewed. The goal is to present the information in sufficient detail to facilitate immediate use by a knowledgeable audience member.

In preparation for this review, a Stakeholders Overview will take place. The required participants will include key members of the SDP architectural design team and knowledgeable stakeholders from SKAO. This review is to ensure that the high-level architecture documentation, for the purposes of providing an introductory, high-level explanation of the design, is sufficiently complete. In preparation for this overview, key members of the SDP Architecture design team recently selected a collection of views for review and this information has been distributed to the SKAO.

Comments and or suggestions made during the Stakeholder Overview and the official M19 review will guide further architectural priorities ahead of M20 (Element pre-CDR).

SDP Architecture doc

Figure 1 - collection of SDP Architecture views ready for review with SKAO

SC17 – Denver, Colorado

SC17 picture
SC17 Phil and Rosie Banner

SuperComputing 2017 (SC17) was held in Denver, Colorado, during mid-November. It is the last such conference to be held before the SDP CDR. The keynote presentation of the conference was given by the SKA Director General, Professor Phil Diamond, and SKA Regional Centre and SDP Project Scientist, Dr Rosie Bolton. They explained the background and relevance of radio astronomy, the aspirations and developments of the SKA and in particular the computing challenges it faces.

The timing of the conference and the topic of the keynote was ideal as the SKA consortia CDRs begin in the coming months and the organisation gears up for construction. The conference provided opportunities for attending SDP members to test industry views on SKA, such as the desire to use Scaled Agile approaches for software construction (especially SAFe), gain insight into vendor roadmaps and where feasible pursue collaborative work on applicable prototyping activities.

SC 17 SKA Keynote

Keynoters Philip Diamond, Director General of SKA, and Rosie Bolton, SKA Regional Centre Project Scientist and SDP Project Scientist on stage at SC17.

The conference also provided an opportunity for SDP Consortium members to meet and get feedback from leading industry experts in Birds of a Feather (BOF) sessions related to SKA computing, and contribute to workshops on topics of direct interest to the community.

SDP Construction Planning

Since the last eNews submission significant progress has been made on the SDP Construction Plan and its supporting documents. This progress consists of:

  • Completion of the Rollout Plan that encapsulates the schedule for Construction.

  • Formulation of the framework and direction of the Construction Plan to ensure alignment with the SAFe framework.

  • Completion of an initial high level Construction Plan based on the framework described above.

  • Completion of an initial Work Breakdown Structure (WBS) in accordance with SAFe ideologies. The Product Breakdown Structure is not affected at this stage.

  • Delivery of a webcast to SDP Consortium members to present the current status and key points of the Construction Plan including the Rollout roadmap (refer to figure below). This was well received with additional discussion points raised for further investigation and consideration.

SDP Construction Roadmap - WIP

Figure 2 – SDP Construction Schedule and Roadmap. The SDP Commissioning needs to be in-step with other aspects of the telescope which is provided by an Assembly Integration and Verification (AIV) system.

Significant progress has been made, but the Construction Plan remains a work in progress. The final version will be ready for April 2018 ahead of the SKA System Review.

SIP – SDP Integration Prototype

The SDP Integration Prototype (SIP) is an active project to produce a lightweight horizontal prototype for the SDP system, providing a verification of the SDP architecture, and a test of all major internal and external SDP interfaces. SIP is under active development by a small distributed Agile team, working closely with the SDP architecture design group and the SDP hardware prototype, an OpenStack software-defined bare metal private cloud.

To take a practical approach towards delivering an initial working system, we have chosen to develop SDP components as a set of independent deployable (and testable) microservices, which make use of container orchestration (currently utilising Docker Swarm). These provide a modular, highly available, and horizontally scalable deployment of the SDP domain specific and platform services. In turn, this set of services support the core functions of the SDP, namely, the real-time and batch science pipeline workflows, and data ingest from Central Signal Processor (CSP).

The SIP team is currently at work developing a minimal set of services and a select number of science pipelines and data ingest workflows with the aim of deploying a system prototype on the SDP test hardware system towards the beginning of 2018. To this aim, we are currently focused on critical services such as the SDP execution control, logging and monitoring (including the Tango interfaces with the Telescope Manager - TM), and prototyping of science workflows which will provide a fairly complete test of the system: visibility ingest from CSP, and two different imaging and calibration pipelines. We then plan to then use the lessons learnt from this first working system to iterate on the design, and develop the prototype further.

SIP picture

Figure 3: SDP Integration Prototype schematic


Housed at Cambridge University, the ALaSKA SDP Performance Prototype Platform has been quick to take advantage of the Big Data Cooperation Agreement signed over the summer with CERN (https://skatelescope.org/news/ska-signs-big-data-cooperation-agreement-cern/). ALaSKA uses OpenStack to deliver a flexible but performant bare metal compute environment to enable SKA project scientists to experiment with and explore software technologies and make objective performance comparisons.

The ALaSKA system uses several OpenStack technologies that are already in full-scale production at CERN.  Conversely, to develop ALaSKA's capability some advanced technologies have been developed by the StackHPC team managing ALaSKA.  The CERN team have identified several areas where ALaSKA's experience can inform the ongoing development of CERN's compute infrastructure.

An informal collaboration has already borne fruit, and that collaboration was jointly presented by Belmiro Moreira from CERN and Stig Telfer from StackHPC at the recent OpenStack summit in Sydney (https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska).

OpenStack SDP P3 code submits

Figure 4: View of code submits into OpenStack coming largely from work within the SDP Performance Platform Prototype work.

OpenStack and Science

Figure 5: Cover of the latest edition of the OpenStack Cloud Computing for Scientific Workloads book.


Related news

Lovell telescope behind new SKA HQ building on 17 Jan 2019

At the end of October 2018 the SKA SDP Consortium submitted its Critical Design Review (CDR) documentation pack. Contained in this were the formal deliverables of the design consortium covering all aspects of the SDP architecture, system engineering and programmatics (the documents are available here).  The documents were received by the CDR reviewers (consisting of 3 external members and 15 internal to SKAO) who proceeded to generate clarification questions, requests and notes in the form of observations (called OARs after the Observation Action Register approach commonly used for them). Where possible these were addressed via communication exchanges in JIRA OAR tickets.

From 15th to 19th January 2019 members of the SDP Consortium then visited the SKAO HQ at Jodrell Bank (see figure 1), where direct discussions took place in the new SKAO Council Chamber (see figure 2), to further explore the architecture and identify risks as part of an SEI Architecture Tradeoff Analysis Method (ATAM).


At the end of October the SDP Consortium submitted its full document set for Critical Design Review. (These can be found at http://ska-sdp.org/publications/sdp-cdr-documentation) together with a large number of supporting memos (http://ska-sdp.org/publications/released-sdp-memos-i and http://ska-sdp.org/publications/released-sdp-memos-ii). Table 1 below shows the documents in three main categories: those associated with software and hardware architecture; those explaining the supporting prototyping work that has been undertaken in support of the architecture, and finally those associated with system engineering (SE) and programmatics aspects (e.g. specifications for how SDP interfaces with the wider telescope systems and how components will be constructed). The documents will receive observations from a panel of reviewers up until the end of December. Responses to the observations and scenarios to ‘test’ the architecture will then be discussed at a review meeting from 15th to 18th January 2019 at Jodrell Bank.