SuperComputing 2017 (SC17) was held in Denver, Colorado, during mid-November. It is the last such conference to be held before the SDP CDR. The keynote presentation of the conference was given by the SKA Director General, Professor Phil Diamond, and SKA Regional Centre and SDP Project Scientist, Dr Rosie Bolton. They explained the background and relevance of radio astronomy, the aspirations and developments of the SKA and in particular the computing challenges it faces.

The timing of the conference and the topic of the keynote was ideal as the SKA consortia CDRs begin in the coming months and the organisation gears up for construction. The conference provided opportunities for attending SDP members to test industry views on SKA, such as the desire to use Scaled Agile approaches for software construction (especially SAFe), gain insight into vendor roadmaps and where feasible pursue collaborative work on applicable prototyping activities.

Keynoters Philip Diamond, Director General of SKA, and Rosie Bolton, SKA Regional Centre Project Scientist and SDP Project Scientist on stage at SC17.

The conference also provided an opportunity for SDP Consortium members to meet and get feedback from leading industry experts in Birds of a Feather (BOF) sessions related to SKA computing, and contribute to workshops on topics of direct interest to the community.

SDP Construction Planning

Since the last eNews submission significant progress has been made on the SDP Construction Plan and its supporting documents. This progress consists of:

• Completion of the Rollout Plan that encapsulates the schedule for Construction.

• Formulation of the framework and direction of the Construction Plan to ensure alignment with the SAFe framework.

• Completion of an initial high level Construction Plan based on the framework described above.

• Completion of an initial Work Breakdown Structure (WBS) in accordance with SAFe ideologies. The Product Breakdown Structure is not affected at this stage.

• Delivery of a webcast to SDP Consortium members to present the current status and key points of the Construction Plan including the Rollout roadmap (refer to figure below). This was well received with additional discussion points raised for further investigation and consideration.

Figure 2 – SDP Construction Schedule and Roadmap. The SDP Commissioning needs to be in-step with other aspects of the telescope which is provided by an Assembly Integration and Verification (AIV) system.

Significant progress has been made, but the Construction Plan remains a work in progress. The final version will be ready for April 2018 ahead of the SKA System Review.

SIP – SDP Integration Prototype

The SDP Integration Prototype (SIP) is an active project to produce a lightweight horizontal prototype for the SDP system, providing a verification of the SDP architecture, and a test of all major internal and external SDP interfaces. SIP is under active development by a small distributed Agile team, working closely with the SDP architecture design group and the SDP hardware prototype, an OpenStack software-defined bare metal private cloud.

To take a practical approach towards delivering an initial working system, we have chosen to develop SDP components as a set of independent deployable (and testable) microservices, which make use of container orchestration (currently utilising Docker Swarm). These provide a modular, highly available, and horizontally scalable deployment of the SDP domain specific and platform services. In turn, this set of services support the core functions of the SDP, namely, the real-time and batch science pipeline workflows, and data ingest from Central Signal Processor (CSP).

The SIP team is currently at work developing a minimal set of services and a select number of science pipelines and data ingest workflows with the aim of deploying a system prototype on the SDP test hardware system towards the beginning of 2018. To this aim, we are currently focused on critical services such as the SDP execution control, logging and monitoring (including the Tango interfaces with the Telescope Manager - TM), and prototyping of science workflows which will provide a fairly complete test of the system: visibility ingest from CSP, and two different imaging and calibration pipelines. We then plan to then use the lessons learnt from this first working system to iterate on the design, and develop the prototype further.

Figure 3: SDP Integration Prototype schematic

OpenStack

Housed at Cambridge University, the ALaSKA SDP Performance Prototype Platform has been quick to take advantage of the Big Data Cooperation Agreement signed over the summer with CERN (https://skatelescope.org/news/ska-signs-big-data-cooperation-agreement-cern/). ALaSKA uses OpenStack to deliver a flexible but performant bare metal compute environment to enable SKA project scientists to experiment with and explore software technologies and make objective performance comparisons.

The ALaSKA system uses several OpenStack technologies that are already in full-scale production at CERN.  Conversely, to develop ALaSKA's capability some advanced technologies have been developed by the StackHPC team managing ALaSKA.  The CERN team have identified several areas where ALaSKA's experience can inform the ongoing development of CERN's compute infrastructure.

An informal collaboration has already borne fruit, and that collaboration was jointly presented by Belmiro Moreira from CERN and Stig Telfer from StackHPC at the recent OpenStack summit in Sydney (https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska).

Figure 4: View of code submits into OpenStack coming largely from work within the SDP Performance Platform Prototype work.

Figure 5: Cover of the latest edition of the OpenStack Cloud Computing for Scientific Workloads book.