SDP Consortium Progress since last eNews submission
M19 - All things Architecture
The next significant milestone for the SDP Consortium is M19 on the 30th November. M19 is a lightweight review of the current state of the SDP software architecture and forms a checkpoint with the SKAO to present the current state of the architecture and progress towards Critical Design Review (CDR).
All relevant SDP SEI style views and other supporting documentation for functionally close blocks of architecture will be reviewed. The goal is to present the information in sufficient detail to facilitate immediate use by a knowledgeable audience member.
In preparation for this review, a Stakeholders Overview will take place. The required participants will include key members of the SDP architectural design team and knowledgeable stakeholders from SKAO. This review is to ensure that the high-level architecture documentation, for the purposes of providing an introductory, high-level explanation of the design, is sufficiently complete. In preparation for this overview, key members of the SDP Architecture design team recently selected a collection of views for review and this information has been distributed to the SKAO.
Comments and or suggestions made during the Stakeholder Overview and the official M19 review will guide further architectural priorities ahead of M20 (Element pre-CDR).
Figure 1 - collection of SDP Architecture views ready for review with SKAO
SC17 – Denver, Colorado
SuperComputing 2017 (SC17) was held in Denver, Colorado, during mid-November. It is the last such conference to be held before the SDP CDR. The keynote presentation of the conference was given by the SKA Director General, Professor Phil Diamond, and SKA Regional Centre and SDP Project Scientist, Dr Rosie Bolton. They explained the background and relevance of radio astronomy, the aspirations and developments of the SKA and in particular the computing challenges it faces.
The timing of the conference and the topic of the keynote was ideal as the SKA consortia CDRs begin in the coming months and the organisation gears up for construction. The conference provided opportunities for attending SDP members to test industry views on SKA, such as the desire to use Scaled Agile approaches for software construction (especially SAFe), gain insight into vendor roadmaps and where feasible pursue collaborative work on applicable prototyping activities.
Keynoters Philip Diamond, Director General of SKA, and Rosie Bolton, SKA Regional Centre Project Scientist and SDP Project Scientist on stage at SC17.
The conference also provided an opportunity for SDP Consortium members to meet and get feedback from leading industry experts in Birds of a Feather (BOF) sessions related to SKA computing, and contribute to workshops on topics of direct interest to the community.
SDP Construction Planning
Since the last eNews submission significant progress has been made on the SDP Construction Plan and its supporting documents. This progress consists of:
Completion of the Rollout Plan that encapsulates the schedule for Construction.
Formulation of the framework and direction of the Construction Plan to ensure alignment with the SAFe framework.
Completion of an initial high level Construction Plan based on the framework described above.
Completion of an initial Work Breakdown Structure (WBS) in accordance with SAFe ideologies. The Product Breakdown Structure is not affected at this stage.
Delivery of a webcast to SDP Consortium members to present the current status and key points of the Construction Plan including the Rollout roadmap (refer to figure below). This was well received with additional discussion points raised for further investigation and consideration.
Figure 2 – SDP Construction Schedule and Roadmap. The SDP Commissioning needs to be in-step with other aspects of the telescope which is provided by an Assembly Integration and Verification (AIV) system.
Significant progress has been made, but the Construction Plan remains a work in progress. The final version will be ready for April 2018 ahead of the SKA System Review.
SIP – SDP Integration Prototype
The SDP Integration Prototype (SIP) is an active project to produce a lightweight horizontal prototype for the SDP system, providing a verification of the SDP architecture, and a test of all major internal and external SDP interfaces. SIP is under active development by a small distributed Agile team, working closely with the SDP architecture design group and the SDP hardware prototype, an OpenStack software-defined bare metal private cloud.
To take a practical approach towards delivering an initial working system, we have chosen to develop SDP components as a set of independent deployable (and testable) microservices, which make use of container orchestration (currently utilising Docker Swarm). These provide a modular, highly available, and horizontally scalable deployment of the SDP domain specific and platform services. In turn, this set of services support the core functions of the SDP, namely, the real-time and batch science pipeline workflows, and data ingest from Central Signal Processor (CSP).
The SIP team is currently at work developing a minimal set of services and a select number of science pipelines and data ingest workflows with the aim of deploying a system prototype on the SDP test hardware system towards the beginning of 2018. To this aim, we are currently focused on critical services such as the SDP execution control, logging and monitoring (including the Tango interfaces with the Telescope Manager - TM), and prototyping of science workflows which will provide a fairly complete test of the system: visibility ingest from CSP, and two different imaging and calibration pipelines. We then plan to then use the lessons learnt from this first working system to iterate on the design, and develop the prototype further.
Figure 3: SDP Integration Prototype schematic
Housed at Cambridge University, the ALaSKA SDP Performance Prototype Platform has been quick to take advantage of the Big Data Cooperation Agreement signed over the summer with CERN (https://skatelescope.org/news/ska-signs-big-data-cooperation-agreement-cern/). ALaSKA uses OpenStack to deliver a flexible but performant bare metal compute environment to enable SKA project scientists to experiment with and explore software technologies and make objective performance comparisons.
The ALaSKA system uses several OpenStack technologies that are already in full-scale production at CERN. Conversely, to develop ALaSKA's capability some advanced technologies have been developed by the StackHPC team managing ALaSKA. The CERN team have identified several areas where ALaSKA's experience can inform the ongoing development of CERN's compute infrastructure.
An informal collaboration has already borne fruit, and that collaboration was jointly presented by Belmiro Moreira from CERN and Stig Telfer from StackHPC at the recent OpenStack summit in Sydney (https://www.openstack.org/videos/sydney-2017/future-science-on-future-openstack-developing-next-generation-infrastructure-at-cern-and-ska).
Figure 4: View of code submits into OpenStack coming largely from work within the SDP Performance Platform Prototype work.
Figure 5: Cover of the latest edition of the OpenStack Cloud Computing for Scientific Workloads book.