At the end of October the SDP Consortium submitted its full document set for Critical Design Review. (These can be found at http://ska-sdp.org/publications/sdp-cdr-documentation) together with a large number of supporting memos (http://ska-sdp.org/publications/released-sdp-memos-i and http://ska-sdp.org/publications/released-sdp-memos-ii). Table 1 below shows the documents in three main categories: those associated with software and hardware architecture; those explaining the supporting prototyping work that has been undertaken in support of the architecture, and finally those associated with system engineering (SE) and programmatics aspects (e.g. specifications for how SDP interfaces with the wider telescope systems and how components will be constructed). The documents will receive observations from a panel of reviewers up until the end of December. Responses to the observations and scenarios to ‘test’ the architecture will then be discussed at a review meeting from 15th to 18th January 2019 at Jodrell Bank.
Table 1: Documents, by category, which have been submitted for the Critical Design Review of the SDP Consortium
SDP core prototyping activities
The SDP architecture has been guided by a lot of prototyping work, and the underlying activities written up into a series of “prototyping reports” with further results and conclusions given in SDP memos. To provide more context and allow the SKAO review panel members to directly learn about and talk through aspects of the code and results, a handful of staff from SKAO visited Cambridge on 22nd and 23rd November for a series of informal presentations and code walkthroughs with the developers (see Figure 1).
Figure 1: Participants of the SDP Informal Demo Days held at The University of Cambridge on 22nd and 23rd November 2018
The main areas included for demonstration and discussion were:
- The Architectural Context of the SDP prototyping work;
- Execution frameworks (ICAL in DASK, MPI and StarPU and SageCal in DALiuGE);
- Vertical prototyping and algorithm efficiency considerations;
- The Algorithm Reference Library (used to provide a reference for functions);
- The Performance Platform Prototype (P3) and
- The System Integration Prototype.
The mapping of the work shown against the SDP architecture can be seen in Figure 2 for services and Figure 3 for processing layers.
Figure 2: A slide from the context discussions showing aspects of the SDP architecture prototyped with “service prototypes”.
Figure 3: A slide from the context discussions showing aspects of the SDP architecture prototyped for the “processing layers”.
The two days involved useful clarifications on what has been achieved thus far and discussions about how the work might best be taken forward in bridging activities (see next section). Figure 4 shows a screenshot taken from the overview of functionality enabled in the Integration Prototype for the SDP Master Device.
Figure 4: A snapshot from the System Integration Prototype work showing working interactions of the SDP Master Device.
SDP ongoing activities
For a year and a half now SDP work has been undertaken in a series of roughly 8 week work sprints. In early November the Consortium management and task leads came together for several days of sprint planning (for sprint 2018F which runs until mid-January) to think about key activities to take forward before the SDP Consortium formally dissolves after CDR close-out in March 2019. Naturally CDR related activities topped the tasking including:
- Responding to review panel observations on the SDP documentation set;
- Preparing for and participating in the SDP informal demonstrations meeting.
However there were some other clear objectives for this period including:
- Putting forward a system-level software architecture team to work in the current SAFe structure;
- Determining the requirements for and building a data island scale demonstration with a proper data model (ideally for CDR);
- Pushing forward the DASK/ARL execution framework work to further demonstrate scaling aspects;
- Distill further work from the internal SDP confluence based wiki (to preserve it beyond the end of the Consortium);
- Refactor the SDP Integration Prototype code (to bring it closer in structure to the latest architecture view) and further develop control components.
In 2019 there will be one or two additional sprints with focus on addressing issues raised during the CDR process and closing out the SDP project (including reviewing and dealing with a large number of backlog work items that have built up in the project JIRA tasking system).
Beyond the SDP Consortium
The SDP Consortium was created to generate a design for the SKA Science Data Processor. With that design now submitted and under review the focus is starting to shift. In particular there is additional clarification work to be done before the SKA System CDR in 2019, and especially before construction starts. To define and allocate this work a series of so called bridging tasks are being generated by the SKA organisation in conjunction with the sub-element consortia. Once the SDP CDR is completed it is hoped that most of the SDP contributing institutes and individuals will receive further funding to work under a new SKAO led work programme. For software areas this includes building up a working approach for construction using the Scaled Agile Framework (SAFe). The bridging activities may include looking at minimum viable products that can be built upon in construction and further studies, ideally with industry engagement, to better understand aspects of the SDP (such as efficiency) and of course refine the architecture. SDP related queries beyond March 2019 should be directed to the SKAO in the first instance.
