Home > Publications > Released SDP Memos I

Released SDP Memos I

SDP Memo Disclaimer: The SDP memos are designed to allow the quick recording of investigations and research done by members of the SDP. They are also designed to raise questions about parts of the SDP design or SDP process. The contents of a memo may be the opinion of the author, not the whole of the SDP.

1. SDP Memo 001: ''SKA SDP Memo 001 PAF De-rotation options trade-off: initial SDP response'' (Lead Author: Rosie Bolton):

AttachmentSize
PDF icon ska-tel_sdp-sdpc-memo-001-a.pdf320.06 KB

2. SDP Memo 002: ''SKA SDP Memo 002 Time and channel averaging'' (Lead Author: Chris Skipper):

AttachmentSize
PDF icon ska-tel-sdp-sdpc-memo-002.pdf1.99 MB

3. SDP Memo 003: ''SDP Memo: Fast Fourier Transforms'' (Lead Author: Stef Salvini):

4. SDP Memo 004: ''SDP Memo: Feasibility analysis of baseline-dependent averaging'' (Lead Author: Stefan Wijinholds):

5. SDP Memo 005: ''SDP MEMO: DATA CHALLENGE SUPPLEMENT'' (Lead Author: Markus Dolensky):

6. SDP Memo 006: ''SDP Memo: Receive and Pre-process Visibility Data'' (Lead Author: Ger van Diepen):

7. SDP Memo 007: ''SDP Memo: The SDP Calibration Component'' (Lead Author: Stef Salvini):

8. SDP Memo 008: ''SDP Memo: The SDP Imaging Pipeline'' (Lead Author: Anna Scaife):

9. SDP Memo 009: ''SDP Memo: Science Data Analysis'' (Lead Author: Melanie Johnston-Hollitt):

10. SDP Memo 010: ''SDP Memo: Estimating the SDP Computational Efficiency'' (Lead Author: Bojan Nikolić):

11. SDP Memo 011: ''Discussion Document: SDP Compute Platform Requirements'' (Lead Author: Bojan Nikolić):

AttachmentSize
PDF icon sdp-memo-0006-computeplatform.pdf279.06 KB

12. SDP Memo 012: ''SDP Memo: iPython Performance Model Handbook'' (Lead Author: Francois Malan):

13. SDP Memo 013: ''SDP Memo: Regional Centres'' (Lead Author: Rob Simmonds):

14. SDP Memo 014: ''SDP Memo: PROT.ISP: File-system Benchmark Report'' (Lead Author: Willi Homberg):

15. SDP Memo 015: ''SDP Memo: Can SDP use existing big- data systems?''​ (Lead Author: Rob Simmonds):

16. SDP Memo 016: ''SDP Memo: SDP Non-Imaging Processing Compute Requirements'' (Lead Author: Ben Stappers):

17. SDP Memo 017: ''SDP Memo 017: Dataflow Choice'' (Lead Author: Markus Dolensky):

18. SDP Memo 018: ''SDP Memo: SDP Data Driven Architecture I: Justification''​ (Lead Author: Bojan Nikolić):

19. SDP Memo 019: ''SDP Memo: SDP Load List'' (Lead Author: Ferdl Graser): 

20. SDP Memo 020: ''SDP Memo: Data-driven Architecture Prototyping Report'' (Lead Author: Markus Dolensky): 

21. SDP Memo 021: ''SDP Memo: Dataflow prototyping report'' (Lead Author: Peter Braam):

22. SDP Memo 022: ''SDP Memo: MeerKAT Report'' (Lead Author: Simon Ratcliffe):

23. SDP Memo 023: ''Horizontal Prototyping Interim Report'' (Lead Author: David Terrett):

24. SDP Memo 024: ''SDP Memo: Organisation of the SDP 2016 PDR Architecture Documentation'' (Lead Author: Peter Braam):

25. SDP Memo 025: ''SDP Memo 025: Updated SDP Cost Basis of Estimate June 2016'' (Lead Author: Ferdl Graser):

26. SDP Memo 026: ''The Infeasibility of High Quality Ionospheric Calibration of SKA1-LOW'' (Lead Author: Tim Cornwell):

27. SDP Memo 027: ''SDP Memo: 027 The Infeasibility of High Quality Ionospheric Calibration of SKA1-LOW: response to comments'' (Lead Author: Tim Cornwell):

28. SDP Memo 028: ''SDP Memo 028: Gridding Computational Intensity'' (Lead Author: Peter Wortmann):

AttachmentSize
PDF icon sdp-gridding-computational.pdf1.45 MB

29. SDP Memo 029: ''SDP Memo 29: Integration Prototype Build and Deployment Infrastructure Design'' (Lead Author: Iain Emsley):

30. SDP Memo 030: ''Summary of SDP information relevant to study potential available data-driven systems'' (Lead Author: Montse Farreras)

31. SDP Memo 031: ''Quantifying Power Efficiency of FFTs on NVidia GPUs'' (Lead Author: James Kent):

AttachmentSize
PDF icon powerefficiencysdp.pdf332.17 KB

32. SDP Memo 032: ''On the Precision Required in SDP Pipelines'' (Lead Author: Stef Salvini):

AttachmentSize
PDF icon precision.pdf285.59 KB

33. SDP Memo 033: ''SDP Memo 033: Sky Model Considerations'' (Lead Author: Ian Heywood):

34. SDP Memo 034: ''Practical Distributed Data Processing using SWIFT/T & CASA'' (Lead Author: Bojan Nikolić):

AttachmentSize
PDF icon 2017-08-casaswift.pdf170.26 KB

35. SDP Memo 035: ''SDP Memo FP32 vs. FP64'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_fp32_vs._fp64.pdf148.67 KB

36. SDP Memo 036: ''SDP Memo 36: Convolution Gridding on CPU, GPU and KNL'' (Lead Author: Jacques du Toit):

AttachmentSize
PDF icon sdp_memo_36.pdf4.91 MB

37. SDP Memo 037: ''SDP Memo 037: IO and Storage Software'' (Lead Author: Peter Braam):

38. SDP Memo 038: ''SDP Memo 038: Pipeline Working Sets and Communication'' (Lead Author: Peter Wortmann):

AttachmentSize
PDF icon pipeline-working-sets.pdf416.84 KB

39. SDP Memo 039: ''SDP Memo 039: Full-Scale DALiuGE Data Simulation and Reduction on Tianhe-2'' (Lead Author: Baoqiang Lao):

40. SDP Memo 040: ''SDP Memo 40: PSRFITS Overview for NIP'' (Lead Author: Robert Lyon):

AttachmentSize
PDF icon psrfits_sdp_memo.pdf334.08 KB

41. SDP Memo 041: ''SDP Memo 041: Calibration and imaging context'' (Lead Author: Tim Cornwell):

42. SDP Memo 042: ''SDP Memo 42: Data Model Summary for Pulsar/Transient Search & Timing'' (Lead Author: Robert Lyon):

AttachmentSize
PDF icon sdp_memo_42_lyon_et_al_c2.pdf375.36 KB

43. SDP Memo 043: ''SDP Memo 43: Pulsar Timing Failure Analysis'' (Lead Author: Robert Lyon):

44. SDP Memo 044: ''Technical Use Case Analysis'' (Author: unstated):

AttachmentSize
PDF icon technical_use_case_memo.pdf178.41 KB

45. SDP Memo 045: ''SDP Memo 045: Overview of Buffer Prototyping and Modeling'' (Lead Author: John Taylor):

46. SDP Memo 046: ''SDP Memo 046: Experiences with the SPEAD protocol'' (Lead Author: Rodrigo Tobar):

47. SDP Memo 047: ''SDP Memo 047: Scheduling Schemes and Buffer Considerations''​ (Lead Author: Yongxin Zhu):

48. SDP Memo 048: ''SDP Memo 048: Two Dimensional Sparse Fourier Transform Algorithms'' (Lead Author: Haihang You):

AttachmentSize
PDF icon memo048-2dsfft-ict.pdf1 MB

50. SDP Memo 050: ''SDP Memo 50: The Accelerator Support of Execution Framework'' (Lead Author: Feng Wang):

AttachmentSize
PDF icon sdp_memo-50_wf_signed.pdf561.44 KB

51. SDP Memo 051: ''SDP Memo 051: Cloud Native Applications on the SDP Architecture'' (Lead Author: Piers Harding):

52. SDP Memo 052: ''SDP Memo 052: Apache Kafka for an SDP log based architecture'' (Lead Author: Piers Harding):

53. SDP Memo 053: ''SDP Memo 053: Monitoring and Logging for the SDP'' (Lead Author: Piers Harding):

54. SDP Memo 054: ''SDP Memo 54: Compute Node Pipeline Efficiency Assessment Framework'' (Lead Author: TN Chan):

AttachmentSize
PDF icon sdp_memo_54_compute_efficiency_v1.1.pdf797.74 KB

55. SDP Memo 055: ''SDP Memo 55: Compute Node Hardware Technology Landscape Appraisal'' (Lead Authors: TN Chan and Chris Broekema):

AttachmentSize
PDF icon sdp_memo_55_hardware_landscape_v1.1.pdf973.07 KB

56. SDP Memo 056: ''SDP Memo 56: Hardware Scaling Co-Design Recommendations'' (Lead Author: TN Chan and Chris Broekema):

AttachmentSize
PDF icon sdp_memo_56_scaling_v1.pdf1014.1 KB

61. SDP Memo 061: ''SDP Memo 061: SDP Bandpass Calibration Requirements during EoR Averaging'' (Lead Author: Daniel Mitchell):

62. SDP Memo 062: ''SDP Memo 62: Numerical Precision'' (Lead Author: Anthony Griffin):

AttachmentSize
PDF icon sdp-memo-62_numerical_precision.pdf841.12 KB

65. SDP Memo 065: ''SDP Memo 065: Fast Implementation of SKA Algorithm Reference Library'' (Lead Author: Haihang You):

66. SDP Memo 066: ''SDP Memo 066: Partitioning SKA Dataflows for Optimal Graph Execution'' (Lead Author: Chen Wu):

71. SDP Memo 071: ''SDP Memo 071: LOFAR pipeline DALiuGE'' (Lead Author: Yan Grange):

AttachmentSize
PDF icon sdp_memo_-_daliuge_with_prefactor.pdf596.09 KB

72. SDP Memo 072: ''SDP Memo 72: Vertical prototyping of the gridding algorithm on GPU'' (Lead Author: Anna Brown):

AttachmentSize
PDF icon memo_72_gridding_part_1_-_signed.pdf3.09 MB

73. SDP Memo 073: ''SDP Memo 73: Chebyshev polynomial approximation of kernels in w-projection gridding algorithm on GPU'' (Lead Author: Anna Brown):

AttachmentSize
PDF icon memo_73_cheb_part_1_-_signed.pdf641.91 KB

74. SDP Memo 074: ''SDP Memo 74: Optimisation of the w-projection gridding algorithm for FPGA using Intel OpenCL'' (Lead Author: Anna Brown):

AttachmentSize
PDF icon memo_74_fpga_part_1_-_signed.pdf365.71 KB

77. SDP Memo 077: ''Using DALiuGE for Distributed SAGECal'' (Lead Author: Feng Wang):

AttachmentSize
PDF icon sagecal-memo_wang_2018_signed.pdf1.31 MB

78. SDP Memo 078: ''SDP Memo 078: Scalability Testing using DALiuGE on Tianhe-2 and Pawsey'' (Lead Author: Baoqiang Lao):

80. SDP Memo 080: ''SDP Memo 080: On Scalable Faraday Tomography/Rotation Measure Synthesis for the Square Kilometre Array'' (Lead Author: Stef Salvini):

81. SDP Memo 081: ''SDP Memo 081: Combining Task-Based Parallelism and Platform Services within a Science Pipeline Prototype'' (Lead Author: Jamie Farnes):

83. SDP Memo 083: ''SDP Memo 83: Distribution of the Rau-Cornwell MSMFS algorithm'' (Lead Author: Tim Cornwell):

Recent news

SDP PRE-CDR OAR Status

Since the last eNews submission in April 2018, the SDP Consortium successfully submitted a documentation pack of roughly 30 documents for its Pre-CDR, M20, milestone to the SKAO. The documentation pack included key Systems Engineering documents, Interface Control Documents (ICDs) and an updated snapshot of the latest SDP Architecture views since the M19 submission and review in November 2017.

In late June, a two-step review process was undertaken between SDP Consortium representatives and the SKAO M20 review panel. The first step was a documentation review process where the objective was to understand and assess the suitability and risks associated with the SDP design before entering the CDR review process. The second step was a face to face meeting which first analysed the suitability of SDP software architecture to meet the needs of its stakeholders, conducted using the SEI ATAM (Architecture Trade-Off Analysis Method) process and based on scenarios generated previously and then discussed observations made against the documentation pack in the first step.

SIP ARL

Since the last eNews submission the SDP has successfully submitted and certified its last milestone, M19, and now is busy preparing for the next milestone, M20, Pre-CDR. The M20 Release Readiness Notice (RRN) was submitted in early March and contains 19 documents within the submission pack. This list includes Systems Engineering documents, Interface documentation (ICDs) and an updated snapshot of the latest SDP Architecture views since the M19 submission.

This pre-CDR milestone is a stepping stone to a successful CDR delivery in October.  An incremental submission allows valuable key stakeholder feedback to be gained in particular on the architectural design to ensure continuous improvement and alignment of architectural priorities.

In addition to the finalisation of deliverables for Pre-CDR submission, other areas of focus for the Consortium are - further iterations of the SDP software architecture, advances in the SEI views of the high-level SDP architecture, continued understanding of the SDP interfaces, progress on the SDP functional model using the Algorithmic Reference Library (ARL), progress on the SDP prototyping testbed (P3) in Cambridge, progress on the SDP Integration Prototype (SIP) including end-to-end testing, consolidation of data models work, monitoring of the hardware costs evolution and updates to the corresponding predicted cost to name a few!

The sections below expand in further detail the recent work efforts and progress in the areas of platform and systems integration prototyping (P3, SIP) as well as the ARL. This work reduces risk and provides the rationale for design choices ahead of System CDR.