Home > Publications > Released SDP Memos 1

Released SDP Memos 1

SDP Memo Disclaimer: The SDP memos are designed to allow the quick recording of investigations and research done by members of the SDP. They are also designed to raise questions about parts of the SDP design or SDP process. The contents of a memo may be the opinion of the author, not the whole of the SDP.

1. SDP Memo 001: ''SKA SDP Memo 001 PAF De-rotation options trade-off: initial SDP response'' (Lead Author: Rosie Bolton):

AttachmentSize
PDF icon ska-tel_sdp-sdpc-memo-001-a.pdf320.06 KB

2. SDP Memo 002: ''SKA SDP Memo 002 Time and channel averaging'' (Lead Author: Chris Skipper):

AttachmentSize
PDF icon ska-tel-sdp-sdpc-memo-002.pdf1.99 MB

3. SDP Memo 003: ''SDP Memo: Fast Fourier Transforms'' (Lead Author: Stef Salvini):

AttachmentSize
PDF icon ska-tel-sdp-0000058_02c_rep_sdpmemofastfouriertransforms_-_signed.pdf905.52 KB

4. SDP Memo 004: ''SDP Memo: Feasibility analysis of baseline-dependent averaging'' (Lead Author: Stefan Wijinholds):

AttachmentSize
PDF icon ska-tel-sdp-0000017_01c_rep_sdpmemobaselinedependentaveraging_-_signed.pdf4.88 MB

5. SDP Memo 005: ''SDP MEMO: DATA CHALLENGE SUPPLEMENT'' (Lead Author: Markus Dolensky):

AttachmentSize
PDF icon ska-tel-sdp-0000024_02c_rep_sdpmemodatachallengesupplement_-_signed.pdf4.17 MB

6. SDP Memo 006: ''SDP Memo: Receive and Pre-process Visibility Data'' (Lead Author: Ger van Diepen):

AttachmentSize
PDF icon ska-tel-sdp-0000028_01c_rep_sdpmemoingestpipeline_-_signed.pdf726.31 KB

7. SDP Memo 007: ''SDP Memo: The SDP Calibration Component'' (Lead Author: Stef Salvini):

AttachmentSize
PDF icon ska-tel-sdp-0000029_01c_rep_sdpmemocalibration_-_signed.pdf810.79 KB

8. SDP Memo 008: ''SDP Memo: The SDP Imaging Pipeline'' (Lead Author: Anna Scaife):

AttachmentSize
PDF icon ska-tel-sdp-0000030_02c_rep_sdpmemoimagingpipeline_-_signed.pdf3.39 MB

9. SDP Memo 009: ''SDP Memo: Science Data Analysis'' (Lead Author: Melanie Johnston-Hollitt):

AttachmentSize
PDF icon ska-tel-sdp-0000031_02c_rep_sdpmemoscienceanalysispipeline_-_signed.pdf1.38 MB

10. SDP Memo 010: ''SDP Memo: Estimating the SDP Computational Efficiency'' (Lead Author: Bojan Nikolic):

AttachmentSize
PDF icon ska-tel-sdp-0000086_c_rep_sdpmemoefficiencyprototyping_-_signed.pdf438.38 KB

12. SDP Memo 012: ''SDP Memo: iPython Performance Model Handbook'' (Lead Author: Francois Malan):

AttachmentSize
PDF icon ska-tel-sdp-0000041_01c_mnl_sdpmemoipythonmodelhandbook_-_signed.pdf919.53 KB

13. SDP Memo 013: ''SDP Memo: Regional Centres'' (Lead Author: Rob Simmonds):

AttachmentSize
PDF icon ska-tel-sdp-0000060_01c_rep_sdpmemoregionalcentres_-_signed.pdf369.95 KB

14. SDP Memo 014: ''SDP Memo: PROT.ISP: File-system Benchmark Report'' (Lead Author: Willi Homberg):

AttachmentSize
PDF icon ska-tel-sdp-0000061_01c_rep_sdpmemoispreport_-_signed.pdf845.92 KB

15. SDP Memo 015: ''SDP Memo: Can SDP use existing big- data systems?''​ (Lead Author: Rob Simmonds):

AttachmentSize
PDF icon ska-tel-sdp-0000072_01c_rep_sdpmemobigdatasystems_-_signed.pdf495.91 KB

16. SDP Memo 016: ''SDP Memo: SDP Non-Imaging Processing Compute Requirements'' (Lead Author: Ben Stappers):

AttachmentSize
PDF icon ska-tel-sdp-0000080_c_rep_memosdpnonimagingpipeline_-_signed.pdf444.53 KB

17. SDP Memo 017: ''SDP Memo 017: Dataflow Choice'' (Lead Author: Markus Dolensky):

AttachmentSize
PDF icon ska-tel-sdp-0000066_dataflow_choice_20150521.pdf406.9 KB

18. SDP Memo 018: ''SDP Memo: SDP Data Driven Architecture I: Justification''​ (Lead Author: Bojan Nikolic):

AttachmentSize
PDF icon ska-tel-sdp-0000082_c_rep_sdpmemodatadrivenprototyping_-_signed.pdf1.37 MB

19. SDP Memo 019: ''SDP Memo: SDP Load List'' (Lead Author: Ferdl Graser): 

AttachmentSize
PDF icon ska-tel-sdp-0000078_c_rep_memosdploadlist_part_1_-_signed.pdf588.04 KB

20. SDP Memo 020: ''SDP Memo: Data-driven Architecture Prototyping Report'' (Lead Author: Markus Dolensky): 

AttachmentSize
PDF icon ska-tel-sdp-0000082_c_rep_sdpmemodatadrivenprototyping_-_signed.pdf1.37 MB

21. SDP Memo 021: ''SDP Memo: Dataflow prototyping report'' (Lead Author: Peter Braam):

AttachmentSize
PDF icon ska-tel-sdp-0000083_c_rep_sdpmemodataflowprototyping_-_signed.pdf4.53 MB

22. SDP Memo 022: ''SDP Memo: MeerKAT Report'' (Lead Author: Simon Ratcliffe):

AttachmentSize
PDF icon ska-tel-sdp-0000084_c_rep_sdpmemomeerkatdataflow_-_signed.pdf1.38 MB

23. SDP Memo 023: ''Horizontal Prototyping Interim Report'' (Lead Author: David Terrett):

AttachmentSize
PDF icon ska-tel-sdp-0000085_c_rep_sdpmemohorizontalprototyping_-_signed.pdf436.1 KB

24. SDP Memo 024: ''SDP Memo: Organisation of the SDP 2016 PDR Architecture Documentation'' (Lead Author: Peter Braam):

AttachmentSize
PDF icon ska-tel-sdp-0000087_c_rep_sdpmemoorganisationofsdparchitecture_part_1_-_signed.pdf427.81 KB

25. SDP Memo 025: ''SDP Memo 025: Updated SDP Cost Basis of Estimate June 2016'' (Lead Author: Ferdl Graser):

AttachmentSize
PDF icon ska-tel-sdp-0000091_c_rep_updatedsdpcostbasisofestimatejune2016.pdf794.96 KB

26. SDP Memo 026: ''The Infeasibility of High Quality Ionospheric Calibration of SKA1-LOW'' (Lead Author: Tim Cornwell):

AttachmentSize
PDF icon the_infeasibility_of_high_quality_ionospheric_calibration_of_ska1_-_reponse_to_comments_part_1_-_signed.pdf1.05 MB

27. SDP Memo 027: ''SDP Memo: 027 The Infeasibility of High Quality Ionospheric Calibration of SKA1-LOW: response to comments'' (Lead Author: Tim Cornwell):

AttachmentSize
PDF icon the_infeasibility_of_high_quality_ionospheric_calibration_of_ska1_-_reponse_to_comments_part_1_-_signed.pdf1.05 MB

28. SDP Memo 028: ''SDP Memo 028: Gridding Computational Intensity'' (Lead Author: Peter Wortmann):

AttachmentSize
PDF icon sdp-gridding-computational_part_1_-_signed.pdf1.49 MB

29. SDP Memo 029: ''SDP Memo 29: Integration Prototype Build and Deployment Infrastructure Design'' (Lead Author: Iain Emsley):

AttachmentSize
PDF icon sdpmemo29-integrationprototypebuildanddeployment_part_1_-_signed.pdf367.02 KB

30. SDP Memo 030: "Apache Storm-based SDP Pipeline" (Authors: Robert Lyon, Lina Levin Preston and Ben Stappers):

AttachmentSize
PDF icon sdp_memo_strom_prototype.pdf613.42 KB

31. SDP Memo 031: ''Quantifying Power Efficiency of FFTs on NVidia GPUs'' (Lead Author: James Kent):

AttachmentSize
PDF icon powerefficiencysdp_part_1_-_signed.pdf370.9 KB

32. SDP Memo 032: ''On the Precision Required in SDP Pipelines'' (Lead Author: Stef Salvini):

AttachmentSize
PDF icon precision_0_part_1_-_signed.pdf330.45 KB

33. SDP Memo 033: ''SDP Memo 033: Sky Model Considerations'' (Lead Author: Ian Heywood):

AttachmentSize
PDF icon sdp_memo_033_sky_model_draft_161017_part_1_-_signed.pdf302.09 KB

34. SDP Memo 034: ''Practical Distributed Data Processing using SWIFT/T & CASA'' (Lead Author: Bojan Nikolic):

AttachmentSize
PDF icon 2017-08-casaswift_part_1_-_signed.pdf207.7 KB

35. SDP Memo 035: ''SDP Memo FP32 vs. FP64'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_fp32_vs._fp64_part_1_-_signed.pdf188.82 KB

36. SDP Memo 036: ''SDP Memo 36: Convolution Gridding on CPU, GPU and KNL'' (Lead Author: Jacques du Toit):

AttachmentSize
PDF icon sdp_memo_36.pdf4.91 MB

37. SDP Memo 037: ''SDP Memo 037: IO and Storage Software'' (Lead Author: Peter Braam):

AttachmentSize
PDF icon 2017-12_sdp_io_and_storage_software_part_1_-_signed.pdf409.85 KB

38. SDP Memo 038: ''SDP Memo 038: Pipeline Working Sets and Communication'' (Lead Author: Peter Wortmann):

AttachmentSize
PDF icon pipeline-working-sets_part_1_-_signed.pdf459.1 KB

39. SDP Memo 039: ''SDP Memo 039: Full-Scale DALiuGE Data Simulation and Reduction on Tianhe-2'' (Lead Author: Baoqiang Lao):

AttachmentSize
PDF icon sdp_memo_-_daliuge_scalabilty_test_on_tianhe2_part_1_-_signed.pdf1.3 MB

40. SDP Memo 040: ''SDP Memo 40: PSRFITS Overview for NIP'' (Lead Author: Robert Lyon):

AttachmentSize
PDF icon psrfits_sdp_memo_part_1_-_signed.pdf364.58 KB

41. SDP Memo 041: ''SDP Memo 041: Calibration and imaging context'' (Lead Author: Tim Cornwell):

AttachmentSize
PDF icon sdp_memo_041_calibration_and_imaging_context.pdf383.6 KB

42. SDP Memo 042: ''SDP Memo 42: Data Model Summary for Pulsar/Transient Search & Timing'' (Lead Author: Robert Lyon):

AttachmentSize
PDF icon sdp_memo_42_lyon_et_al_c2.pdf375.36 KB

43. SDP Memo 043: ''SDP Memo 43: Pulsar Timing Failure Analysis'' (Lead Author: Robert Lyon):

AttachmentSize
PDF icon nipsoftwarefailureanalysis_timing_c1.pdf522.55 KB

44. SDP Memo 044: ''Technical Use Case Analysis'' (Author: unstated):

AttachmentSize
PDF icon technical_use_case_memo.pdf178.41 KB

45. SDP Memo 045: ''SDP Memo 045: Overview of Buffer Prototyping and Modeling'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_045_overview_of_buffer_prototyping_part_1_-_signed.pdf1.44 MB

46. SDP Memo 046: ''SDP Memo 046: Experiences with the SPEAD protocol'' (Lead Author: Rodrigo Tobar):

AttachmentSize
PDF icon sdp_memo_046_experiences_with_the_spead_protocol.pdf648.76 KB

47. SDP Memo 047: ''SDP Memo 047: Scheduling Schemes and Buffer Considerations''​ (Lead Author: Yongxin Zhu):

AttachmentSize
PDF icon sdp_memo_scheduling_schemes_v1.0_-_signed.pdf1.19 MB

48. SDP Memo 048: ''SDP Memo 048: Two Dimensional Sparse Fourier Transform Algorithms''​ (Lead Author: Haihang You):

AttachmentSize
PDF icon memo048-2dsfft-ict.pdf1 MB

49. SDP Memo 049: ''SDP Memo 049: SDP limitations on wide-area mapping mode'' (Lead Author: Mark Ashdown):

AttachmentSize
PDF icon sdp_memo_049_sdp_limitations_on_wide_area_mapping_mode_part_1_-_signed.pdf290.2 KB

50. SDP Memo 050: ''SDP Memo 50: The Accelerator Support of Execution Framework'' (Lead Author: Feng Wang):

AttachmentSize
PDF icon sdp_memo-50_wf_signed.pdf561.44 KB

51. SDP Memo 051: ''SDP Memo 051: Cloud Native Applications on the SDP Architecture'' (Lead Author: Piers Harding):

AttachmentSize
PDF icon sdp_memo_051_-_cloud_native_applications_on_the_sdp_architecture_part_1_-_signed.pdf1.23 MB

52. SDP Memo 052: ''SDP Memo 052: Apache Kafka for an SDP log based architecture'' (Lead Author: Piers Harding):

AttachmentSize
PDF icon sdp_memo_052_-_apache_kafka_for_an_sdp_log_based_architecture_part_1_-_signed.pdf1.18 MB

53. SDP Memo 053: ''SDP Memo 053: Monitoring and Logging for the SDP'' (Lead Author: Piers Harding):

AttachmentSize
PDF icon sdp_memo_053_-_monitoring_and_logging_for_the_sdp_part_1_-_signed.pdf2.34 MB

54. SDP Memo 054: ''SDP Memo 54: Compute Node Pipeline Efficiency Assessment Framework'' (Lead Author: TN Chan):

AttachmentSize
PDF icon sdp_memo_54_compute_efficiency_v1.1.pdf797.74 KB

55. SDP Memo 055: ''SDP Memo 55: Compute Node Hardware Technology Landscape Appraisal'' (Lead Authors: TN Chan and Chris Broekema):

AttachmentSize
PDF icon sdp_memo_55_hardware_landscape_v1.1.pdf973.07 KB

56. SDP Memo 056: ''SDP Memo 56: Hardware Scaling Co-Design Recommendations'' (Lead Author: TN Chan and Chris Broekema):

AttachmentSize
PDF icon sdp_memo_56_scaling_v1.pdf1014.1 KB

57. SDP Memo 057: "Effects of large w visibility points on computation and Dynamic Range" (Lead Author: Daniel Muscat):

AttachmentSize
PDF icon ska-tel-sdp-img-wmax.pdf1.18 MB

59. SDP Memo 059: ''SDP Memo 059: Modeling and Evaluating the IO of MID1 ICAL Pipeline on Spark'' (Lead Author: Qiuhong Li):

AttachmentSize
PDF icon sdp_memo_ef_911.pdf1.65 MB

61. SDP Memo 061: ''SDP Memo 061: SDP Bandpass Calibration Requirements during EoR Averaging'' (Lead Author: Daniel Mitchell):

AttachmentSize
PDF icon sdp_bandpass_calibration_requirements_during_eor_averaging_part_1_-_signed.pdf267.57 KB

62. SDP Memo 062: ''SDP Memo 62: Numerical Precision'' (Lead Author: Anthony Griffin):

AttachmentSize
PDF icon sdp_memo_62_v3_part_1_-_signed.pdf973.29 KB

63. SDP Memo 063: ''Considerations for the SDP Operating System'' (Lead Author: Nicolás Erdödy):

AttachmentSize
PDF icon sdp_memo_063_os_signed_21.10.18.pdf987.94 KB

64. SDP Memo 064: ''Security for the SDP Architecture Considerations'' (Lead Author: Nicolás Erdödy):

AttachmentSize
PDF icon sdp_memo_064_security_signed_21.10.18.pdf669.07 KB

65. SDP Memo 065: ''SDP Memo 065: Fast Implementation of SKA Algorithm Reference Library'' (Lead Author: Haihang You):

AttachmentSize
PDF icon memo_065_fast_implementation_of_ska_algorithm_reference_library.pdf467.65 KB

66. SDP Memo 066: ''SDP Memo 066: Partitioning SKA Dataflows for Optimal Graph Execution'' (Lead Author: Chen Wu):

AttachmentSize
PDF icon sdp_memo_066_partitioning_ska_dataflow.pdf874.17 KB

68. SDP Memo 068: ''SDP Memo 068: Performance Prototype Platform (P3-ALaSKA) Monitoring & Logging'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_068_p3-alaska_monitoring_logging_prototyping_part_1_-_signed.pdf1.6 MB

69. SDP Memo 069: ''SDP Memo 069: Performance Prototype Platform (P3-ALaSKA) OpenStack Prototyping'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_069_p3-alaska_openstack_prototyping_part_1_-_signed.pdf1.61 MB

70. SDP Memo 070: ''SDP Memo 070: P3-ALaSKA Container Orchestration and Compute-Storage Provisioning Interfaces'' (Lead Author: John Taylor):

AttachmentSize
PDF icon sdp_memo_070_p3-alaska_container_orchestration_and_compute-storage_provisioning_interfaces_part_1_-_signed.pdf1.32 MB

71. SDP Memo 071: ''SDP Memo 071: LOFAR pipeline DALiuGE'' (Lead Author: Yan Grange):

AttachmentSize
PDF icon sdp_memo_-_daliuge_with_prefactor.pdf596.09 KB

Recent news

Lovell telescope behind new SKA HQ building on 17 Jan 2019

SDP passes its Critical Design Review

8 February 2019

At the end of October 2018 the SKA SDP Consortium submitted its Critical Design Review (CDR) documentation pack. Contained in this were the formal deliverables of the design consortium covering all aspects of the SDP architecture, system engineering and programmatics (the documents are available here).  The documents were received by the CDR reviewers (consisting of 3 external members and 15 internal to SKAO) who proceeded to generate clarification questions, requests and notes in the form of observations (called OARs after the Observation Action Register approach commonly used for them). Where possible these were addressed via communication exchanges in JIRA OAR tickets.

From 15th to 19th January 2019 members of the SDP Consortium then visited the SKAO HQ at Jodrell Bank (see figure 1), where direct discussions took place in the new SKAO Council Chamber (see figure 2), to further explore the architecture and identify risks as part of an SEI Architecture Tradeoff Analysis Method (ATAM).

SDP1

SDP eNews - December 2018

11 January 2019

At the end of October the SDP Consortium submitted its full document set for Critical Design Review. (These can be found at http://ska-sdp.org/publications/sdp-cdr-documentation) together with a large number of supporting memos (http://ska-sdp.org/publications/released-sdp-memos-i and http://ska-sdp.org/publications/released-sdp-memos-ii). Table 1 below shows the documents in three main categories: those associated with software and hardware architecture; those explaining the supporting prototyping work that has been undertaken in support of the architecture, and finally those associated with system engineering (SE) and programmatics aspects (e.g. specifications for how SDP interfaces with the wider telescope systems and how components will be constructed). The documents will receive observations from a panel of reviewers up until the end of December. Responses to the observations and scenarios to ‘test’ the architecture will then be discussed at a review meeting from 15th to 18th January 2019 at Jodrell Bank.