Standard Data Format: HDF5
It would be great to define a standard data format / event container to store recorded data in. @hanno.perrey suggested the use of HDF5 since it is established, flexible, allows for the storage of meta-data as well as undecoded binary data. In addition, there are libraries for reading and writing these files in many languages. I fully agree that this seems a great fit.
Points we should discuss:
-
Layout of the file - where to store what -
Structure/hierarchy with single or multiple satellites feeding their CDTP stream into one data receiver -
How to store the configuration of the respective satellite? Fetch independently or receive in some event? -
Should we introduce begin-of-run and end-of-run events in CDTP so we can transmit additional metadata and have clear markers for start and end of a measurement? -
We should be following FAIR principles as much as possible. See also the reference paper.
Available libraries:
- C++: https://github.com/BlueBrain/HighFive
- C++ alternative (main lib): https://docs.hdfgroup.org/archive/support/HDF5/doc1.8/cpplus_RM/index.html
- Python: https://docs.h5py.org/en/stable/