Skip to content
Snippets Groups Projects
Commit e7140bcb authored by Mikhail Karnevskiy's avatar Mikhail Karnevskiy
Browse files

Merge branch 'feature/web-doc-update' into 'develop' 

Include changes on Documentation:

See merge request asapo/asapo!212
parents 8e206ea1 d45b9463
Branches
Tags
No related merge requests found
Hide whitespace changes
Inline Side-by-side
Showing
with 42 additions and 42 deletions
docs/site/docs/compare-to-others.md
+ 17
− 17
View file @ e7140bcb
......@@ -15,7 +15,7 @@ HiDRA can work in two modes - wether data is transferred via it or data is writt
### ASAPO
ASAPO does not work with files, rather with data streams. Well, behind the scene it does use files, but in principle what a user see is a stream of messages, where a message typically consists of metadata and data blobs. Data blob can be a single image, a file with arbitrary content or whatever else, even null. Important is - what goes to one end, appears on the other. And that each message must have a consecutive index. These messages must be ingested to an ASAPO data stream in some way (e.g. using ASAPO Producer API or HiDRA) and data, if not yet, will be transferred and stored in the data center. A user can then read the messages from the stream and process it in a way he likes.
ASAPO does not work with files, rather with data streams. Well, behind the scenes it does use files, but in principle what a user see is a stream of messages, where a message typically consists of metadata and data blobs. Data blob can be a single image, a file with arbitrary content or whatever else, even null. Important is - what goes to one end, appears on the other. And that each message must have a consecutive index. These messages must be ingested to an ASAPO data stream in some way (e.g. using ASAPO Producer API or HiDRA) and data, if not yet, will be transferred and stored in the data center. A user can then read the messages from the stream and process it in a way he likes.
### Compare by categories
......@@ -23,27 +23,27 @@ In the table below we compare the approaches from different points of view and i
| Category | Filesystem | Filesystem+Kafka | HiDRA | ASAPO |
|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Data ingest | traditional way - write to disk | same as Filesystem, additionally a message is send to a Kafka topic with basic file metadata (name, size, creation date, ...) | outperforms data ingest via NFS/SMB, uses ZeroMQ(TCP), saturates 10GE bandwidth. To be tested with 100GE. Same metadata as with Kafka | uses parallel TCP connections & in the future RDMA. Can add arbitrary metadata (JSON format). Can send data to various streams to create arbitrary data hierarchy (e.g. stream per scan). Saves data to a memory buffer and on disk. |
| Offline analysis | traditional way - read from disk. Need to know the filename to read, usually not a problem. | same as Filesystem, there is no benefits reading Kafka queue after all data is arrived | not possible, need to fallback to Filesystem | same as online analysis (see below) |
| Online analysis | no efficient way to recognise that new data is available - periodically read folder content and compare with previous state?, periodically check file appeared? | using a subscription to a "new files" topic, a user software can be made aware of new data quite soon and react correspondingly. | one can subscribe to a HiDRA's ZeroMQ stream and consume data. If data arrives faster than it can be processed or e.g. user software crashes - data might be skipped. | one can get data from various streams in different ways - get next unprocessed message ordered by index, get last, get by id. Since everything is stored in persistent storage, processing is possible with arbitrary slow (but also fast) consumers. Resilient to connections loss or consumer crashes. |
| Performance | as good as read/write to disk. Can be an issue, especially with new detectors 100GE+ networks. | as good as read/write to disk + some latency for the file to be written to the beamline filesystem, copied to the core filesystem and a message to go through Kafka. | data is available as soon as it is received by HiDRA. If multiple consumer groups want to read same data (one consumer is known - the process that writes the file), data will be transferred multiple times, which influences the network performance. | data is available to be consumed as soon as it is arrived and saved to beamline filesystem (later can be optimised by using persistent memory instead). No need to read from disk since it also remains in memory buffer. |
| Data ingest | Traditional way - write to disk | Same as Filesystem, additionally a message is send to a Kafka topic with basic file metadata (name, size, creation date, ...) | Outperforms data ingest via NFS/SMB, uses ZeroMQ(TCP), saturates 10GE bandwidth. To be tested with 100GE. Same metadata as with Kafka |Uses parallel TCP connections & in the future RDMA. Can add arbitrary metadata (JSON format). Can send data to various streams to create arbitrary data hierarchy (e.g. stream per scan). Saves data to a memory buffer and on disk. |
| Offline analysis | Traditional way - read from disk. Need to know the filename to read, usually not a problem. | Same as Filesystem, there is no benefits reading Kafka queue after all data is arrived | Not possible, need to fallback to Filesystem | Same as online analysis (see below) |
| Online analysis | No efficient way to recognise that new data is available - periodically read folder content and compare with previous state?, periodically check file appeared? | Using a subscription to a "new files" topic, a user software can be made aware of new data quite soon and react correspondingly. | One can subscribe to a HiDRA's ZeroMQ stream and consume data. If data arrives faster than it can be processed or e.g. user software crashes - data might be skipped. | One can get data from various streams in different ways - get next unprocessed message ordered by index, get last, get by id. Since everything is stored in persistent storage, processing is possible with arbitrary slow (but also fast) consumers. Resilient to connections loss or consumer crashes. |
| Performance | As good as read/write to disk. Can be an issue, especially with new detectors 100GE+ networks. | As good as read/write to disk + some latency for the file to be written to the beamline filesystem, copied to the core filesystem and a message to go through Kafka. | Data is available as soon as it is received by HiDRA. If multiple consumer groups want to read same data (one consumer is known - the process that writes the file), data will be transferred multiple times, which influences the network performance. | Data is available to be consumed as soon as it has arrived and saved to beamline filesystem (later can be optimised by using persistent memory instead). No need to read from disk since it also remains in memory buffer. |
| Parallelisation | Parallelisation is easily possible e.g. with an MPI library. | Parallelisation is possible if Kafka's topics are partitioned (which is not the case in the moment) | Not out of the box, possible with some changes from user's side | Parallelisation is easily possible, one can consume data concurrently with different consumers from the same stream. Normally, synchronisation between consumers is not needed, but this might depend on a use case. When configured, data can be resent if not acknowledged during a specified time period. |
| Search/filter | hardly possible, manually parsing some metadata file, using POSIX commands? | same as Filesystem. There is Kafka SQL query language which could be used if there would be metadata in messages, which is not the case (yet?). | not possible | one can use a set of SQL queries to ingested metadata. |
| General comments | Might be ok for slow detectors or/and a use case without online analysis requirements. Might be the only way to work with legacy applications | Fits well for the cases where a software just need a trigger that some new data has arrived, when processing order, extra metadata, parallelisation is not that important or implemented by other means. Some delay between an image is generated and the event is emitted by Kafka is there, but probably not that significant (maximum a couple of seconds). Might be not that appropriate for very fast detectors since still using filesystem to write/read data. | Works quite well to transfer files from detector to the data center. Also a good candidate for live viewers, where the last available "image" should be displayed. Does not work for offline analysis or for near real-time analysis where image processing can take longer than image taking. | Tries to be a general solution which improves in areas where other approaches not suffice: single code for offline/near real-time/online analysis, parallelisation, extended metadata, efficient memory/storage management, getting data without access to filesystem (e.g. from detector pc without core filesystem mounted), computational pipelines, ... Everything has its own price: user software must be modified to use ASAPO, a wrapper might be needed for legacy software that cannot be modified, user/beamtime scientist should better structure the data - e.g. consecutive indexes must be available for each image, one has to define to which stream write/read data, what is the format of the data, ... |
| Search/filter | Hardly possible, manually parsing some metadata file, using POSIX commands? | Same as Filesystem. There is Kafka SQL query language which could be used if there would be metadata in messages, which is not the case (yet?). | Not possible | One can use a set of SQL queries to ingested metadata. |
| General comments | Might be ok for slow detectors or/and a use case without online analysis requirements. Might be the only way to work with legacy applications | Fits well for the cases where a software just needs a trigger that some new data has arrived, when processing order, extra metadata, parallelisation is not that important or implemented by other means. Some delay between an image is generated and the event is emitted by Kafka is there, but probably not that significant (maximum a couple of seconds). Might be not that appropriate for very fast detectors since it is still using filesystem to write/read data. | Works quite well to transfer files from detector to the data center. Also a good candidate for live viewers, where the last available "image" should be displayed. Does not work for offline analysis or for near real-time analysis where image processing can take longer than image taking. | Tries to be a general solution which improves in areas where other approaches not suffice: single code for offline/near real-time/online analysis, parallelisation, extended metadata, efficient memory/storage management, getting data without access to filesystem (e.g. from detector pc without core filesystem mounted), computational pipelines, ... Everything has its own price: user software must be modified to use ASAPO, a wrapper might be needed for legacy software that cannot be modified, user/beamtime scientist should better structure the data - e.g. consecutive indexes must be available for each image, one has to define to which stream to write/read data, what is the format of the data, ... |
### Compare by features
| Feature | Filesystem | Filesystem+Kafka | HiDRA | ASAPO |
|--------------------------------------------------------------------------------------------------------------------|------------------------------------|------------------------------------|------------------------------------|------------------------------------|
| send metadata with image | No | No | No | Yes |
| get last image | No | No | Yes | Yes |
| get image by id | No | No | No | Yes |
| get image in order | No | No | No | Yes |
| Send metadata with image | No | No | No | Yes |
| Get last image | No | No | Yes | Yes |
| Get image by id | No | No | No | Yes |
| Get image in order | No | No | No | Yes |
| Immediately get informed that a new image is arrived | No | Yes | Yes | Yes |
| access image remotely, without reading filesystem | No | No | Yes, if it is still in buffer | Yes |
| access past images | Yes | Yes | No | Yes |
| need to change user code | No | Yes | Yes | Yes |
| parallelisation | Yes (if user software allows that) | Not out of the box | Not out of the box | Yes |
| legacy applications | Yes | No (wrapper could be a workaround) | No (wrapper could be a workaround) | No (wrapper could be a workaround) |
| transparent restart/continuation of simulations in case e.g. worker process crashes, also for parallel simulations | Not out of the box | Yes | No | Yes |
| Access image remotely, without reading filesystem | No | No | Yes, if it is still in buffer | Yes |
| Access past images | Yes | Yes | No | Yes |
| Need to change user code | No | Yes | Yes | Yes |
| Parallelisation | Yes (if user software allows that) | Not out of the box | Not out of the box | Yes |
| Legacy applications | Yes | No (wrapper could be a workaround) | No (wrapper could be a workaround) | No (wrapper could be a workaround) |
| Transparent restart/continuation of simulations in case e.g. worker process crashes, also for parallel simulations | Not out of the box | Yes | No | Yes |
docs/site/docs/consumer-clients.md
+ 9
− 9
View file @ e7140bcb
......@@ -6,20 +6,20 @@ Consumer client (or consumer) is a part of a distributed streaming system that i
![Docusaurus](/img/consumer-clients.png)
Consumer API is available for C++ and Python and has the following main functionality:
Consumer API is available for C++ and Python and has the following main functionalities:
- Create a consumer instance and bind it to a specific beamtime and data source
- multiple instances can be created (also within a single application) to receive data from different sources
- an instance id and pipeline step id can be set to allow pipeline monitoring
- a beamtime token is used for access control
- If needed (mainly for get_next_XX operations), create a consumer group that allows to process messages independently from other groups
- Multiple instances can be created (also within a single application) to receive data from different sources
- An instance id and pipeline step id can be set to allow pipeline monitoring
- A beamtime token is used for access control
- If needed (mainly for 'get_next_XX' operations), create a consumer group that allows to process messages independently from other groups
- Receive messages from a specific stream (you can read more [here](data-in-asapo) about data in ASAPO)
- GetNext to receive process messages one after another without need to know message indexes
- `get_next` to receive process messages one after another without need to know message indexes
- Consumer API returns a message with index 1, then 2, ... as they were set by producer.
- This also works in parallel so that payload is distributed within multiple consumers within same consumer group or between threads of a single consumer instance. In parallel case order of indexes of the messages is not determined.
- This also works in parallel so that the payload is distributed within multiple consumers within same consumer group or between threads of a single consumer instance. In parallel case order of indexes of the messages is not determined.
- Optionaly it can return messages in their arrival order (return next available message). It is useful for the fast data-processing agnostic to message index.
- GetLast to receive last available message - for e.g. live visualisation
- GetById - get message by index - provides random access
- `get_last` to receive last available message - for e.g. live visualisation
- `get_by_id` to get message by index, providing random access
- Make queries based on metadata contained in a message - returns all messages in a stream with specific metadata. A subset of SQL language is used
......
docs/site/docs/cookbook/simple-producer.mdx
+ 2
− 2
View file @ e7140bcb
......@@ -6,7 +6,7 @@ title: Simple Producer
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
This example produces a simple message. This page provides snippets for simple producer for both Python and C++. You can go to BitBucket to see the whole example at once. The files there is a working example ready for launch.
This example produces a simple message. This page provides snippets for simple producer for both Python and C++. You can click the link on the bottom right of the code snippets to see the whole example at once. In the referenced file there is a working example ready for launch.
First step is to create an instance of the producer.
......@@ -145,4 +145,4 @@ $ ./asapo-produce
</TabItem>
</Tabs>
You should see the "successfuly sent" message in the logs, and the file should appear in the corresponding folder (by default in ```/var/tmp/asapo/global_shared/data/test_facility/gpfs/test/2019/data/asapo_test```).
You should see the "successfuly sent" message in the logs, and the file should appear in the corresponding folder (for the ASAPO standalone docker container in ```/tmp/asapo/receiver/files/test_facility/gpfs/test/2019/data/asapo_test/processed/``` by default).
docs/site/docs/core-architecture.md
+ 1
− 1
View file @ e7140bcb
......@@ -22,7 +22,7 @@ the workflow can be split into two more or less independent tasks - data ingesti
[Consumer client](consumer-clients.md)) is usually a user application that retrieves data from the system to analyse/process it.
The first step to retrieve a message via Consumer API is to pass the request to the Data Broker (1r). The Data Broker retrieves the metadata information about the message from the database (2r) and returns it to the Consumer Client. The Consumer Client analyses the metadata information and decides how to get the data. It the data is still in the Receiver memory cache, the client requests data from there via a Data Server (which is a part of ASAPO Receiver). Otherwise, client gets the data from the filesystem - directly if the filesystem is accessible on the machine where the client is running or via File Transfer Service if not.
The first step to retrieve a message via Consumer API is to pass the request to the Data Broker (1r). The Data Broker retrieves the metadata information about the message from the database (2r) and returns it to the Consumer Client. The Consumer Client analyses the metadata information and decides how to get the data. It the data is still in the Receiver memory cache, the client requests data from there via a Data Server (which is a part of ASAPO Receiver). Otherwise, the client gets the data from the filesystem - directly if the filesystem is accessible on the machine where the client is running or via File Transfer Service if not.
......
docs/site/docs/data-in-asapo.md
+ 4
− 4
View file @ e7140bcb
......@@ -13,15 +13,15 @@ This is the top level. Contains all data collected/produced during a single beam
During a beamtime, data can be produced by different sources. For example, a detector is a data source, if multiple detectors are used during an experiment, they can be different data sources or the same data source (more details below in datasets section). A user application that simulates or analyses data can also act as an ASAPO data source. Each data source has its own unique name within a beamtime.
#### Data Stream
Each data source can emit multiple data streams. Each stream has a unique within a specific data source name.
Each data source can emit multiple data streams. For each specific data source name, stream names must be unique.
#### Message
Data streams consist of smaller entities - messages. The content of a message is quite flexible, to be able to cover a broad amount of use cases. Usually it is a metadata and some binary data (e.g. a detector image, or an hdf5 file with multiple images). At the moment ASAPO itself is agnostic to the data and sees it as a binary array. Later some specific cases might be handled as well (the most prominent use case - an hdf5 file with multiple images).
Data streams consist of smaller entities - messages. The content of a message is quite flexible, to be able to cover a broad amount of use cases. Usually it is metadata and some binary data (e.g. a detector image, or an hdf5 file with multiple images). At the moment ASAPO itself is agnostic to the data and sees it as a binary array. Later some specific cases might be handled as well (the most prominent use case - an hdf5 file with multiple images).
An important aspect is that each message within a data stream must be assigned a consecutive integer index. Therefore, a streams always contain messages with index = 1,2,3 ... . This is different to traditional messaging systems where messages have timestamps or arbitrary unique hash IDs. The reason is that with timestamps the order of messages saved in the system might differ from the order the were generated by the data source (e.g. detector). And keeping correct order is required in many cases. Second reason is that it makes a random access to a specific message quite straightforward.
#### Datasets/Dataset substreams
In some cases multiple detectors are using during an experiment. E.g. a 3D image is composed from multiple 2D images created by different detectors. In this case these 2D images can be composed to a dataset so that it a be processed later as a whole. One would then use a single data source (which would mean a set of detectors or "multi-detector" in this case), single data stream and, to compose a dataset, for each of it's components (each 2D image in our example) the corresponding detector would send a message with same id but to a different dataset substream.
In some cases multiple detectors are being used during an experiment. E.g. a 3D image is composed from multiple 2D images created by different detectors. In this case these 2D images can be composed to a dataset so that it a be processed later as a whole. One would then use a single data source (which would mean a set of detectors or "multi-detector" in this case), single data stream and, to compose a dataset, for each of it's components (each 2D image in our example) the corresponding detector would send a message with same id but to a different dataset substream.
So, for the case without datasets (single detector) the data hierarchy is Beamtime→Data Source → Data Stream → Message:
......@@ -35,4 +35,4 @@ And with datasets (multi-detector) the data hierarchy is Beamtime→Data Source
Data transferred to asapo service is stored in files on cache buffer or/and (depending on ingest mode) in disc. Asapo disc storage is bounded to DESY infrastructure. Mounting point is either located at beamline gpfs (so-called `raw` data type) or at core gpfs (so-called `processed` data type). Exact path is defined inside the asapo authorization system.
Currently, asapo caches data in a circular buffer with a size of multiple hundreds of GB. This cache is not dedicated to a certain producer, but rather global across asapo service.
\ No newline at end of file
Currently, asapo caches data in a circular buffer with a size of multiple hundreds of GB. This cache is not dedicated to a certain producer, but rather global across asapo service.
docs/site/docs/getting-started.mdx
+ 2
− 2
View file @ e7140bcb
......@@ -80,7 +80,7 @@ Now you can install Python packages or C++ libraries for ASAPO Producer and Cons
## Using the access token
By default the access token are generated for asapo automatically during the beamsime start. For testing purposes the keys need to be generated by hand. You can use this pre-generated key for the asapo-standalone package:
By default the access token are generated for asapo automatically during the beamtime start. For testing purposes the keys need to be generated by hand. You can use this pre-generated key for the asapo-standalone package:
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjk1NzE3MTAyMTYsImp0aSI6Ind0ZmlzdGhpcyIsInN1YiI6ImJ0X2FzYXBvX3Rlc3QiLCJFeHRyYUNsYWltcyI6eyJBY2Nlc3NUeXBlcyI6WyJ3cml0ZSIsIndyaXRlcmF3IiwicmVhZCJdfX0.cz6R_kVf4yh7IJD6bJjDdgTaxPN3txudZx9DE6WaTtk
......@@ -90,5 +90,5 @@ This token will give you full access to the `asapo_test` beamtime. Or you can us
## Code examples
Please refer to the [Code Examples](cookbook/overview) sections to see the code snippets for various usage scenarious.
Please refer to the [Code Examples](cookbook/overview) sections to see the code snippets for various usage scenarios.
docs/site/docs/overview.md
+ 1
− 1
View file @ e7140bcb
......@@ -4,7 +4,7 @@ title: Overview
ASAP::O (or ASAPO) is a high performance distributed streaming platform. It is being developed at DESY and is mainly aimed to support online/offline analysis of experimental data produced at its facilities. The ideas behind are quite similar to that of Apache Kafka and similar messaging solutions, but ASAPO is developed and tuned for scientific use cases with their specific workflows and where the size of the messages is much large (MBs to GBs as compared to KBs in traditional systems).
ASAP::O (or ASAPO) is a high performance distributed streaming platform. It is being developed at DESY and is mainly aimed to support online/offline analysis of experimental data produced at its facilities. The ideas behind are quite similar to that of Apache Kafka and other messaging solutions, yet ASAPO is developed and tuned for scientific use cases with their specific workflows and where the size of the messages is much large (MBs to GBs as compared to KBs in traditional systems).
......
docs/site/docs/producer-clients.md
+ 4
− 4
View file @ e7140bcb
......@@ -2,14 +2,14 @@
title: Producer Clients
---
Producer client (or producer) is a part of a distributed streaming system that is responsible for creating data streams (i.e. ingesting data in the system). It is usually a user (beamline scientist, detector developer, physicist, ... ) responsibility to develop a client for specific beamline, detector or experiment using ASAPO Producer API and ASAPO responsibility to make sure data is transferred and saved an in efficient and reliable way.
Producer client (or producer) is a part of a distributed streaming system that is responsible for creating data streams (i.e. ingesting data in the system). It is usually a user's (beamline scientist, detector developer, physicist, ... ) responsibility to develop a client for specific beamline, detector or experiment using ASAPO Producer API and ASAPO's responsibility to make sure data is transferred and saved an in efficient and reliable way.
![Docusaurus](/img/producer-clients.png)
Producer API is available for C++ and Python and has the following main functionality:
Producer API is available for C++ and Python and has the following main functionalities:
- Create a producer instance and bind it to a specific beamtime and data source
multiple instances can be created (also within a single application) to send data from different sources. An instance id and pipeline step id can be set to allow pipeline monitoring.
- Create a producer instance and bind it to a specific beamtime and data source.
- Multiple instances can be created (also within a single application) to send data from different sources. An instance id and pipeline step id can be set to allow pipeline monitoring.
- Send messages to a specific stream (you can read more [here](data-in-asapo) about data in ASAPO)
- each message must have a consecutive integer index, ASAPO does not create indexes automatically
- to compose datasets, dataset substream (and dataset size) should be send along with each message
......
docs/site/versioned_examples/version-23.12.0/install_python_clients_pip.sh
+ 2
− 2
View file @ e7140bcb
#!/usr/bin/env bash
pip3 install --user --trusted-host nims.desy.de --find-links=http://nims.desy.de/extra/asapo/linux_wheels asapo_producer==23.12.0-alpha
pip3 install --user --trusted-host nims.desy.de --find-links=http://nims.desy.de/extra/asapo/linux_wheels asapo_consumer==23.12.0-alpha
python3 -m pip install --trusted-host nims.desy.de --find-links=http://nims.desy.de/extra/asapo/linux_wheels asapo_producer==23.12.0-alpha
python3 -m pip install --trusted-host nims.desy.de --find-links=http://nims.desy.de/extra/asapo/linux_wheels asapo_consumer==23.12.0-alpha
# you might need to update pip if the above commands error: pip3 install --upgrade pip
# if that does not work (abi incompatibility, etc) you may try to install source packages
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment

imprint | privacy | Conditions of Use