AsapoWorker merge requestshttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests2020-09-23T09:54:40+02:00https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/1Feat: Add data to metadata stream.2020-09-23T09:54:40+02:00Mikhail KarnevskiyFeat: Add data to metadata stream.@tim.schoof
Now metadata stream always ask for the last entrie. Both data and user_metadata are given to pre_scan function.
It would make sense to use the same order to arguments: data, metadata. However in this case dependent project...@tim.schoof
Now metadata stream always ask for the last entrie. Both data and user_metadata are given to pre_scan function.
It would make sense to use the same order to arguments: data, metadata. However in this case dependent projects will not work.
Any suggestions?https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/2Add option to receive dataset2020-12-18T15:49:33+01:00Mikhail KarnevskiyAdd option to receive dataset@tim.schoof
In case cases (e.g. lambda detector) image data may come from several modules of one detector. In this case data may be packed in one dataset. Current MR adds an option for AsapoWorker to receive dataset.@tim.schoof
In case cases (e.g. lambda detector) image data may come from several modules of one detector. In this case data may be packed in one dataset. Current MR adds an option for AsapoWorker to receive dataset.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/3Compare substreams of receiver and sender2020-09-23T15:50:36+02:00Mikhail KarnevskiyCompare substreams of receiver and sender@tim.schoof
AsapoWorker is design to automatize processing of multiple substreams. This MR adds an algorithm to derive a next substream, which is not jet processed. An idea is to compare processed substreams (present in sender stream) ...@tim.schoof
AsapoWorker is design to automatize processing of multiple substreams. This MR adds an algorithm to derive a next substream, which is not jet processed. An idea is to compare processed substreams (present in sender stream) and all available in reveiver stream.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/5Feat: Add missing virtual functions2020-09-23T17:16:48+02:00Mikhail KarnevskiyFeat: Add missing virtual functions@tim.schoof
Several functions of worker are called from streamer. Most of these functions are defined in the worker class. This MR adds missing functions, which are called in the streamer but was not defined in the worker.@tim.schoof
Several functions of worker are called from streamer. Most of these functions are defined in the worker class. This MR adds missing functions, which are called in the streamer but was not defined in the worker.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/4Expose flag meta_only to command line2020-09-24T11:03:31+02:00Mikhail KarnevskiyExpose flag meta_only to command line@tim.schoof
Flag meta_only exists in Stream class, but not exposed to application. Therefore I haven't seen a possibility to activate it. Now it is exposed to command line.@tim.schoof
Flag meta_only exists in Stream class, but not exposed to application. Therefore I haven't seen a possibility to activate it. Now it is exposed to command line.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/7Feat: Expose stream to receiver2020-09-24T11:05:09+02:00Mikhail KarnevskiyFeat: Expose stream to receiver@tim.schoof
Stream information may be useful. However seems receiver does not know it class. This MR introduces class variable stream and stores this information in the receiver metadata.@tim.schoof
Stream information may be useful. However seems receiver does not know it class. This MR introduces class variable stream and stores this information in the receiver metadata.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/8Add cli parser test for flattened configuration2020-10-08T08:49:01+02:00Tim SchoofAdd cli parser test for flattened configurationIn some environments, the `argparse` module complains about an "conflicting option" for flattened configurations. This test tries to reproduce that error.In some environments, the `argparse` module complains about an "conflicting option" for flattened configurations. This test tries to reproduce that error.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/6Feat: Add flag to fix metadata substream2020-10-08T12:05:47+02:00Mikhail KarnevskiyFeat: Add flag to fix metadata substream@tim.schoof
In came cases metadata stream is common for all receiver substreams. This MR introduces a flag to fix substream of the metadata stream.@tim.schoof
In came cases metadata stream is common for all receiver substreams. This MR introduces a flag to fix substream of the metadata stream.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/10Feat/new stream opt2020-10-30T11:55:37+01:00Mikhail KarnevskiyFeat/new stream opt@tim.schoof
Different naming scheme of the substreams may be used. This MR offers didderent approaches to chose next substream depending on parameter given in the command line.@tim.schoof
Different naming scheme of the substreams may be used. This MR offers didderent approaches to chose next substream depending on parameter given in the command line.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/11Fix error for worker classes with no arguments2020-11-02T13:59:19+01:00Tim SchoofFix error for worker classes with no arguments@mikhail.karnevskiy Could you please check that the error is indeed fixed, because I didn't have a time to create a test case yet?@mikhail.karnevskiy Could you please check that the error is indeed fixed, because I didn't have a time to create a test case yet?https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/12Use asapo image until access to nims is fixed2020-12-18T15:45:21+01:00Tim SchoofUse asapo image until access to nims is fixedhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/13Retry getting group id in case of errors2022-11-11T12:27:20+01:00Tim SchoofRetry getting group id in case of errorsThis is a quick fix for the random connection errors that can happen when starting a new substream. A similar fix might be necessary for the receiver.
An alternative solution is to reuse the same group id of the previous substream. I th...This is a quick fix for the random connection errors that can happen when starting a new substream. A similar fix might be necessary for the receiver.
An alternative solution is to reuse the same group id of the previous substream. I think that the effects of group ids are local to substreams, so that might be safe to do and it avoids the error completely.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/14Don't create new broker and producer instances2020-12-18T15:51:57+01:00Tim SchoofDon't create new broker and producer instancesBefore, new broker and producer instances were temporarily created for each
substream and immediately replaced with the instances stored in the
application object.
The large amount of new instances is unnecessary and triggered a
resource...Before, new broker and producer instances were temporarily created for each
substream and immediately replaced with the instances stored in the
application object.
The large amount of new instances is unnecessary and triggered a
resource leak in the Python broker wrapper.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/15Feat: Add substream name in logs2021-01-07T15:12:10+01:00Mikhail KarnevskiyFeat: Add substream name in logs@tim.schoof
Suggested changes add a substream name in logs:
```
2021-01-07 11:18:18,915 substream_20201022_144923 AsapoWorker asapo_receiver.py:76 INFO Requesting next record for group_id=btvhe5edspmei3eh0qm0.
```@tim.schoof
Suggested changes add a substream name in logs:
```
2021-01-07 11:18:18,915 substream_20201022_144923 AsapoWorker asapo_receiver.py:76 INFO Requesting next record for group_id=btvhe5edspmei3eh0qm0.
```https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/19Feat: Add handling of csv files2021-02-18T18:13:21+01:00Mikhail KarnevskiyFeat: Add handling of csv files@tim.schoof
Outputting to csv files is a feature requested by beamline scientists.@tim.schoof
Outputting to csv files is a feature requested by beamline scientists.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/20Feat: Add parameters to pre_scan2021-02-24T09:44:19+01:00Mikhail KarnevskiyFeat: Add parameters to pre_scan@tim.schoof
Metadata stream brings parameters to the worker function pre_scan. In this MR there parameters can be described in nested class Parameters. If this class exists, parameters are compared with information coming from Metadata ...@tim.schoof
Metadata stream brings parameters to the worker function pre_scan. In this MR there parameters can be described in nested class Parameters. If this class exists, parameters are compared with information coming from Metadata stream.
* If Parameters are not described in the worker class, nothing changed compared to what was before
* If parameters are described, they are compared with information from Metadata stream.
* If Some of described parameters are missing, error if raised.
* If type of described parameters is standard and not coincides with one from metadata stream, error is raised.
* If type of described parameter is not standard (can not be derived from json). Variable from metadata stream is casted to this type.
* If Metadata have variables, which are not described in the Parameters of worker error is not raised. These variables may be used in the next worker of the pipeline.
Only some standard classes can be de-serialized from json of the metadata stream.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/21Use internal CI runner with access to nims2021-03-04T10:00:43+01:00Tim SchoofUse internal CI runner with access to nimshttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/16update consumer and procuser to new api2021-03-04T13:29:07+01:00Mikhail Karnevskiyupdate consumer and procuser to new apiThis branches switches asapoworker to new api of ASAPO(version 20.12.0)This branches switches asapoworker to new api of ASAPO(version 20.12.0)https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/17Fix compatibility with Python 3.52021-03-24T08:59:18+01:00Tim SchoofFix compatibility with Python 3.5https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/22Feat/next stream update2021-04-07T09:53:36+02:00Mikhail KarnevskiyFeat/next stream update@tim.schoof
in ASAPO 21.03 get_stream_list function return a list of dicts, which contains information about stream:
```
[{'lastId': 1, 'name': 'default', 'timestampCreated': 1617037228745609571, 'timestampLast': 1617037228745609571, ...@tim.schoof
in ASAPO 21.03 get_stream_list function return a list of dicts, which contains information about stream:
```
[{'lastId': 1, 'name': 'default', 'timestampCreated': 1617037228745609571, 'timestampLast': 1617037228745609571, 'finished': False, 'nextStream': ''}, {'lastId': 4, 'name': 'test001', 'timestampCreated': 1617037509990713974, 'timestampLast': 1617038135593899580, 'finished': True, 'nextStream': ''}, ...
```
Presented changes adopt function get_next_stream for the API changes of ASAPO.
CI tests are supplied.