AsapoWorker merge requestshttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests2020-09-24T11:03:31+02:00https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/4Expose flag meta_only to command line2020-09-24T11:03:31+02:00Mikhail KarnevskiyExpose flag meta_only to command line@tim.schoof
Flag meta_only exists in Stream class, but not exposed to application. Therefore I haven't seen a possibility to activate it. Now it is exposed to command line.@tim.schoof
Flag meta_only exists in Stream class, but not exposed to application. Therefore I haven't seen a possibility to activate it. Now it is exposed to command line.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/30Feat/acknowledge slim2021-05-17T14:29:35+02:00Mikhail KarnevskiyFeat/acknowledge slim@tim.schoof
* Implementation of acknowledge functionality
* Removing of Serial Worker
* Processing starts from the beginning of earliest stream, which contains unacnowledge messages.@tim.schoof
* Implementation of acknowledge functionality
* Removing of Serial Worker
* Processing starts from the beginning of earliest stream, which contains unacnowledge messages.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/10Feat/new stream opt2020-10-30T11:55:37+01:00Mikhail KarnevskiyFeat/new stream opt@tim.schoof
Different naming scheme of the substreams may be used. This MR offers didderent approaches to chose next substream depending on parameter given in the command line.@tim.schoof
Different naming scheme of the substreams may be used. This MR offers didderent approaches to chose next substream depending on parameter given in the command line.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/22Feat/next stream update2021-04-07T09:53:36+02:00Mikhail KarnevskiyFeat/next stream update@tim.schoof
in ASAPO 21.03 get_stream_list function return a list of dicts, which contains information about stream:
```
[{'lastId': 1, 'name': 'default', 'timestampCreated': 1617037228745609571, 'timestampLast': 1617037228745609571, ...@tim.schoof
in ASAPO 21.03 get_stream_list function return a list of dicts, which contains information about stream:
```
[{'lastId': 1, 'name': 'default', 'timestampCreated': 1617037228745609571, 'timestampLast': 1617037228745609571, 'finished': False, 'nextStream': ''}, {'lastId': 4, 'name': 'test001', 'timestampCreated': 1617037509990713974, 'timestampLast': 1617038135593899580, 'finished': True, 'nextStream': ''}, ...
```
Presented changes adopt function get_next_stream for the API changes of ASAPO.
CI tests are supplied.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/1Feat: Add data to metadata stream.2020-09-23T09:54:40+02:00Mikhail KarnevskiyFeat: Add data to metadata stream.@tim.schoof
Now metadata stream always ask for the last entrie. Both data and user_metadata are given to pre_scan function.
It would make sense to use the same order to arguments: data, metadata. However in this case dependent project...@tim.schoof
Now metadata stream always ask for the last entrie. Both data and user_metadata are given to pre_scan function.
It would make sense to use the same order to arguments: data, metadata. However in this case dependent projects will not work.
Any suggestions?https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/6Feat: Add flag to fix metadata substream2020-10-08T12:05:47+02:00Mikhail KarnevskiyFeat: Add flag to fix metadata substream@tim.schoof
In came cases metadata stream is common for all receiver substreams. This MR introduces a flag to fix substream of the metadata stream.@tim.schoof
In came cases metadata stream is common for all receiver substreams. This MR introduces a flag to fix substream of the metadata stream.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/9Feat: Add function for likely_done case2020-10-30T11:55:59+01:00Mikhail KarnevskiyFeat: Add function for likely_done case@tim.schoof
Creation of new thread for new substream is quite special situation. It may be useful for worker to know it.
I have a used case: My worker have ZMQ connection on certain port. New connection can not be established in the n...@tim.schoof
Creation of new thread for new substream is quite special situation. It may be useful for worker to know it.
I have a used case: My worker have ZMQ connection on certain port. New connection can not be established in the new thread, because port is busy. Old connection should be closed.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/19Feat: Add handling of csv files2021-02-18T18:13:21+01:00Mikhail KarnevskiyFeat: Add handling of csv files@tim.schoof
Outputting to csv files is a feature requested by beamline scientists.@tim.schoof
Outputting to csv files is a feature requested by beamline scientists.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/5Feat: Add missing virtual functions2020-09-23T17:16:48+02:00Mikhail KarnevskiyFeat: Add missing virtual functions@tim.schoof
Several functions of worker are called from streamer. Most of these functions are defined in the worker class. This MR adds missing functions, which are called in the streamer but was not defined in the worker.@tim.schoof
Several functions of worker are called from streamer. Most of these functions are defined in the worker class. This MR adds missing functions, which are called in the streamer but was not defined in the worker.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/20Feat: Add parameters to pre_scan2021-02-24T09:44:19+01:00Mikhail KarnevskiyFeat: Add parameters to pre_scan@tim.schoof
Metadata stream brings parameters to the worker function pre_scan. In this MR there parameters can be described in nested class Parameters. If this class exists, parameters are compared with information coming from Metadata ...@tim.schoof
Metadata stream brings parameters to the worker function pre_scan. In this MR there parameters can be described in nested class Parameters. If this class exists, parameters are compared with information coming from Metadata stream.
* If Parameters are not described in the worker class, nothing changed compared to what was before
* If parameters are described, they are compared with information from Metadata stream.
* If Some of described parameters are missing, error if raised.
* If type of described parameters is standard and not coincides with one from metadata stream, error is raised.
* If type of described parameter is not standard (can not be derived from json). Variable from metadata stream is casted to this type.
* If Metadata have variables, which are not described in the Parameters of worker error is not raised. These variables may be used in the next worker of the pipeline.
Only some standard classes can be de-serialized from json of the metadata stream.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/15Feat: Add substream name in logs2021-01-07T15:12:10+01:00Mikhail KarnevskiyFeat: Add substream name in logs@tim.schoof
Suggested changes add a substream name in logs:
```
2021-01-07 11:18:18,915 substream_20201022_144923 AsapoWorker asapo_receiver.py:76 INFO Requesting next record for group_id=btvhe5edspmei3eh0qm0.
```@tim.schoof
Suggested changes add a substream name in logs:
```
2021-01-07 11:18:18,915 substream_20201022_144923 AsapoWorker asapo_receiver.py:76 INFO Requesting next record for group_id=btvhe5edspmei3eh0qm0.
```https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/7Feat: Expose stream to receiver2020-09-24T11:05:09+02:00Mikhail KarnevskiyFeat: Expose stream to receiver@tim.schoof
Stream information may be useful. However seems receiver does not know it class. This MR introduces class variable stream and stores this information in the receiver metadata.@tim.schoof
Stream information may be useful. However seems receiver does not know it class. This MR introduces class variable stream and stores this information in the receiver metadata.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/33Fix and speedup2021-07-12T09:46:33+02:00Mikhail KarnevskiyFix and speedup@tim.schoof
Fix a few small points, which may speedup to run across streams.
* Check number of unacknowledged messages before closing stream only if receiver will retry.
* Starting stream first, because it may take time.
* Try to set s...@tim.schoof
Fix a few small points, which may speedup to run across streams.
* Check number of unacknowledged messages before closing stream only if receiver will retry.
* Starting stream first, because it may take time.
* Try to set starting position only if worker have corresponding method
* Set stream before call set_starting_position (this was a bug)https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/17Fix compatibility with Python 3.52021-03-24T08:59:18+01:00Tim SchoofFix compatibility with Python 3.5https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/37Fix configuration with values considered as False2021-12-10T11:41:42+01:00Tim SchoofFix configuration with values considered as FalseFor values like 0 that are considered as `False` in if statements, the default value was used instead of the given option.For values like 0 that are considered as `False` in if statements, the default value was used instead of the given option.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/50Fix conversion of boolean cli options2022-09-12T15:20:22+02:00Tim SchoofFix conversion of boolean cli optionsBefore, using `--receiver.has_filesystem false` or similar on the command line would result in `has_filesystem` being `True`.Before, using `--receiver.has_filesystem false` or similar on the command line would result in `has_filesystem` being `True`.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/32Fix dependency on hdf5plugin2021-07-08T15:12:31+02:00Tim SchoofFix dependency on hdf5pluginThis commit also removes numcodecs from the CI environment.This commit also removes numcodecs from the CI environment.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/11Fix error for worker classes with no arguments2020-11-02T13:59:19+01:00Tim SchoofFix error for worker classes with no arguments@mikhail.karnevskiy Could you please check that the error is indeed fixed, because I didn't have a time to create a test case yet?@mikhail.karnevskiy Could you please check that the error is indeed fixed, because I didn't have a time to create a test case yet?https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/57Fix for get_next_dataset2023-10-24T12:59:32+02:00Mikhail KarnevskiyFix for get_next_dataset@tim.schoof
Consider AsapoPartialDataError as TemporaryError.
Add parameter for min size of dataset.@tim.schoof
Consider AsapoPartialDataError as TemporaryError.
Add parameter for min size of dataset.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/62Fix last_success_timestamp assignment:2023-12-13T16:23:17+01:00Diana RuedaFix last_success_timestamp assignment:- Set last_success_timestamp as None for the case when there has not
been a success in process stream.- Set last_success_timestamp as None for the case when there has not
been a success in process stream.Diana RuedaDiana Rueda