AsapoWorker merge requestshttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests2024-03-06T18:49:46+01:00https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/64introduce varialbe to get next ordered2024-03-06T18:49:46+01:00Mikhail Karnevskiyintroduce varialbe to get next ordered@tim.schoof@tim.schoofhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/63Improve pipeline_step2023-12-20T10:48:45+01:00Mikhail KarnevskiyImprove pipeline_step@tim.schoof
This MR set pipeline step of all involved receivers and consumers: (receiver consumer, metadata consumer, sender producer and sender consumer) to the group_id. Group_id (if not given) is set to sender data_source (if exists).@tim.schoof
This MR set pipeline step of all involved receivers and consumers: (receiver consumer, metadata consumer, sender producer and sender consumer) to the group_id. Group_id (if not given) is set to sender data_source (if exists).https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/62Fix last_success_timestamp assignment:2023-12-13T16:23:17+01:00Diana RuedaFix last_success_timestamp assignment:- Set last_success_timestamp as None for the case when there has not
been a success in process stream.- Set last_success_timestamp as None for the case when there has not
been a success in process stream.Diana RuedaDiana Ruedahttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/61Add option to set a shutdown timeout in seconds:2023-12-11T15:57:59+01:00Diana RuedaAdd option to set a shutdown timeout in seconds:It checks the time of last successful get_next and if the time
passed is longer than the shutdown_timeout it shuts down.
The default value is set to 10 minutes.It checks the time of last successful get_next and if the time
passed is longer than the shutdown_timeout it shuts down.
The default value is set to 10 minutes.Diana RuedaDiana Ruedahttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/60Add flag to disable automatic stream switching2023-12-05T18:07:10+01:00Diana RuedaAdd flag to disable automatic stream switchingDiana RuedaDiana Ruedahttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/59Refactor serial worker2023-12-07T11:54:14+01:00Mikhail KarnevskiyRefactor serial worker@tim.schoof
Serial worker implement a functionality to go across ordered messages of stream. This MR switched from `get_next` to `get_by_id` asapo API. The choice of `start_id` is now transparently on client side.@tim.schoof
Serial worker implement a functionality to go across ordered messages of stream. This MR switched from `get_next` to `get_by_id` asapo API. The choice of `start_id` is now transparently on client side.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/58Remove stream naming scheme.2023-12-01T16:06:05+01:00Mikhail KarnevskiyRemove stream naming scheme.@tim.schoof
Different stream naming scheme were introduced because of the missing API in asapo service. Currently asapo delivers streams ordered by creation time. Current MR removes all naming schemes and fix the logic to use date nami...@tim.schoof
Different stream naming scheme were introduced because of the missing API in asapo service. Currently asapo delivers streams ordered by creation time. Current MR removes all naming schemes and fix the logic to use date naming scheme.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/57Fix for get_next_dataset2023-10-24T12:59:32+02:00Mikhail KarnevskiyFix for get_next_dataset@tim.schoof
Consider AsapoPartialDataError as TemporaryError.
Add parameter for min size of dataset.@tim.schoof
Consider AsapoPartialDataError as TemporaryError.
Add parameter for min size of dataset.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/56Add option set set stream to latest2023-06-19T10:02:57+02:00Mikhail KarnevskiyAdd option set set stream to latest@tim.schoof
Often p02.2 wants to start pipeline with latest stream to process currently taking data. I would suggest to introduce this option here.@tim.schoof
Often p02.2 wants to start pipeline with latest stream to process currently taking data. I would suggest to introduce this option here.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/55Add support for max_streams command line option.2023-06-01T12:39:08+02:00Parthasarathy TirumalaiAdd support for max_streams command line option.Hi Mika, Tim,
Please review the changes and test before merging.
Thanks,
ParthaHi Mika, Tim,
Please review the changes and test before merging.
Thanks,
ParthaMikhail KarnevskiyMikhail Karnevskiyhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/54Raise exception if serial worker have n_resend_nacs>02023-05-11T22:36:50+02:00Mikhail KarnevskiyRaise exception if serial worker have n_resend_nacs>0@tim.schoof@tim.schoofhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/53Add Simple dataset receiver2023-05-11T22:44:28+02:00Mikhail KarnevskiyAdd Simple dataset receiver@tim.schoof@tim.schoofhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/52Order list of meta dicts in the dataset2023-05-10T10:10:19+02:00Mikhail KarnevskiyOrder list of meta dicts in the dataset@tim.schoof
I check for the key `dataset_substream` to be sure, that code will work if it will be not there.@tim.schoof
I check for the key `dataset_substream` to be sure, that code will work if it will be not there.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/51Make resend timeout configurable2022-11-11T12:34:12+01:00Tim SchoofMake resend timeout configurablePreviously, the general receiver timeout was used for the time to wait before resending unacknowledged message. This time is usually under 5s which is too short for this purpose.
I am not sure what is a good default. Too short, and mess...Previously, the general receiver timeout was used for the time to wait before resending unacknowledged message. This time is usually under 5s which is too short for this purpose.
I am not sure what is a good default. Too short, and messages will be resent unnecessarily while the original message is still being processed, too long and the message might already be dropped from the memory buffer and cannot be resent if not persisted.
Closed #11Mikhail KarnevskiyMikhail Karnevskiyhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/50Fix conversion of boolean cli options2022-09-12T15:20:22+02:00Tim SchoofFix conversion of boolean cli optionsBefore, using `--receiver.has_filesystem false` or similar on the command line would result in `has_filesystem` being `True`.Before, using `--receiver.has_filesystem false` or similar on the command line would result in `has_filesystem` being `True`.https://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/49Expoce functions to get/send stream metadata2022-08-09T23:02:03+02:00Mikhail KarnevskiyExpoce functions to get/send stream metadata@tim.schoof@tim.schoofMikhail KarnevskiyMikhail Karnevskiyhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/48Implement seedee lib for serialisation2022-08-09T15:36:31+02:00Mikhail KarnevskiyImplement seedee lib for serialisation@tim.schoof@tim.schoofMikhail KarnevskiyMikhail Karnevskiyhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/47Fix signal handling2022-09-23T15:10:43+02:00Tim SchoofFix signal handlingWhen the environment variable is not set, a library used by ASAPO
installs a signal handler that immediately exits, preventing a graceful
shutdown.When the environment variable is not set, a library used by ASAPO
installs a signal handler that immediately exits, preventing a graceful
shutdown.Tim SchoofTim Schoofhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/46Allow mixing Config and attr.ib parameters2022-07-04T16:26:06+02:00Tim SchoofAllow mixing Config and attr.ib parametersThis is useful to allow "hidden" parameters that can be used in the
constructor, but are not exposed as options on the command line
or in the configuration file.This is useful to allow "hidden" parameters that can be used in the
constructor, but are not exposed as options on the command line
or in the configuration file.Tim SchoofTim Schoofhttps://gitlab.desy.de/fs-sc/asapoworker/-/merge_requests/45Handle AsapoDataNotInCacheError in AsapoReceiver2022-07-07T09:54:23+02:00Tim SchoofHandle AsapoDataNotInCacheError in AsapoReceiverIn case the data is not in the cache any more, the data is lost, and
thus an AsapoDataNotInCacheError is mapped to a MissingDataError.
SimpleAsapoReceiver can now also throw a MissingDataError, but this
should be ok as a worker either e...In case the data is not in the cache any more, the data is lost, and
thus an AsapoDataNotInCacheError is mapped to a MissingDataError.
SimpleAsapoReceiver can now also throw a MissingDataError, but this
should be ok as a worker either expects this error or it falls back to
the defaul no-op handler.
The implementation revealed an error in the testing code, which is now
fixed in the first commit.
Closes #10