Skip to content
Snippets Groups Projects

Feat: Add parameters to pre_scan

Merged Mikhail Karnevskiy requested to merge feat/meta_to_param into master

@tim.schoof Metadata stream brings parameters to the worker function pre_scan. In this MR there parameters can be described in nested class Parameters. If this class exists, parameters are compared with information coming from Metadata stream.

  • If Parameters are not described in the worker class, nothing changed compared to what was before
  • If parameters are described, they are compared with information from Metadata stream.
    • If Some of described parameters are missing, error if raised.
    • If type of described parameters is standard and not coincides with one from metadata stream, error is raised.
    • If type of described parameter is not standard (can not be derived from json). Variable from metadata stream is casted to this type.
    • If Metadata have variables, which are not described in the Parameters of worker error is not raised. These variables may be used in the next worker of the pipeline.

Only some standard classes can be de-serialized from json of the metadata stream.

Edited by Mikhail Karnevskiy

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
195 195 d = d[k]
196 196 d[keys[-1]] = value
197 197 return options
198
199
200 def attrs_to_dict(attrs):
201 attrs_dict = {}
202 for attribute in attrs:
203 config_entry = attribute.metadata[CONFIGURABLE_CONFIG_ENTRY]
204 attrs_dict[attribute.name] = {
205 'type': attribute.type,
206 'help': config_entry.description
207 }
208 return attrs_dict
  • I guess the purpose of this conversion to dict is to have a unified access to fields of the Attribute itself and the _ConfigEntry extension saved in the metadata field. However, this manual approach of copying single fields doesn't scale very well and a dict is not the nice interface for an object which fixed fields/keys. Better approaches would be to write

    • a function get_config_entry_option(attribute, name) that first tries to access attribute.metadata[CONFIGURABLE_CONFIG_ENTRY].name and if that fails falls back to attribute.name
    • write a wrapper class which does the same, e.g., config_entry = ConfigWrapper(attribute); print(config_entry.type, config_entry.help)
    • a wrapper class, which makes a clearer separation, e.g., config_wrapper = ConfigWrapper(attribute); print(config_wrapper.attribute.type, config_wrapper.config_entry.help)
  • Please register or sign in to reply
  • 257 259 return
    258 260
    259 261 log.info("Performing pre-scan setup")
    260 self.worker.pre_scan(data, substream_metadata)
    262 parameters = self._meta_to_parameters(substream_metadata['meta'])
    263 self.worker.pre_scan(data, substream_metadata, **parameters)
  • 270 273 finally:
    271 274 self._shutdown()
    272 275
    276 def _meta_to_parameters(self, metadata):
    277 parameters = {}
    278 if hasattr(self.worker, "Parameters"):
    279 parameters = attr.asdict(create_instance_from_configurable(self.worker.Parameters, metadata))
    280
    281 # check parameters type
  • added 1 commit

    • 9f5ccaa9 - Refactor parameters function

    Compare with previous version

  • 43 43 return flattened_entries
    44 44
    45 45
    46 def get_config_entry_option(attribute, name):
    47 value = None
    48 if hasattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name):
    49 value = getattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name)
    50 else:
    51 value = getattr(attribute, name)
    52 return value
    53
    54
    55 def check_type(parameter_class, parameters):
  • 44 44
    45 45
    46 def get_config_entry_option(attribute, name):
    47 value = None
    48 if hasattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name):
    49 value = getattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name)
    50 else:
    51 value = getattr(attribute, name)
    52 return value
    53
    54
    55 def check_type(parameter_class, parameters):
    56 for attribute in attr.fields(parameter_class):
    57 expected_type = get_config_entry_option(attribute, 'type')
    58 variable = getattr(parameters, attribute.name)
    59 if isinstance(variable, (list, str, dict, float, bool, int)):
  • 46 def get_config_entry_option(attribute, name):
    47 value = None
    48 if hasattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name):
    49 value = getattr(attribute.metadata[CONFIGURABLE_CONFIG_ENTRY], name)
    50 else:
    51 value = getattr(attribute, name)
    52 return value
    53
    54
    55 def check_type(parameter_class, parameters):
    56 for attribute in attr.fields(parameter_class):
    57 expected_type = get_config_entry_option(attribute, 'type')
    58 variable = getattr(parameters, attribute.name)
    59 if isinstance(variable, (list, str, dict, float, bool, int)):
    60 if expected_type != type(variable):
    61 raise TypeError(f"Variable {attribute.name} in metadata stream have wrong type. "
  • added 1 commit

    • 38f221a8 - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • added 1 commit

    • 93ecd34d - Restore python 3.5 compatobility

    Compare with previous version

  • added 1 commit

    Compare with previous version

  • Did I overlooks something, or fine now?

  • Tim Schoof
  • 195 195 d = d[k]
    196 196 d[keys[-1]] = value
    197 197 return options
    198
    199
    200 def attrs_to_dict(attrs):
  • You could also add a few tests for get_config_entry_option and check_type.

  • added 1 commit

    Compare with previous version

  • added 1 commit

    • 07265d23 - Add test for Parameters functionality

    Compare with previous version

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading