configuration package

Submodules

configuration.configuration module

configuration.configuration_dag module

configuration.util module

Module contents

Module to implement a Configuration parser which enhances parsing functionality of configparser

Author(s):

Michael Skarlinski (michael.skarlinski@weightwatchers.com)

Carl Anderson (carl.anderson@weightwatchers.com)

class configuration.Configuration(config_location, is_dict_config=False, dict_config=None)

Bases: object

Stores user defined configuration for primrose job

check_config()

check the configuration as much as we can as early as we can

Raises

various exceptions if any checks fail

check_metadata()

checks some dependencies among metadata keys

Raises

ConfigurationError is issues found

check_sections()

Check that all the sections in implementation are supported ones. Either the user supplied metata.section_registry, or they are using default sections

Raises
  • ConfigurationError if declaring metadata.section_registry and sections from implementation were not found in metadata

  • or vice versa, or if using default operations but sections found that were not supported

config_for_instance(instance_name)

get the configuration for a given node / instance_name

Returns

JSON chunk for this instance

dict_raise_on_duplicates(ordered_pairs)

Reject duplicate keys in JSON string, ie. sections and node names.

Parameters

ordered_pairs (list) – list of key:values from the config Example: ordered_pairs [(‘class’, ‘CsvReader’), (‘filename’, ‘data/tennis.csv’), (‘destinations’, [‘write_output’])] ordered_pairs [(‘read_data’, {‘class’: ‘CsvReader’, ‘filename’: ‘data/tennis.csv’, ‘destinations’: [‘write_output’]})]

Returns

dictionary of key (node type) and value (node name)

Return type

dictionary (dict)

static perform_any_config_fragment_substitution(config_str)

Given some configuration file content string, look for subtitutions given by $$FILE=path/to/config/file/fragment.json$$ and make the replacements using the filenames provided For example: { $$FILE=/tmp/metadata.json$$ “implementation_config”: { $$FILE= config/read_write_fragment.json $$ } } will inject /tmp/metadata.json into the 2nd line of that config.

Parameters

config_str (str) – content of some configuration file that may or may not contain substition variables

Returns

the post-substituted configuration string

Return type

config_str (str)

sections_in_order()

Return list of section names in order, either explicitly from metadata or from default Enum order

Note

If there is a non-empty section_run list in metadata return that elif there is a non-empty section_registry in metadata return that otherwise return sections present from default OperationType enum.

We need this method because the config sections are a dictionary not a list so we can’t guarantee order of keys. This method imposes an expected order.

Returns

tuple containing:

section names (list): list of sections

source (str): where did the list come from? section_run, section_registry, or default?

Return type

(tuple)