primrose.writers package

Submodules

primrose.writers.abstract_file_writer module

Abstract class to write some data to a some file

Author(s):

Carl Anderson (carl.anderson@weightwatchers.com)

class primrose.writers.abstract_file_writer.AbstractFileWriter(configuration, instance_name)

Bases: primrose.base.writer.AbstractWriter

write some data to a some file

static necessary_config(node_config)

Necessary inputs for file writers:

Parameters

node_config (dict) – set of parameters / attributes for the node

Note

key: key that identifies object to write dir: directory to write csv (must already exist) filename: name of file to be written

Returns

set of necessary configuration keys

primrose.writers.csv_writer module

write some dataframe to a local CSV file.

Author(s):

Carl Anderson (carl.anderson@weightwatchers.com)

class primrose.writers.csv_writer.CsvWriter(configuration, instance_name)

Bases: primrose.writers.abstract_file_writer.AbstractFileWriter

write some dataframe to a local CSV file.

get_optional_config()

Optionally get kwargs to pass to pandas csv reader.

Notes

kwargs (dict): dictionary of kwargs key-value pairs

Example

“csv_reader”: { “class”: “CsvReader”, “filename”: “data/mydata.csv”, “kwargs”: { “header”: None, “sep”: “:” } }

run(data_object)

write some dataframe to a local CSV file

Returns

tuple containing:

data_object (DataObject): instance of DataObject

terminate (bool): terminate the DAG?

Return type

(tuple)

primrose.writers.dill_writer module

write some data to a local file, serialized with dill

Author(s):

Mike Skarlinski (michael.skarlinski@weightwatchers.com)

class primrose.writers.dill_writer.DillWriter(configuration, instance_name)

Bases: primrose.writers.abstract_file_writer.AbstractFileWriter

write some specified data object to a dill file

run(data_object)

serialize some data to a local filesystem using Dill

Note

this will write object under config[‘key’] in data_object

Parameters

data_object (DataObject) –

Returns

tuple containing:

data_object (DataObject): instance of DataObject

terminate (bool): terminate the DAG?

Return type

(tuple)

primrose.writers.file_writer module

write some data to a local file.

Author(s):

Carl Anderson (carl.anderson@weightwatchers.com)

class primrose.writers.file_writer.FileWriter(configuration, instance_name)

Bases: primrose.writers.abstract_file_writer.AbstractFileWriter

write some data object to a local file

run(data_object)

write some data to a local file

Note

this will write object under config[‘key’] in data_object

Returns

tuple containing:

data_object (DataObject): instance of DataObject

terminate (bool): terminate the DAG?

Return type

(tuple)

primrose.writers.s3_writer module

write some data to a S3 bucket, via a local file.

Author(s):

Carl Anderson (carl.anderson@weightwatchers.com)

class primrose.writers.s3_writer.S3Writer(configuration, instance_name)

Bases: primrose.base.writer.AbstractWriter

a writer that writes a dataframe to disk and from there to S3 bucket

static necessary_config(node_config)

Necessary keys for S3Writer object

Parameters
  • node_config (dict) – set of parameters / attributes for the node

  • node_config – set of parameters / attributes for the node

Note

dir: directory to write, then upload file bucket_name: s3 bucket where the file will be written bucket_filename: filename for the s3 bucket

Returns

set of necessary config fields

run(data_object)

write some data to a local CSV file then from there to S3 bucket

Returns

tuple containing:

data_object (DataObject): instance of DataObject

terminate (bool): terminate the DAG?

Return type

(tuple)

primrose.writers.serializer module

write some data to a local or gcs file, serialized with dill or pickle.

Author(s):

Mike Skarlinski (michael.skarlinski@weightwatchers.com) Brian Graham (brian.graham@weightwatchers.com)

class primrose.writers.serializer.Serializer(configuration, instance_name)

Bases: primrose.writers.abstract_file_writer.AbstractFileWriter

Serialize some object to a local file.

SUPPORTED_SERIALIZERS = {'dill': <module 'dill' from '/anaconda3/envs/ds-user-models/lib/python3.6/site-packages/dill/__init__.py'>, 'pickle': <module 'pickle' from '/anaconda3/envs/ds-user-models/lib/python3.6/pickle.py'>}
static necessary_config(node_config)

Returns the necessary configuration keys for the Serializer object:

Parameters

node_config (dict) – set of parameters / attributes for the node

Note

serializer (str): ‘dill’ or ‘pickle’

Returns

set of necessary configuration keys

run(data_object)

serialize some data to a local filesystem using Dill

Note

this will write object under config[‘key’] in data_object

Parameters

data_object (DataObject) –

Returns

tuple containing:

data_object (DataObject): instance of DataObject

terminate (bool): terminate the DAG?

Return type

(tuple)

Module contents