writers package¶
Submodules¶
writers.abstract_file_writer module¶
Abstract class to write some data to a some file
- Author(s):
Carl Anderson (carl.anderson@weightwatchers.com)
-
class
writers.abstract_file_writer.
AbstractFileWriter
(configuration, instance_name)¶ Bases:
primrose.base.writer.AbstractWriter
write some data to a some file
-
static
necessary_config
(node_config)¶ Necessary inputs for file writers:
- Parameters
node_config (dict) – set of parameters / attributes for the node
Note
key: key that identifies object to write dir: directory to write csv (must already exist) filename: name of file to be written
- Returns
set of necessary configuration keys
-
static
writers.csv_writer module¶
write some dataframe to a local CSV file.
- Author(s):
Carl Anderson (carl.anderson@weightwatchers.com)
-
class
writers.csv_writer.
CsvWriter
(configuration, instance_name)¶ Bases:
primrose.writers.abstract_file_writer.AbstractFileWriter
write some dataframe to a local CSV file.
-
get_optional_config
()¶ Optionally get kwargs to pass to pandas csv reader.
Notes
kwargs (dict): dictionary of kwargs key-value pairs
Example
“csv_reader”: { “class”: “CsvReader”, “filename”: “data/mydata.csv”, “kwargs”: { “header”: None, “sep”: “:” } }
-
run
(data_object)¶ write some dataframe to a local CSV file
- Returns
tuple containing:
data_object (DataObject): instance of DataObject
terminate (bool): terminate the DAG?
- Return type
(tuple)
-
writers.dill_writer module¶
write some data to a local file, serialized with dill
- Author(s):
Mike Skarlinski (michael.skarlinski@weightwatchers.com)
-
class
writers.dill_writer.
DillWriter
(configuration, instance_name)¶ Bases:
primrose.writers.abstract_file_writer.AbstractFileWriter
write some specified data object to a dill file
-
run
(data_object)¶ serialize some data to a local filesystem using Dill
Note
this will write object under config[‘key’] in data_object
- Parameters
data_object (DataObject) –
- Returns
tuple containing:
data_object (DataObject): instance of DataObject
terminate (bool): terminate the DAG?
- Return type
(tuple)
-
writers.file_writer module¶
write some data to a local file.
- Author(s):
Carl Anderson (carl.anderson@weightwatchers.com)
-
class
writers.file_writer.
FileWriter
(configuration, instance_name)¶ Bases:
primrose.writers.abstract_file_writer.AbstractFileWriter
write some data object to a local file
-
run
(data_object)¶ write some data to a local file
Note
this will write object under config[‘key’] in data_object
- Returns
tuple containing:
data_object (DataObject): instance of DataObject
terminate (bool): terminate the DAG?
- Return type
(tuple)
-
writers.s3_writer module¶
write some data to a S3 bucket, via a local file.
- Author(s):
Carl Anderson (carl.anderson@weightwatchers.com)
-
class
writers.s3_writer.
S3Writer
(configuration, instance_name)¶ Bases:
primrose.base.writer.AbstractWriter
a writer that writes a dataframe to disk and from there to S3 bucket
-
static
necessary_config
(node_config)¶ Necessary keys for S3Writer object
- Parameters
node_config (dict) – set of parameters / attributes for the node
node_config – set of parameters / attributes for the node
Note
dir: directory to write, then upload file bucket_name: s3 bucket where the file will be written bucket_filename: filename for the s3 bucket
- Returns
set of necessary config fields
-
run
(data_object)¶ write some data to a local CSV file then from there to S3 bucket
- Returns
tuple containing:
data_object (DataObject): instance of DataObject
terminate (bool): terminate the DAG?
- Return type
(tuple)
-
static
writers.serializer module¶
write some data to a local or gcs file, serialized with dill or pickle.
- Author(s):
Mike Skarlinski (michael.skarlinski@weightwatchers.com) Brian Graham (brian.graham@weightwatchers.com)
-
class
writers.serializer.
Serializer
(configuration, instance_name)¶ Bases:
primrose.writers.abstract_file_writer.AbstractFileWriter
Serialize some object to a local file.
-
SUPPORTED_SERIALIZERS
= {'dill': <module 'dill' from '/anaconda3/envs/ds-user-models/lib/python3.6/site-packages/dill/__init__.py'>, 'pickle': <module 'pickle' from '/anaconda3/envs/ds-user-models/lib/python3.6/pickle.py'>}¶
-
static
necessary_config
(node_config)¶ Returns the necessary configuration keys for the Serializer object:
- Parameters
node_config (dict) – set of parameters / attributes for the node
Note
serializer (str): ‘dill’ or ‘pickle’
- Returns
set of necessary configuration keys
-
run
(data_object)¶ serialize some data to a local filesystem using Dill
Note
this will write object under config[‘key’] in data_object
- Parameters
data_object (DataObject) –
- Returns
tuple containing:
data_object (DataObject): instance of DataObject
terminate (bool): terminate the DAG?
- Return type
(tuple)
-