dml_util.core.config

dml_util.core.config#

Configuration management for DaggerML utilities.

DaggerML adapters are configured in two main ways: 1. Environment variables: The EnvConfig class loads configuration from environment variables. 2. Input data: Typically passed in via stdin, the InputConfig class stores the data passed from dml

Classes

EnvConfig(s3_bucket, s3_prefix, log_group, ...)

ENVVARS for a DaggerML run.

InputConfig(cache_path, cache_key, kwargs, dump)

Configuration for input data.
class dml_util.core.config.EnvConfig(s3_bucket, s3_prefix, log_group, run_id, debug)[source]#

Bases: object

ENVVARS for a DaggerML run.

This class is used to configure the DaggerML adapter and is typically specified via environment variables.

Parameters:

  • s3_bucket (str)

  • s3_prefix (str)

  • log_group (str)

  • run_id (str)

  • debug (bool)

s3_bucket#

S3 bucket for the data store.

Type:

str

s3_prefix#

S3 prefix for the data store.

Type:

str

debug#

Debug flag.

Type:

bool

run_id#

UUID identifying this run.

Type:

str

log_group#

Log group for the current function.

Type:

str

debug: bool#
dumps()[source]#

Serialize the configuration to a JSON string.

Return type:

str

classmethod from_env(debug=False)[source]#

Load configuration from environment variables.

Parameters:

debug (bool)

Return type:

EnvConfig

classmethod loads(data)[source]#

Deserialize the configuration from a JSON string.

Parameters:

data (str)

Return type:

EnvConfig

log_group: str#
run_id: str#
s3_bucket: str#
s3_prefix: str#
to_dict()[source]#

Convert the configuration to a dictionary.

Return type:

dict

to_envvars()[source]#

Convert the configuration to environment variables.

Return type:

dict

class dml_util.core.config.InputConfig(cache_path, cache_key, kwargs, dump)[source]#

Bases: object

Configuration for input data.

This class is used to specify the input data configuration for a DaggerML run.

Parameters:

  • cache_path (str)

  • cache_key (str)

  • kwargs (dict | None)

  • dump (str)

cache_path#

Path to the cache directory.

Type:

str

cache_key#

The execution’s cache key.

Type:

str

kwargs#

The function’s specific data. May include a sub-adapter and URI.

Type:

dict | None

dump#

The dag dump.

Type:

str

cache_key: str#
cache_path: str#
dump: str#
get_sub()[source]#

Get sub-adapter, URI, and kwargs for the sub function’s stdin.

kwargs: dict | None#
Contents