Example Usage

Example Usage#

Initializing and Basic Operations#

Firstly, dml_util expects both DML_S3_BUCKET and DML_S3_PREFIX to be set. This is where we’ll store data. For this example, we set it to random nonsense (we won’t be using s3 here).

from contextlib import redirect_stderr, redirect_stdout

from daggerml import  Dml, Error
from dml_util import funkify

# Create a dml instance
dml = Dml()

# Create a new DAG
dag = dml.new('my_dag', 'example dag')
# Put simple values
node1 = dag._put(list(range(5)))
# you can always get the value of a node
node1.value()

[0, 1, 2, 3, 4]

Because our node is a list, we can do list stuff with it…

# Indexing and slicing
print(f"{node1[0] = } -- {node1[0].value()}")

# Create complex structures
complex_node = dag._put({'x': node1, 'y': 'z'})

# Iterate and access items
for k, v in complex_node.items():
    print(f"{k = !s}, {v = } -- {v.value() = !r}")

# Commit the DAG
dag.result = complex_node

node1[0] = Node(node/231a9420336d1a186ff152eb2dfffb9c) -- 0

k = x, v = ListNode(node/dc7b6be9cf9f17f40ea73447470f8ef9) -- v.value() = [0, 1, 2, 3, 4]

k = y, v = Node(node/a57049d7237acea0c1914c4d80d7dfa4) -- v.value() = 'z'

Calling functions#

Let’s define a function that takes \(n\ge 2\) arguments, sums up all but the last, and then divides by the last.

@funkify
def my_funk(dag):
    numer = sum(dag.argv[1:-1].value())
    return numer / dag.argv[-1].value()

dag = dml.new('funk-dag', 'dag to build a funk')
dag.my_funk = my_funk
with redirect_stdout(None), redirect_stderr(None):
    result = dag.my_funk(*range(5))
print(f"{result = } -- {result.value() = }")

result = Node(node/8a6aa0c8ab4c821b2afb2d6321832327) -- result.value() = 1.5

And we can save this as our dag’s “result” and then import it later without having to consider the infrastructure it’s running on, or anything else.

dag.result = dag.my_funk

try:
    with dml.new("new-dag", "my cool new dag that will unforunately fail") as dag:
        dag.funk = dml.load("funk-dag").result
        with redirect_stdout(None), redirect_stderr(None):
            dag.funk(1, 2, 3, 4, 0, name="bad-guy")  # note we're dividing by zero!
except Error as e:
    print("ruh roh! We divided by zero...")
    print(e)

ruh roh! We divided by zero...
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/dml_util/funk.py", line 207, in aws_fndag
    yield dag
  File "/tmp/dml.41eg9jt1/script.py", line 13, in <module>
    res = my_funk(dag)
  File "/tmp/dml.41eg9jt1/script.py", line 6, in my_funk
    return numer / dag.argv[-1].value()
           ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: division by zero

And because we used the dag as a context manager, that error is stored as the dag’s result, and we can query it later.

print(dml("dag", "describe", "new-dag")["error"])

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.13.5/x64/lib/python3.13/site-packages/dml_util/funk.py", line 207, in aws_fndag
    yield dag
  File "/tmp/dml.41eg9jt1/script.py", line 13, in <module>
    res = my_funk(dag)
  File "/tmp/dml.41eg9jt1/script.py", line 6, in my_funk
    return numer / dag.argv[-1].value()
           ~~~~~~^~~~~~~~~~~~~~~~~~~~~~
ZeroDivisionError: division by zero

Or in another dag:

dag = dml.new("newest")
try:
    dml.load("new-dag")["bad-guy"].value()
except Error as e:
    print("reraised?", "ZeroDivisionError" in str(e))

reraised? True

This funkify decorator can be used to send your code into the cloud, or have a function run in a different environment, or do whatever you want.

For more, read the examples on the daggerml and dml-util repos.

Contents