Should transport graph dirty_nodes be a 'set' rather than a 'list'?

[FyzHsn]

Every time transport graph nodes are initialized or updated, the corresponding node_id gets appended to the result.lattice.transport_graph.dirty_nodes attribute. The latter attribute is a list. When printing out the dirty nodes for a workflow, using the following code snippet:

import covalent as ct

@ct.electron
def task_1(x, y):
    return {'key_1': x * y, 'key_2': x + y}


@ct.electron
def task_2(z):
    return z


@ct.lattice
def workflow(a, b):
    res_1 = task_1(a, b)["key_1"]
    res_2 = task_2(res_1)
    return res_2

dispatch_id = ct.dispatch(workflow)(a=1, b=2)
result = ct.get_result(dispatch_id, wait=False)
print(result.lattice.transport_graph.dirty_nodes)

we get,

[0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4]

Updating the node metadata in the database, each time we have duplicate node_id entries seems very inefficient. Should we rather convert dirty _nodes to a set in order to avoid the duplication issue and hence writing the same information to the database multiple times?

Each time a node is added, in the transport graph object, we simply perform self.dirty_nodes.add(node_id) and each time the node data is written to the database, we do result.lattice.transport_graph.dirty_nodes.remove(node_id). If we make this change, the dirty nodes above would simply be:

{0, 1, 2, 3, 4}

[FyzHsn]

A quick workaround is to just use set(transport_graph.dirty_nodes) when using it. This avoids having to change the TransportGraph implementation which might have unintended consequences.