Can I (and does it make sense to) execute electrons in arbitrary python environments?

[svandenhaute]

The workflow I'm aiming to implement combines a variety of tools which all have python interfaces but which have been compiled with different toolchains and on different flavors of linux, making them in general incompatible.
Ideally, each electron would be able to run in its own environment, with its own python version and libraries.

For example; one electron performs a quantum chemistry calculation using some difficult-to-compile piece of fortran code, which is installed on a SLURM-accessible HPC with a Python 3.4 interface. Another electron uses a recent version of Torch in combination with Python 3.10 because library X requires it. How can I compose a workflow with both electrons in there?
Could the DepsBash functionality be used for this? I don't fully see how it could actually change the Python environment.

[santoshkumarradha]

Hi @svandenhaute , in theory, you can. But note that all electrons must be constructed in the same environment. If you there are conflicting packages, the ideal way would be to use @ct.leptons.bash (Or DepsBash as you mentioned) to evoke the python shell and a script to run it on a remote backend, whereby we bypass the need to pickle the function/code while building the lattice.

I would personally try directly, even with version conflict, just using the normally pickled electrons and making sure it does not work first as most of the versions (except for edge cases) as long as the python version/package versions ~ match each other, things would go well.

Do let us know if you are unable to do it, we can cook some example scripts to help you out.

PS: Covalent does not support python 3.4 and needs > python3.8 it is mostly to do with how multiprocessing changed completely from python >=3.8.