Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to restore the output files of each step in the remote machine when using the DispatcherExecutor? #800

Open
gdbhcxhmjk-z opened this issue May 16, 2024 · 1 comment

Comments

@gdbhcxhmjk-z
Copy link

The overall workflow can be executed successfully but the output files of each step can only be downloaded from the website "workflow.deepmodeling.com". How can I retore the output files of each step in the remote machine?
All the computing results will be cleaned when the calculation is finished. I'm wondering if there's a way to save the files in the appointed file instead of the temporary files with the hash name.
Here is my DispatcherExecutor:
DispatcherExecutor(
host=host,
port=port,
username=username,
password=password,
remote_root="/data/home/tmp",
image_pull_policy="IfNotPresent",
machine_dict={
"batch_type": "slurm",
"local_root": "/data/home/usr",
"context_type": "SSHContext",
"clean_asynchronously":false},
resources_dict={
"number_node": 1,
"cpu_per_node": 10,
"gpu_per_node": 1,
"queue_name": "gpu",
"group_size": 1,
"custom_flags": [
"#SBATCH --time=0-1000:00:00"
],
"source_list": ["activate deepmd-kit"],
"batch_type": "Slurm"},
merge_sliced_step=False
)
The parameter "local_root" doesn't help. There is nothing in the local_root.

@zjgemi
Copy link
Collaborator

zjgemi commented May 28, 2024

There is an argument clean in DispatcherExecutor. Set it to False to preventing cleaning remote working directories.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants