-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
support for partial workflow execution #54
Comments
Note: the new Yadage version should support partial workflow execution as well, so after we upgrade CWL and Yadage, we can take advantage of the new named steps feature in Serial, and we could add a generalised support for the three workflow engines for the partial execution functionality. |
as we add features, maybe the REANA docs should have a feature matrix showing which engines have which features |
@lukasheinrich Yes, we currently have "To pick a workflow engine" under https://reana.readthedocs.io/en/latest/userguide.html#capture-your-workflows We can enrich and publicise it better. |
For steps:
- environment: 'reanahub/reana-env-root6'
commands:
- mkdir -p results
- root -b -q 'code/gendata.C(${events},"${data}")' | tee gendata.log
- root -b -q 'code/fitdata.C("${data}","${plot}")' | tee fitdata.log This is only one step, that contains three commands. We should decide what is a step from a user perspective: All |
One step can consist of running several commands. The above example could be split into two steps, the "gendata" step (containing one command) and the "fitdata" step (containing the other command). But in theory there can be N steps and each step Si can consist of many commands... so running a step means running all the preceding steps and all of step's own commands. |
* Add target as operational option. * Addresses reanahub#54. Signed-off-by: Daniel Prelipcean <[email protected]>
I have made a pr tackling the last comment. If the implementation seems ok, I could go ahead for more functionality, e.g.: $ reana-client start -o step_start=1 -o step=3 |
* Add target as operational option. * Addresses reanahub#54. Signed-off-by: Daniel Prelipcean <[email protected]>
* Add target as operational option. * Addresses reanahub#54. Signed-off-by: Daniel Prelipcean <[email protected]>
* Add target as operational option. * Addresses reanahub#54. Signed-off-by: Daniel Prelipcean <[email protected]>
The 'cwl tool' functionality for target is actually the opposite of what we had in mind, i.e. for them We should think whether the engines should be consistent among each other (preferred). However, if this brings too much overhead, than some changes could be accepted, as the end user will use in the end only one workflow. |
* Add target as operational option. * Addresses reanahub#54. Signed-off-by: Daniel Prelipcean <[email protected]>
Similarly to CWL and Yadage, the Serial workflow engine should support partial workflow execution.
This will be useful for incremental workflow development and debugging, e.g. run preparatory steps 1-3 until the process succeeds, then run filtering steps 4-10 until success, and then run the plotting steps 11-13 until success.
If we don't have names for steps, the CLI could look like:
$ reana-client start --target step7
If we do introduce names for steps, the CLI could look like:
$ reana-client start --target myfit
See also reanahub/reana-client#202
The text was updated successfully, but these errors were encountered: