Unpredictable behavior of file caching

[FabianDK]

Dear nextflow devs,

We have experienced that the caching of the nextflow runs is sometimes behaving unpredictably. We are running nextflow on Ubuntu 20.04.2 LTS and NXF_VER=22.10.3, and this happened for several different pipelines and users.

To give an example, I was running a pipeline successfully and then tried to rerun it just with an additional .view() command in the workflow definition, but for some reason nextflow starts running the pipeline from the middle again. It even restarts when I remove the .view() command again.

In another unexplained case, I have modified the output of a successfully run process (change of simply just one path to a tuple of multiple vals and paths), and nextflow was running this process again. What is strange is that when I similarly modified the output definition of other processes, nextflow used the cached output files.

What could be the reason for this? Is there a way to check what went wrong by e.g. looking at the files in the cache folder?

[mribeirodantas]

These readings can be useful, Fabian:

• https://www.nextflow.io/blog/2022/caching-behavior-analysis.html
• https://www.nextflow.io/blog/2019/troubleshooting-nextflow-resume.html
• https://training.nextflow.io/basic_training/cache_and_resume/#resume-troubleshooting

[danoreper]

Hi Nextflow team,

I'm running into similar caching issues. But even after reading the above docs on caching and resume, I'm stuck.
In a bit more detail: despite using -resume, I'm seeing successful processes restart with the exact same hashes.

The workflow has changed between runs, but minimally: I've added subscribe statements to get some printouts.
But again, the hashes have remained the same! Are there any recommendations for how to go about debugging this sort of unexpected lack of caching?

[bentsherman]

If the hash is the same but the task isn't resumed, it suggests there is something wrong with the work directory which prevented the outputs from being recovered. If you run with trace logging enabled (nextflow -trace nextflow run ...) it will log why a cached task couldn't be recovered in this case.

By the way, those blog posts have been consolidated into a docs page which might also have some additional guidance:

https://nextflow.io/docs/latest/cache-and-resume.html