nextflow inspect not working with nf-core pipelines

[stevekm]

duplicate of nf-core/scrnaseq#347 because I am not sure which project this is more appropriate for.

When I use nextflow inspect with a regular pipeline, it works fine;

$ nextflow inspect "https://github.com/stevekm/bobs-pipeline" -r "master"
{
    "processes": [
        {
            "name": "MAKE_HAMBURGERS:ADD_BOTTOM_BUN",
            "container": "ubuntu:22.04"
        },
        {
            "name": "MAKE_HAMBURGERS:ADD_TOP_BUN",
            "container": "ubuntu:22.04"
        },
        {
            "name": "MAKE_HAMBURGERS:ADD_CHEESE",
            "container": "ubuntu:22.04"
        },
        {
            "name": "MAKE_HAMBURGERS:ADD_BURGER_PATTY",
            "container": "ubuntu:22.04"
        }
    ]
}

But when I try to use nextflow inspect on nf-core pipelines, it breaks a lot with errors about required parameters, even if I use -ignore-errors

$ nextflow inspect "https://github.com/nf-core/rnaseq" -r "master" -ignore-errors
ERROR ~ ERROR: Validation of pipeline parameters failed!


The following invalid input values have been detected:

* Missing required parameter: --input
* Missing required parameter: --outdir

# with a dummy params file
$ nextflow inspect "https://github.com/nf-core/rnaseq" -r "master" -ignore-errors -params-file params.json
Genome fasta file not specified with e.g. '--fasta genome.fa' or via a detectable config file.


# same errors with scrnaseq pipelines

$ nextflow inspect "https://github.com/nf-core/scrnaseq" -r "master" -i
ERROR ~ ERROR: Validation of pipeline parameters failed!

Any ideas on how to get this to work in a generic manner that works with all pipelines?

• nextflow version 24.04.2.5914

[pditommaso]

The nextflow inspect command executes the pipeline in a kind of "dry-run-mode" behind the scene. As such all parameters expected by the pipeline should be provided. You can use the test profile with nf-core pipelines

[stevekm]

That does not really work with my intended usage. I ultimately wanted to be able to query the list of pipelines from Nextflow Tower, or query e.g. the latest Run from Tower, and then nextflow inspect each pipeline it so that I can docker pull, push, etc., all the containers as needed, for misc. tasks.

Being required to have a-priori knowledge of what each pipeline is, and then supply a pre-curated input params for each pipeline, does not work. The pipelines could be custom pipelines, they could be nf-core pipelines, they could some other public pipelines, etc.. I was under the assumption that nextflow inspect would "just work" regardless of what type of pipeline it is, in order to retrieve the list of containers. I guess that is not the case?

[pditommaso]

I'm afraid, but that's how it works. Either you provide the expected parameters or you create a configuration profile fulfilling all required params for testing purpose (suggested approach)

[bentsherman]

@stevekm I think we'll be able to support your use case at some point in the future, by listing all containers used by the pipeline regardless of the params.

[stevekm]

Thanks for the feedback. Ultimately we need to be able to get this list of all containers used in a pipeline, in an easy-to-automate and generic manner. So that we can push them to our container registry. Being required to pre-create a config for every pipeline will not work, because we have no clue what params will be required in the users' pipelines. The users themselves likely will not know either for publicly imported pipelines.

Additionally, I actually tried doing this, and it was effectively impossible anyway. It seems like a lot of pipelines that are using the nf-validation or nf-schema plugin, have a lot of draconian sanity-checks built in, such that there is no way to "test" the pipeline, because it requires that a full set of input files exists, including all reference files, requires that things like sample sheets with specific headers exist, requires that directories exist in specific manner, etc.. Whatever arbitrary restriction was applied in the pipeline input files and params, becomes a blocker for being able to use nextflow inspect in its current implementation, it seems. This goes beyond just nf-core pipelines that usually have a "test" profile, but applies to any pipeline that the developer implemented input sanity checking for, essentially breaking nextflow inspect.

when it works, nextflow inspect is just about perfect, but not being able to suppress the errors that arise from arbitrary pipeline input validation restrictions makes it unusable for the situations that matter the most; when you arent familiar with the pipeline and just need a list of containers from it

thanks :)