How to get the list in a channel from input file (yaml) as an argument?

[justinjj24]

Hi,
I have workflow which reads the list samples from an input file params.yml by running command line with -params-file params.yml and it works.

I need to run this through an argument --input_list params.yml instead of -params-file or any otherway of giving the params.yaml to make nextflow to recognize this as params-file function

Reason being I'm submitting the jobs to aws batch from tower.nf dashboard which has a field Pipeline parameters equivalent to the Nextflow -params-file option allowing to paste json/yaml format of the parameters, but it has number of characters limition, so can't paste large number of samples list. To overcome this I need to pass the param.yml as an argument in the Pipeline parameters field

--input_list: s3://my-bucket/params.yml

samples:
-
 biosample_id: WGS001
 bam: WGS001.bam
-
 biosample_id: WGS0002
 bam: WGS0002.bam

Workflow works well in the following code by getting the list from the channel samples and used by number of process in the workflow. The job run in the commandline with -params-file params.ymlas well tower.nf dashboard with small number of samples list within the allowed character limit...

 Channel
    .fromList( params.samples )
    .ifEmpty { ['biosample_id': params.biosample_id, 'bam': params.bam] }
    .set { samples }

...
...

Channel
    samples.branch { rec ->
    ....

Channel
    samples.map { it.biosample_id }
    ....

Thanks in advance!

[bentsherman]

Here is how you can load the yaml file into a channel:

import groovy.yaml.YamlSlurper

params.inputs_list = "inputs.yaml"

workflow {
    inputs = new YamlSlurper().parse(params.inputs_list as File)

    Channel.fromList(inputs['samples'])
    | view
}

[justinjj24]

@bentsherman

1. Facing another issue, I couldn't run it succesfully in tower.nf which is my requirement. I use nextflowVersion = '!>=23.09.1-edge' in nextflow.config but nftower could get 23.08.1-edge!

Exit status : null
Error message : Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge
Error report : Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge
Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge

Workflow execution completed unsuccessfully
The workflow execution failed to start. Exit status: 1

Essential container in task exited

N E X T F L O W  ~  version 23.08.1-edge
Pulling c-BIG/... ...
 downloaded from https://github.com/c-BIG/...
Launching `https://github.com/c-BIG/...` [npm-sample-qc-CE-fusionv2-inputs-list] DSL2 - revision: 9ff81408f2 [feature/fix-input-config-params]
N E X T F L O W  ~  version 23.08.1-edge 5874
NPM-sample-qc  ~  version 0.10.4
User name    : root
Command Line : nextflow run 'https://github.com/c-BIG/...' -name CE-fusionv2-inputs-list -params-file 'https://api.tower.nf/ephemeral/bWVqBBXv93So0qWycZaHhw.yaml' -with-tower -r feature/fix-input-config-params
Project Dir  : /.nextflow/assets/c-BIG/....
Launch Dir   : /
Work Dir     : /npm-grids/work-fusionv2-inputs-list
Results Dir  : s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2-inputs-list
Info Dir     : s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2-inputs-list/pipeline_info
Profile      : standard
Workflow execution stopped with the following message:
Exit status   : null
Error message : Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge
Error report  : Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge
Nextflow version 23.08.1-edge does not match workflow required version: >=23.09.1-edge
Saving cache: .nextflow/cache/68a600b0-b001-4c4c-af44-76fab57af087 => s3://npm-grids/work-fusionv2-inputs-list/.nextflow/cache/68a600b0-b001-4c4c-af44-76fab57af087

2. I change to nextflowVersion = '!>=23.08.1-edge' attempt to rerun but it couldn't find the input file! the input path on s3 contain single '/' - s3:/ But the 'Result Dir' path seems getting correctly with 'double '//' - s3://

The params in the Pipeline parameter field as

inputs_list: s3://npm-grids/gnanakkan/npm-sample-qc/inputs.yaml
publish_dir: s3://npm-grids/gnanakkan/npm-sample-qc/sg100k-cram-test-v0104_precise_CE-fusionv2-inputs-list

OR

{
  "publish_dir": "s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2-inputs-list",
  "inputs_list": "s3://npm-grids/inputs.yaml"
}

Exit status : null
Error message : s3:/npm-grids/inputs.yaml
Error report : s3:/npm-grids/inputs.yaml
ERROR ~ No such file or directory: s3:/npm-grids/inputs.yaml

Workflow execution completed unsuccessfully
The workflow execution failed to start. Exit status: 1

Essential container in task exited

N E X T F L O W  ~  version 23.08.1-edge
Pulling c-BIG/... ...
 downloaded from https://github.com/c-BIG/...
Launching `https://github.com/c-BIG/....` [cram-fusionv2-inputs-list] DSL2 - revision: da3234c36c [feature/fix-input-config-params]
N E X T F L O W  ~  version 23.08.1-edge 5874
NPM-sample-qc  ~  version 0.10.4
User name    : root
Command Line : nextflow run 'https://github.com/c-BIG/....' -name cram-fusionv2-inputs-list -params-file 'https://api.tower.nf/ephemeral/eQs5SrX8hJnjqdntCceYjA.yaml' -with-tower -r feature/fix-input-config-params
Project Dir  : /.nextflow/assets/c-BIG/....
Launch Dir   : /
Work Dir     : /npm-grids/workdir-cram-fusionv2-inputs-list
Results Dir  : s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2
Info Dir     : s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2/pipeline_info
Profile      : standard
Workflow execution stopped with the following message:
Exit status   : null
Error message : s3:/npm-grids/inputs.yaml
Error report  : s3:/npm-grids/inputs.yaml
ERROR ~ No such file or directory: s3:/npm-grids/inputs.yaml
 -- Check script '.nextflow/assets/c-BIG/..../main.nf' at line: 118 or see 'nf-1j201C8I7qaqff.log' file for more details
Saving cache: .nextflow/cache/e7674d61-988b-4ec7-af4c-0c7972bd4e5e => s3://npm-grids/workdir-cram-fusionv2-inputs-list/.nextflow/cache/e7674d61-988b-4ec7-af4c-0c7972bd4e5e

3. The same error I'm getting running the workflow locally also with both the version23.09.1-edge and 23.08.1-edge if accessing the input yaml file from s3 bucket. It only parse single '/' - s3:/
   nftower run - inputs_list: s3://npm-grids/inputs.yaml
   local run - --inputs_list s3://npm-grids/inputs.yaml

Both nextflow edge version works fine if the inputs.yaml at local disk --inputs_list ./inputs.yaml

[bentsherman]

It looks like in the JSON params there are three slashes:

{
  "publish_dir": "s3://npm-grids/100k-cram-test-v0104_precise_CE-fusionv2-inputs-list",
  "inputs_list": "s3:///npm-grids/inputs.yaml"
}

[justinjj24]

No, it failed with regular s3 pattern of two slashes s3://npm-grids/inputs.yaml -> ERROR ~ No such file or directory: s3:/npm-grids/inputs.yaml
The three slashes was just a try to see whether it could return with double slashes!

[bentsherman]

Oh I think I know... my original code assumed a local file, but if it's an S3 object then you should replace:

params.inputs_list as File

with

file(params.inputs_list)

Actually the second one should work in both cases.

[justinjj24]

yeah, good catch... works well with s3 path s3://npm-grids/inputs.yaml
Thank you so much
Tested with 23.08.1-edge and 23.09.1-edge

import groovy.yaml.YamlSlurper

inputs = new YamlSlurper().parse(file(params.inputs_list))

    Channel
        .fromList(inputs['samples'])
        .ifEmpty { ['biosample_id': params.biosample_id, 'aln': params.aln] }
        .set { samples }