Replies: 4 comments 26 replies
-
|
In general it is not possible to run multiple instances of Nextflow from the same launch directory, because each instance will acquire a lock on the |
Beta Was this translation helpful? Give feedback.
-
|
@bentsherman by saying "If you want to run many workflows concurrently, you need to launch each workflow from a different directory" do you mean I have to have a nextflow installed in a different folder? |
Beta Was this translation helpful? Give feedback.
-
|
no, you just need to launch each pipeline in a different directory |
Beta Was this translation helpful? Give feedback.
-
|
@bentsherman Thank you. I will run the pipeline with different datasets and drop the output in different folders. |
Beta Was this translation helpful? Give feedback.
-
|
Hello @bentsherman. I submitted 2 datasets and dropped the outputs into different folders. (I am running the analysis on a Beluga cluster) N E X T F L O W ~ version 23.04.3 Common reasons for this error are:
You can see which process is holding the lock file by using the following command:
-- Check '.nextflow.log' file for details |
Beta Was this translation helpful? Give feedback.
-
|
Did you launch them from different directories? |
Beta Was this translation helpful? Give feedback.
-
|
Hello, @bentsherman!
Could you, please explain to me how to launch from a different directory if
I have wf-pore-c-0.1.0 installed just in one place and you told me I don't
need to install a WF in other folder.
Best,
Oksana
…On Tue, 12 Dec 2023 at 10:25, Ben Sherman ***@***.***> wrote:
Did you launch them from different directories?
—
Reply to this email directly, view it on GitHub
<#3987 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AXZT5EVPGWNKZSLX7OB6METYJBZOXAVCNFSM6AAAAAAYWHMWU6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TQMZSGE3DG>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
Oksana
|
Beta Was this translation helpful? Give feedback.
-
|
Oh I see what you mean now. The best practice here is to not clone the pipeline code manually. Instead you can do something like: nextflow run epi2me-labs/wf-pore-c -r 0.1.0in any directory, and Nextflow will clone the repo into cd run1 ; nextflow run epi2me-labs/wf-pore-c -r 0.1.0 --input ./inputs1
cd run2 ; nextflow run epi2me-labs/wf-pore-c -r 0.1.0 --input ./inputs2
# ...Each run will use the same copy of your pipeline code in |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @bentsherman for the explanations. as follow: module load apptainer/1.1.8 java/17.0.2 nextflow/23.04.3 --fastq /home/scratch/45G.fastq If I understand you correctly now I need to modify it like this for the r1: module load apptainer/1.1.8 java/17.0.2 nextflow/23.04.3 nextflow run /home/oksanak/scratch/wf-pore-c-0.1.0/wf-pore-c/main.nf -r 0.1.0 -c nextflow.config \ --fastq /home/scratch/45G.fastq And for the r2 i have to be somewhere other than /home/scratch/wf-pore-c-0.1.0/wf-pore-c and submit the following .sh script: module load apptainer/1.1.8 java/17.0.2 nextflow/23.04.3 nextflow run /home/oksanak/scratch/wf-pore-c-0.1.0/wf-pore-c/main.nf -r 0.1.0 -c nextflow.config \ --fastq /home/scratch/60G.fastq Am I right? |
Beta Was this translation helpful? Give feedback.
-
|
sure that should work |
Beta Was this translation helpful? Give feedback.
-
|
Hello @bentsherman Unfortunately, it didn't work. N E X T F L O W ~ version 23.04.3 |
Beta Was this translation helpful? Give feedback.
-
|
Because you have already downloaded the pipeline code and are using the absolute path of the main script, you don't need to specify You should either do: # it will download the repo for you into $HOME/.nextflow/assets
nextflow run epi2me-labs/wf-pore-c -r 0.1.0or nextflow run /home/oksanak/scratch/wf-pore-c-0.1.0/wf-pore-c/main.nf |
Beta Was this translation helpful? Give feedback.
-
|
Hi, @bentsherman! I was able to start runs, but now for both of them I have an error: [e3/86e1b2] NOTE: Process executor > slurm (1) Completed at: 14-Dec-2023 16:20:59 Even I resume run, i keep getting this error. |
Beta Was this translation helpful? Give feedback.
-
|
I guess it's because your error strategy is set to "ignore" so it does not print the error message |
Beta Was this translation helpful? Give feedback.
-
|
Hello, @bentsherman . Could you, please, help me either change the error strategy to print the error or how I can see the error? |
Beta Was this translation helpful? Give feedback.
-
|
You can set the error strategy in your nextflow config: process.errorStrategy = 'terminate' |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @bentsherman . Dec-18 12:47:51.850 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[jobId: 43260382; id: 1755; name: POREC:pairsToCooler (577); status: COMPLETED; exit>Dec-18 12:47:51.902 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=548; failedCount=4; ignoredCount=0; cachedCount=1182; pendingCou>Dec-18 12:47:51.902 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file But in slurm.out: ERROR ~ Error executing process > 'POREC:makeReport (1)' Caused by: |
Beta Was this translation helpful? Give feedback.
-
|
yes, the goal of changing the error strategy was to make it show a more informative error message. Is there not more detail in the |
Beta Was this translation helpful? Give feedback.
-
|
This is the tail of slurm.out Tip: you can replicate the issue by changing to the process work dir and entering the command -- Check '.nextflow.log' file for details executor > slurm (566) Caused by: Command executed: echo '[ Command output: Command error: Work dir: Tip: you can replicate the issue by changing to the process work dir and entering the command -- Check '.nextflow.log' file for details |
Beta Was this translation helpful? Give feedback.
-
|
There's your error: |
Beta Was this translation helpful? Give feedback.
-
|
I see this error now. But I can't understand why I got this error if I just changed a dataset. And I don't see a place where I can change the work directory and even in .command.sh I see the same info: #!/bin/bash -euo pipefail |
Beta Was this translation helpful? Give feedback.
-
When I do not apply -resume, I can submit multiple nextflow tasks at the same time, and can generate multiple session IDs in the cache of .nextflow. But when I use -resume, even if only one task is submitted, it will appear "Unable to acquire lock on session with ID" error.What is the problem?
Beta Was this translation helpful? Give feedback.
All reactions