Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On torque cluster, piped_bamprep "result not ready, cannot allocate memory" error #3713

Open
fouriejoubert opened this issue Jun 27, 2023 · 1 comment

Comments

@fouriejoubert
Copy link

Hi Folks

I am having trouble getting bcbio to run on my torque cluster with variant2.

I updated to the latest development release, but it was giving the same problem before.

The cluster has 10x nodes with 28 cores and 128 GB RAM each.

The analysis is of a Trusight cancer panel sample, Illumina PE, sequencing 96 genes.

The ref here is still hg19 for comparison to previous results.

When I run the job on a single machine, everything is fine.

When I run the job over multiple nodes using torque with:

bcbio_nextgen.py ../config/omega.yaml -t ipython -n 112 -s torque -q long

it get past bwa mapping, and then seems to die on piped_bamprep as below.

Here is the relevant part of my bcbio_system.yaml file:

resources:

default options, used if other items below are not present

avoids needing to configure/adjust for every program

default:
memory: 4G
cores: 28
jvm_opts: ["-Xms4000m", "-Xmx4000m"]

Can also set program specific options

gatk:
jvm_opts: ["-Xms4000m", "-Xmx4000m"]
snpeff:
jvm_opts: ["-Xms4000m", "-Xmx4000m"]
qualimap:
memory: 4g
express:
memory: 8g
dexseq:
memory: 8g
macs2:
memory: 18g
seqcluster:
memory: 8g

(PS For jvm_opts, is the memory also set per core?)

All config and log files are appended below.

The error is:
...
...
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_bins
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: normalize_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Timing: hla typing
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: call_hla
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Resource requests: gatk, gatk-haplotype, picard; memory: 4.00, 4.00, 4.00; cores: 1, 28, 28
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Configuring 79 jobs to run, using 1 cores each with 4.00g of memory reserved for each job
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Timing: alignment post-processing
[2023-06-27T14:37Z] wonko.bi.up.ac.za: ipython: piped_bamprep
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Did not stop IPython cluster correctly
Traceback (most recent call last):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 689, in _ordered_iter
rlist = self.get(0)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 171, in get
raise error.TimeoutError("Result not ready.")
ipyparallel.error.TimeoutError: Result not ready.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 1151, in cluster_view
yield cluster_view.view
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/prun.py", line 56, in start
yield ipython.runner(view, parallel, dirs, config)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 152, in variant2pipeline
samples = region.parallel_prep_region(samples, run_parallel)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/region.py", line 164, in parallel_prep_region
"piped_bamprep", _add_combine_info, file_key, ["config"])
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/split.py", line 62, in parallel_split_combine
split_output = parallel_fn(parallel_name, split_args)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/ipython.py", line 137, in run
for data in view.map_sync(fn, items, track=False):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 670, in iter
for r in it():
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 695, in _ordered_iter
for r in self._yield_child_results(child):
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 681, in _yield_child_results
error.collect_exceptions(rlist, self._fname)
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 233, in collect_exceptions
raise e
File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 231, in collect_exceptions
raise CompositeError(msg, elist)
ipyparallel.error.CompositeError: one or more exceptions from call to method: piped_bamprep
[53:apply]: ImportError: /apps/bcbio/anaconda/lib/python3.7/site-packages/numpy/fft/_pocketfft_internal.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object: Cannot allocate memory

The system seems to find the node memory more-or-less correctly:

provenance/system-ipython-long.yaml shows:

  • cores: 28
    memory: 3023.428493499756
    name: wonko1.bi.up.ac.za

Any help with this problem would be very highly appreciated!

Best regards!

Fourie

Version info

  • bcbio version 1.2.9
  • OS name and version CentOS Linux release 7.3.1611 (Core)

To Reproduce
bcbio_nextgen.py ../config/omega.yaml -t ipython -n 112 -s torque -q longRunning bcbio version: 1.2.9

Your yaml configuration file:
details:

  • algorithm:
    tools_off: peddy
    align_split_size: 5000000
    aligner: bwa
    jointcaller: gatk-haplotype-joint
    mark_duplicates: true
    nomap_split_targets: 100
    platform: illumina
    quality_format: standard
    realign: false
    recalibrate: false
    remove_lcr: true
    variant_regions: /nlustre/users/dewalde/OmegaAll/bcbio/trusight_cancer_manifest_a.bed
    variantcaller: gatk-haplotype
    analysis: variant2
    description: BRB101
    files:
    • /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R1_001.fastq
    • /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R2_001.fastq
      genome_build: hg19
      metadata:
      batch: omega_all
      phenotype: breast_cancer
      sex: female
      fc_date: '2018-03-15'
      fc_name: omega
      upload:
      dir: ../final```

**Log files (could be found in work/log)**
Please attach (10MB max):  `bcbio-nextgen-commands.log`, and `bcbio-nextgen-debug.log`.

2023-06-27T14:33Z] wonko8.bi.up.ac.za:  /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 14 -c /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R2_001.fastq > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp29advcus/BRB101_S20_L001_R2_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za:  /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 14 -c /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R1_001.fastq > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjgaufwfs/BRB101_S20_L001_R1_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/grabix index /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp_hh35yvg/BRB101_S20_L001_R2_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/grabix index /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpwbnd40zj/BRB101_S20_L001_R1_001.fastq.gz
[2023-06-27T14:33Z] wonko7.bi.up.ac.za: unset JAVA_HOME && /apps/bcbio/galaxy/../anaconda/bin/bwa mem   -c 250 -M -t 28  -R '@RG\tID:BRB101\tPL:illumina\tPU:1_2018-03-15_omega\tSM:BRB101' -v 1 /apps/bcbio/genomes/Hsapiens/hg19/bwa/hg19.fa <(grabix grab /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align_prep/BRB101_S20_L001_R1_001.fastq.gz 1 4422864) <(grabix grab /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align_prep/BRB101_S20_L001_R2_001.fastq.gz 1 4422864)  | /apps/bcbio/galaxy/../anaconda/bin/samtools sort -n -@ 28 -m 2G -O bam -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmponwzg8nx/BRB101-sort-1_4422864-sorttmp-namesort -o /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmponwzg8nx/BRB101-sort-1_4422864.bam -
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/split/BRB101-sort-1_4422864.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: bamcat level=0 tmpfile=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam-bammerge `cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.list` | bamsormadup threads=24 tmpfile=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam-bamsormaduptmp indexfilename=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam.bai > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/trusight_cancer_manifest_a.bed | grep -v ^track | grep -v ^browser | grep -v ^@ | grep -v ^# |  /apps/bcbio/anaconda/bin/python -c 'from bcbio.variation import bedutils; bedutils.remove_bad()' | sort -V -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjufvh8zi -k1,1 -k2,2n > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjufvh8zi/cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpe_4qjlub/cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpz19qh9mc/cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/bedtools merge  -i /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp3hthcdav/cleaned-trusight_cancer_manifest_a-merged.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpuvuq6lvt/cleaned-trusight_cancer_manifest_a-merged.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpuc9855qd/cleaned-trusight_cancer_manifest_a-merged.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed | grep -v ^track | grep -v ^browser | grep -v ^@ | grep -v ^# |  /apps/bcbio/anaconda/bin/python -c 'from bcbio.variation import bedutils; bedutils.remove_bad()' | sort -V -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpap0loisy -k1,1 -k2,2n > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpap0loisy/svregions-cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/svregions-cleaned-trusight_cancer_manifest_a.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpy_8xci9g/svregions-cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjn2tw5g6/svregions-cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: unset JAVA_HOME && export PATH=/apps/bcbio/anaconda/bin:"$PATH" && picard -Xms4000m -Xmx4000m -XX:+UseSerialGC CollectSequencingArtifactMetrics -REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa -INPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam -OUTPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpi_0m0le_/BRB101/BRB101 --VALIDATION_STRINGENCY SILENT
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: unset JAVA_HOME && export PATH=/apps/bcbio/anaconda/bin:"$PATH" && picard -Xms4000m -Xmx4000m -XX:+UseSerialGC -XX:+UseSerialGC ConvertSequencingArtifactToOxoG --INPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/metrics/artifact/BRB101/BRB101 --OUTPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpztrmznc0/BRB101/BRB101 --REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa --VALIDATION_STRINGENCY SILENT
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: export MOSDEPTH_Q0=NO_COVERAGE && export MOSDEPTH_Q1=LOW_COVERAGE && export MOSDEPTH_Q2=CALLABLE && /apps/bcbio/anaconda/envs/htslib1.10/bin/mosdepth -t 28 -F 1804 -Q 1 --no-per-base --by /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed --quantize 0:1:4: /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpy_kdwoq3/BRB101-variant_regions /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: /apps/bcbio/anaconda/envs/htslib1.10/bin/mosdepth -t 28 -F 1804  --no-per-base --by /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/svregions-cleaned-trusight_cancer_manifest_a.bed  /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpgmgzr58f/BRB101-sv_regions /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: hts_nim_tools count-reads -t 28 -F 1804 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/coverage/BRB101/counts/fullgenome.bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmphiaivr7g/fullgenome-1804-counts.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: hts_nim_tools count-reads -t 28 -F 1804 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp4762gxcu/cleaned-trusight_cancer_manifest_a-merged-1804-counts.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools stats -@ 28 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpybx_tjj9/BRB101.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools idxstats /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmph42xousp/BRB101-idxstats.txt





[2023-06-27T14:33Z] wonko8.bi.up.ac.za:  /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 14 -c /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R2_001.fastq > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp29advcus/BRB101_S20_L001_R2_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za:  /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 14 -c /nlustre/users/dewalde/OmegaAll/bcbio/fastq/BRB101_S20_L001_R1_001.fastq > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjgaufwfs/BRB101_S20_L001_R1_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/grabix index /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp_hh35yvg/BRB101_S20_L001_R2_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/grabix index /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpwbnd40zj/BRB101_S20_L001_R1_001.fastq.gz
[2023-06-27T14:33Z] wonko7.bi.up.ac.za: unset JAVA_HOME && /apps/bcbio/galaxy/../anaconda/bin/bwa mem   -c 250 -M -t 28  -R '@RG\tID:BRB101\tPL:illumina\tPU:1_2018-03-15_omega\tSM:BRB101' -v 1 /apps/bcbio/genomes/Hsapiens/hg19/bwa/hg19.fa <(grabix grab /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align_prep/BRB101_S20_L001_R1_001.fastq.gz 1 4422864) <(grabix grab /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align_prep/BRB101_S20_L001_R2_001.fastq.gz 1 4422864)  | /apps/bcbio/galaxy/../anaconda/bin/samtools sort -n -@ 28 -m 2G -O bam -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmponwzg8nx/BRB101-sort-1_4422864-sorttmp-namesort -o /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmponwzg8nx/BRB101-sort-1_4422864.bam -
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/split/BRB101-sort-1_4422864.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: bamcat level=0 tmpfile=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam-bammerge `cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.list` | bamsormadup threads=24 tmpfile=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam-bamsormaduptmp indexfilename=/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam.bai > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp66p58k4d/bcbiotx/tmp1g5qnv3q/BRB101-sort.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools quickcheck -v /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/trusight_cancer_manifest_a.bed | grep -v ^track | grep -v ^browser | grep -v ^@ | grep -v ^# |  /apps/bcbio/anaconda/bin/python -c 'from bcbio.variation import bedutils; bedutils.remove_bad()' | sort -V -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjufvh8zi -k1,1 -k2,2n > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjufvh8zi/cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpe_4qjlub/cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpz19qh9mc/cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/bedtools merge  -i /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp3hthcdav/cleaned-trusight_cancer_manifest_a-merged.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpuvuq6lvt/cleaned-trusight_cancer_manifest_a-merged.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpuc9855qd/cleaned-trusight_cancer_manifest_a-merged.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a.bed | grep -v ^track | grep -v ^browser | grep -v ^@ | grep -v ^# |  /apps/bcbio/anaconda/bin/python -c 'from bcbio.variation import bedutils; bedutils.remove_bad()' | sort -V -T /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpap0loisy -k1,1 -k2,2n > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpap0loisy/svregions-cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: cat /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/svregions-cleaned-trusight_cancer_manifest_a.bed  | /apps/bcbio/galaxy/../anaconda/bin/bgzip --threads 28 -c > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpy_8xci9g/svregions-cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/tabix -f -p bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpjn2tw5g6/svregions-cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: unset JAVA_HOME && export PATH=/apps/bcbio/anaconda/bin:"$PATH" && picard -Xms4000m -Xmx4000m -XX:+UseSerialGC CollectSequencingArtifactMetrics -REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa -INPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam -OUTPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpi_0m0le_/BRB101/BRB101 --VALIDATION_STRINGENCY SILENT
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: unset JAVA_HOME && export PATH=/apps/bcbio/anaconda/bin:"$PATH" && picard -Xms4000m -Xmx4000m -XX:+UseSerialGC -XX:+UseSerialGC ConvertSequencingArtifactToOxoG --INPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/metrics/artifact/BRB101/BRB101 --OUTPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpztrmznc0/BRB101/BRB101 --REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa --VALIDATION_STRINGENCY SILENT
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: export MOSDEPTH_Q0=NO_COVERAGE && export MOSDEPTH_Q1=LOW_COVERAGE && export MOSDEPTH_Q2=CALLABLE && /apps/bcbio/anaconda/envs/htslib1.10/bin/mosdepth -t 28 -F 1804 -Q 1 --no-per-base --by /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed --quantize 0:1:4: /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpy_kdwoq3/BRB101-variant_regions /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: /apps/bcbio/anaconda/envs/htslib1.10/bin/mosdepth -t 28 -F 1804  --no-per-base --by /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/svregions-cleaned-trusight_cancer_manifest_a.bed  /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpgmgzr58f/BRB101-sv_regions /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: hts_nim_tools count-reads -t 28 -F 1804 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/coverage/BRB101/counts/fullgenome.bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmphiaivr7g/fullgenome-1804-counts.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: hts_nim_tools count-reads -t 28 -F 1804 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bedprep/cleaned-trusight_cancer_manifest_a-merged.bed /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmp4762gxcu/cleaned-trusight_cancer_manifest_a-merged-1804-counts.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools stats -@ 28 /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpybx_tjj9/BRB101.txt
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: /apps/bcbio/galaxy/../anaconda/bin/samtools idxstats /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam > /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmph42xousp/BRB101-idxstats.txt
dewalde@wonko:/nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/log/ #cat bcbio-nextgen-debug.log
[2023-06-27T14:32Z] wonko.bi.up.ac.za: System YAML configuration: /apps/bcbio/galaxy/bcbio_system.yaml.
[2023-06-27T14:32Z] wonko.bi.up.ac.za: Locale set to en_US.utf8.
[2023-06-27T14:32Z] wonko.bi.up.ac.za: Resource requests: bwa, sambamba, samtools; memory: 4.00, 4.00, 4.00; cores: 28, 28, 28
[2023-06-27T14:32Z] wonko.bi.up.ac.za: Configuring 4 jobs to run, using 28 cores each with 112.1g of memory reserved for each job
[2023-06-27T14:33Z] wonko.bi.up.ac.za: Timing: organize samples
[2023-06-27T14:33Z] wonko.bi.up.ac.za: ipython: organize_samples
[2023-06-27T14:33Z] wonko5.bi.up.ac.za: Using input YAML configuration: /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/config/omega.yaml
[2023-06-27T14:33Z] wonko5.bi.up.ac.za: Checking sample YAML configuration: /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/config/omega.yaml
[2023-06-27T14:33Z] wonko5.bi.up.ac.za: Retreiving program versions from /apps/bcbio/manifest/python-packages.yaml.
[2023-06-27T14:33Z] wonko5.bi.up.ac.za: Retreiving program versions from /apps/bcbio/manifest/r-packages.yaml.
[2023-06-27T14:33Z] wonko5.bi.up.ac.za: Testing minimum versions of installed programs
[2023-06-27T14:33Z] wonko.bi.up.ac.za: Timing: alignment preparation
[2023-06-27T14:33Z] wonko.bi.up.ac.za: ipython: prep_align_inputs
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: bgzip input file
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: bgzip input file
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: Skipping trimming of BRB101.
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: Resource requests: ; memory: 1.00; cores: 1
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: Configuring 2 jobs to run, using 1 cores each with 1.00g of memory reserved for each job
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: Index input with grabix: BRB101_S20_L001_R2_001.fastq.gz
[2023-06-27T14:33Z] wonko8.bi.up.ac.za: Index input with grabix: BRB101_S20_L001_R1_001.fastq.gz
[2023-06-27T14:33Z] wonko.bi.up.ac.za: ipython: disambiguate_split
[2023-06-27T14:33Z] wonko.bi.up.ac.za: Timing: alignment
[2023-06-27T14:33Z] wonko.bi.up.ac.za: ipython: process_alignment
[2023-06-27T14:33Z] wonko7.bi.up.ac.za: bwa mem alignment from fastq: BRB101
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] analyzing insert size distribution for orientation FF...
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] (25, 50, 75) percentile: (144, 263, 421)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 975)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] mean and std.dev: (275.19, 166.10)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for proper pairs: (1, 1252)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] analyzing insert size distribution for orientation FR...
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] (25, 50, 75) percentile: (159, 240, 343)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 711)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] mean and std.dev: (260.61, 134.36)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for proper pairs: (1, 895)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] analyzing insert size distribution for orientation RF...
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] (25, 50, 75) percentile: (143, 160, 1468)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 4118)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] mean and std.dev: (765.45, 1039.50)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for proper pairs: (1, 5443)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] analyzing insert size distribution for orientation RR...
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] (25, 50, 75) percentile: (157, 321, 812)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2122)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] mean and std.dev: (363.52, 355.94)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] low and high boundaries for proper pairs: (1, 2777)
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] skip orientation FF
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] skip orientation RF
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [M::mem_pestat] skip orientation RR
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [main] Version: 0.7.17-r1188
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [main] CMD: /apps/bcbio/galaxy/../anaconda/bin/bwa mem -c 250 -M -t 28 -R @RG\tID:BRB101\tPL:illumina\tPU:1_2018-03-15_omega\tSM:BRB101 -v 1 /apps/bcbio/genomes/Hsapiens/hg19/bwa/hg19.fa /dev/fd/63 /dev/fd/62
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [main] Real time: 17.687 sec; CPU: 304.184 sec
[2023-06-27T14:34Z] wonko7.bi.up.ac.za: [bam_sort_core] merging from 0 files and 28 in-memory blocks...
[2023-06-27T14:34Z] wonko.bi.up.ac.za: ipython: delayed_bam_merge
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: Ensure integrity of input merge BAM files
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: Merge bam files to BRB101-sort.bam
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 1048576
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2000085	07:18814200	MemUsage(size=5308.34,rss=1800.58,peak=5316.7)	AutoArrayMemUsage(memusage=3768.1,peakmemusage=3934.35,maxmem=1.75922e+13)
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2097152
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2212631
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] finished MemUsage(size=62.5625,rss=6.12109,peak=62.7539)
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2212631	07:81111800	MemUsage(size=5321.52,rss=1847.81,peak=5321.52)	AutoArrayMemUsage(memusage=3770.29,peakmemusage=3934.35,maxmem=1.75922e+13)	final
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] flushing read ends lists...done.
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] merging read ends lists/computing duplicates...done, time 01:12381500
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] num dups 358443
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: # bamsormadup
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: ##METRICS
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: LIBRARY	UNPAIRED_READS_EXAMINED	READ_PAIRS_EXAMINED	UNMAPPED_READS	UNPAIRED_READ_DUPLICATES	READ_PAIR_DUPLICATES	READ_PAIR_OPTICAL_DUPLICATES	PERCENT_DUPLICATION	ESTIMATED_LIBRARY_SIZE
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: Unknown Library	8108	1097308	8708	6587	175928	96	0.162727	3046922
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: ## HISTOGRAM
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: BIN	VALUE
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 1	1.00007
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 2	1.6977
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 3	2.18436
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 4	2.52384
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 5	2.76066
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 6	2.92585
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 7	3.04109
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 8	3.12148
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 9	3.17756
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 10	3.21668
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 11	3.24397
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 12	3.263
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 13	3.27628
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 14	3.28554
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 15	3.29201
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 16	3.29651
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 17	3.29966
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 18	3.30185
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 19	3.30338
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 20	3.30445
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 21	3.30519
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 22	3.30571
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 23	3.30608
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 24	3.30633
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 25	3.3065
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 26	3.30663
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 27	3.30671
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 28	3.30677
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 29	3.30682
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 30	3.30684
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 31	3.30686
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 32	3.30688
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 33	3.30689
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 34	3.3069
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 35	3.3069
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 36	3.3069
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 37	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 38	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 39	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 40	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 41	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 42	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 43	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 44	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 45	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 46	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 47	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 48	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 49	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 50	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 51	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 52	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 53	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 54	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 55	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 56	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 57	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 58	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 59	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 60	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 61	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 62	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 63	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 64	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 65	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 66	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 67	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 68	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 69	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 70	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 71	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 72	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 73	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 74	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 75	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 76	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 77	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 78	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 79	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 80	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 81	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 82	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 83	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 84	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 85	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 86	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 87	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 88	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 89	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 90	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 91	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 92	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 93	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 94	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 95	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 96	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 97	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 98	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 99	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: 100	3.30691
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] blocks generated in time 15:19779499
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] number of blocks to be merged is 3 using 2730 blocks per input with block size 1048576
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 1777322 68934400
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2212631 86993500
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] 2212631
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [D]	md5	1218863054268aa5a4438f0aaf13addb
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] checksum ok
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] blocks merged in time 04:16847399
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: [V] run time 19:36761699 (19.3676 s)	MemUsage(size=11606.2,rss=275.164,peak=12882.3)
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: Check for valid merged BAM
[2023-06-27T14:34Z] wonko5.bi.up.ac.za: Check for valid merged BAM after transfer
[2023-06-27T14:34Z] wonko.bi.up.ac.za: ipython: merge_split_alignments
[2023-06-27T14:34Z] wonko.bi.up.ac.za: Timing: callable regions
[2023-06-27T14:34Z] wonko.bi.up.ac.za: ipython: prep_samples
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: Prepare cleaned BED file : BRB101
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: bgzip cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: tabix index cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: Prepare merged BED file : BRB101
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: bgzip cleaned-trusight_cancer_manifest_a-merged.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: tabix index cleaned-trusight_cancer_manifest_a-merged.bed.gz
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: Prepare cleaned BED file : BRB101
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: bgzip svregions-cleaned-trusight_cancer_manifest_a.bed
[2023-06-27T14:34Z] wonko8.bi.up.ac.za: tabix index svregions-cleaned-trusight_cancer_manifest_a.bed.gz
[2023-06-27T14:34Z] wonko.bi.up.ac.za: ipython: postprocess_alignment
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: Picard: CollectSequencingArtifactMetrics
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: 16:34:39.950 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/apps/bcbio/anaconda/envs/java/share/picard-2.27.5-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:34:39 SAST 2023] CollectSequencingArtifactMetrics --INPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/align/BRB101/BRB101-sort.bam --OUTPUT /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpi_0m0le_/BRB101/BRB101 --VALIDATION_STRINGENCY SILENT --REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa --MINIMUM_QUALITY_SCORE 20 --MINIMUM_MAPPING_QUALITY 30 --MINIMUM_INSERT_SIZE 60 --MAXIMUM_INSERT_SIZE 600 --INCLUDE_UNPAIRED false --INCLUDE_DUPLICATES false --INCLUDE_NON_PF_READS false --TANDEM_READS false --USE_OQ true --CONTEXT_SIZE 1 --ASSUME_SORTED true --STOP_AFTER 0 --VERBOSITY INFO --QUIET false --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[2023-06-27T14:34Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:34:39 SAST 2023] Executing as [email protected] on Linux 3.10.0-514.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_332-b09; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: Version:2.27.5
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: INFO	2023-06-27 16:35:00	SinglePassSamProgram	Processed     1,000,000 records.  Elapsed time: 00:00:20s.  Time for last 1,000,000:   20s.  Last read position: chr10:104,263,763
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: INFO	2023-06-27 16:35:11	SinglePassSamProgram	Processed     2,000,000 records.  Elapsed time: 00:00:31s.  Time for last 1,000,000:   11s.  Last read position: chr17:66,522,074
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:35:16 SAST 2023] picard.analysis.artifacts.CollectSequencingArtifactMetrics done. Elapsed time: 0.61 minutes.
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: Runtime.totalMemory()=4054515712
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: Picard: ConvertSequencingArtifactToOxoG
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: 16:35:17.653 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/apps/bcbio/anaconda/envs/java/share/picard-2.27.5-0/picard.jar!/com/intel/gkl/native/libgkl_compression.so
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:35:17 SAST 2023] ConvertSequencingArtifactToOxoG --INPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/metrics/artifact/BRB101/BRB101 --OUTPUT_BASE /nlustre/users/dewalde/OmegaAll/bcbio/fj_omega_jun_2023/work/bcbiotx/tmpztrmznc0/BRB101/BRB101 --VALIDATION_STRINGENCY SILENT --REFERENCE_SEQUENCE /apps/bcbio/genomes/Hsapiens/hg19/seq/hg19.fa --VERBOSITY INFO --QUIET false --COMPRESSION_LEVEL 5 --MAX_RECORDS_IN_RAM 500000 --CREATE_INDEX false --CREATE_MD5_FILE false --GA4GH_CLIENT_SECRETS client_secrets.json --help false --version false --showHidden false --USE_JDK_DEFLATER false --USE_JDK_INFLATER false
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:35:17 SAST 2023] Executing as [email protected] on Linux 3.10.0-514.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_332-b09; Deflater: Intel; Inflater: Intel; Provider GCS is not available; Picard version: Version:2.27.5
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: [Tue Jun 27 16:35:17 SAST 2023] picard.analysis.artifacts.ConvertSequencingArtifactToOxoG done. Elapsed time: 0.00 minutes.
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: Runtime.totalMemory()=4054515712
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: Calculating coverage: BRB101 variant_regions
[2023-06-27T14:35Z] wonko6.bi.up.ac.za: Calculating coverage: BRB101 sv_regions
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: Count mapped reads: BRB101
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: Count mapped reads: BRB101
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: BRB101: Assigned coverage as 'regional' with 0.0% genome coverage and 47.8% offtarget coverage
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: samtools stats : BRB101
[2023-06-27T14:36Z] wonko6.bi.up.ac.za: samtools index stats : BRB101
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: combine_sample_regions
[2023-06-27T14:36Z] wonko7.bi.up.ac.za: Identified 79 parallel analysis blocks
Block sizes:
  min: 598851
  5%: 6608243.800000001
  25%: 28046955.0
  median: 35158926.0
  75%: 42428054.0
  95%: 64058731.69999993
  99%: 84564426.15999998
  max: 95639668
Between block sizes:
  min: 369
  5%: 433.8
  25%: 3626.0
  median: 161487.0
  75%: 3771270.0
  95%: 28298445.799999993
  99%: 38151235.58
  max: 38580440

[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_bins
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: calculate_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: normalize_sv_coverage
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Timing: hla typing
[2023-06-27T14:36Z] wonko.bi.up.ac.za: ipython: call_hla
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Resource requests: gatk, gatk-haplotype, picard; memory: 4.00, 4.00, 4.00; cores: 1, 28, 28
[2023-06-27T14:36Z] wonko.bi.up.ac.za: Configuring 79 jobs to run, using 1 cores each with 4.00g of memory reserved for each job
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Timing: alignment post-processing
[2023-06-27T14:37Z] wonko.bi.up.ac.za: ipython: piped_bamprep
[2023-06-27T14:37Z] wonko.bi.up.ac.za: Did not stop IPython cluster correctly
Traceback (most recent call last):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 689, in _ordered_iter
    rlist = self.get(0)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 171, in get
    raise error.TimeoutError("Result not ready.")
ipyparallel.error.TimeoutError: Result not ready.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 1151, in cluster_view
    yield cluster_view.view
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/prun.py", line 56, in start
    yield ipython.runner(view, parallel, dirs, config)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/main.py", line 152, in variant2pipeline
    samples = region.parallel_prep_region(samples, run_parallel)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/pipeline/region.py", line 164, in parallel_prep_region
    "piped_bamprep", _add_combine_info, file_key, ["config"])
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/split.py", line 62, in parallel_split_combine
    split_output = parallel_fn(parallel_name, split_args)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/ipython.py", line 137, in run
    for data in view.map_sync(fn, items, track=False):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 670, in __iter__
    for r in it():
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 695, in _ordered_iter
    for r in self._yield_child_results(child):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/client/asyncresult.py", line 681, in _yield_child_results
    error.collect_exceptions(rlist, self._fname)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 233, in collect_exceptions
    raise e
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/ipyparallel/error.py", line 231, in collect_exceptions
    raise CompositeError(msg, elist)
ipyparallel.error.CompositeError: one or more exceptions from call to method: piped_bamprep
[53:apply]: ImportError: /apps/bcbio/anaconda/lib/python3.7/site-packages/numpy/fft/_pocketfft_internal.cpython-37m-x86_64-linux-gnu.so: failed to map segment from shared object: Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/prun.py", line 56, in start
    yield ipython.runner(view, parallel, dirs, config)
  File "/apps/bcbio/anaconda/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 1153, in cluster_view
    cluster_view.stop()
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 1121, in stop
    _stop(self.profile, self.cluster_id)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 980, in _stop
    subprocess.check_call(args)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 358, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 341, in call
    return p.wait(timeout=timeout)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1653, in _wait
    (pid, sts) = self._try_wait(0)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1611, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/bcbio/distributed/ipython.py", line 92, in stop
    ipython_cluster.stop_from_view(view)
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 973, in stop_from_view
    _stop(view.clusterhelper["profile"], view.clusterhelper["cluster_id"])
  File "/apps/bcbio/anaconda/lib/python3.7/site-packages/cluster_helper/cluster.py", line 980, in _stop
    subprocess.check_call(args)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 358, in check_call
    retcode = call(*popenargs, **kwargs)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 341, in call
    return p.wait(timeout=timeout)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1019, in wait
    return self._wait(timeout=timeout)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1653, in _wait
    (pid, sts) = self._try_wait(0)
  File "/apps/bcbio/anaconda/lib/python3.7/subprocess.py", line 1611, in _try_wait
    (pid, sts) = os.waitpid(self.pid, wait_flags)
KeyboardInterrupt
@naumenko-sa
Copy link
Contributor

We ran it on slurm a lot, but not on torque, so there might be unsolved issues.

You could try to increase the timeout and controller job memory (-r conmem=2)
https://github.com/naumenko-sa/bioscripts/blob/master/clusters/bcbio.ipython.o2.sh

However, in your case it is easier to run one node bcbio projects/sample without ipython (small gene panel).
With such a small gene panel you could even run many samples on one node.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants