python exited with 1. Theano error.

[lovelycatZ]

Bug Report

Affected tool(s) or class(es)

GATK-gCNV-GermlineCaller

Affected version(s)

4.3.0.0

Description

`
00:16:50.019 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/share/gatk4-4.3.0.0-0/gatk-package-4.3.0.0-local.jar!/com/intel/gkl/native/libgkl_compression.so
00:16:50.035 DEBUG NativeLibraryLoader - Extracting libgkl_compression.so to /tmp/libgkl_compression18004866504412299404.so
00:16:50.108 INFO GermlineCNVCaller - ------------------------------------------------------------
00:16:50.108 INFO GermlineCNVCaller - The Genome Analysis Toolkit (GATK) v4.3.0.0
00:16:50.108 INFO GermlineCNVCaller - For support and documentation go to https://software.broadinstitute.org/gatk/
00:16:50.108 INFO GermlineCNVCaller - Executing as xiangxud@cu09 on Linux v3.10.0-1160.el7.x86_64 amd64
00:16:50.108 INFO GermlineCNVCaller - Java runtime: OpenJDK 64-Bit Server VM v11.0.8-internal+0-adhoc..src
00:16:50.108 INFO GermlineCNVCaller - Start Date/Time: December 12, 2024 at 12:16:49 AM CST
00:16:50.108 INFO GermlineCNVCaller - ------------------------------------------------------------
00:16:50.108 INFO GermlineCNVCaller - ------------------------------------------------------------
00:16:50.109 INFO GermlineCNVCaller - HTSJDK Version: 3.0.1
00:16:50.109 INFO GermlineCNVCaller - Picard Version: 2.27.5
00:16:50.109 INFO GermlineCNVCaller - Built for Spark Version: 2.4.5
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.BUFFER_SIZE : 131072
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.COMPRESSION_LEVEL : 2
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.CREATE_INDEX : false
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.CREATE_MD5 : false
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.CUSTOM_READER_FACTORY :
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.DISABLE_SNAPPY_COMPRESSOR : false
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.EBI_REFERENCE_SERVICE_URL_MASK : https://www.ebi.ac.uk/ena/cram/md5/%s
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.NON_ZERO_BUFFER_SIZE : 131072
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.REFERENCE_FASTA : null
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.SAM_FLAG_FIELD_FORMAT : DECIMAL
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false
00:16:50.110 INFO GermlineCNVCaller - HTSJDK Defaults.USE_CRAM_REF_DOWNLOAD : false
00:16:50.110 DEBUG ConfigFactory - Configuration file values:
00:16:50.112 DEBUG ConfigFactory - gcsMaxRetries = 20
00:16:50.113 DEBUG ConfigFactory - gcsProjectForRequesterPays =
00:16:50.113 DEBUG ConfigFactory - gatk_stacktrace_on_user_exception = false
00:16:50.113 DEBUG ConfigFactory - samjdk.use_async_io_read_samtools = false
00:16:50.113 DEBUG ConfigFactory - samjdk.use_async_io_write_samtools = true
00:16:50.113 DEBUG ConfigFactory - samjdk.use_async_io_write_tribble = false
00:16:50.113 DEBUG ConfigFactory - samjdk.compression_level = 2
00:16:50.113 DEBUG ConfigFactory - spark.kryoserializer.buffer.max = 512m
00:16:50.113 DEBUG ConfigFactory - spark.driver.maxResultSize = 0
00:16:50.113 DEBUG ConfigFactory - spark.driver.userClassPathFirst = true
00:16:50.113 DEBUG ConfigFactory - spark.io.compression.codec = lzf
00:16:50.113 DEBUG ConfigFactory - spark.executor.memoryOverhead = 600
00:16:50.113 DEBUG ConfigFactory - spark.driver.extraJavaOptions =
00:16:50.113 DEBUG ConfigFactory - spark.executor.extraJavaOptions =
00:16:50.113 DEBUG ConfigFactory - codec_packages = [htsjdk.variant, htsjdk.tribble, org.broadinstitute.hellbender.utils.codecs]
00:16:50.113 DEBUG ConfigFactory - read_filter_packages = [org.broadinstitute.hellbender.engine.filters]
00:16:50.113 DEBUG ConfigFactory - annotation_packages = [org.broadinstitute.hellbender.tools.walkers.annotator]
00:16:50.115 DEBUG ConfigFactory - cloudPrefetchBuffer = 40
00:16:50.115 DEBUG ConfigFactory - cloudIndexPrefetchBuffer = -1
00:16:50.115 DEBUG ConfigFactory - createOutputBamIndex = true
00:16:50.115 INFO GermlineCNVCaller - Deflater: IntelDeflater
00:16:50.116 INFO GermlineCNVCaller - Inflater: IntelInflater
00:16:50.116 INFO GermlineCNVCaller - GCS max retries/reopens: 20
00:16:50.116 INFO GermlineCNVCaller - Requester pays: disabled
00:16:50.116 INFO GermlineCNVCaller - Initializing engine
00:16:50.118 DEBUG ScriptExecutor - Executing:
00:16:50.118 DEBUG ScriptExecutor - python
00:16:50.118 DEBUG ScriptExecutor - -c
00:16:50.118 DEBUG ScriptExecutor - import gcnvkernel

INFO (theano.gof.compilelock): Waiting for existing lock by process '49240' (I am process '17815')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
00:17:27.676 DEBUG ScriptExecutor - Result: 0
00:17:27.676 INFO GermlineCNVCaller - Done initializing engine
00:17:27.684 INFO GermlineCNVCaller - Intervals specified...
00:17:27.981 DEBUG GenomeLocParser - Prepared reference sequence contig dictionary
00:17:27.982 DEBUG GenomeLocParser - chr1 (248956422 bp)

....

00:17:28.573 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA06986.tsv (1 / 16)
00:17:28.729 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA06989.tsv (2 / 16)
00:17:28.880 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA07051.tsv (3 / 16)
00:17:29.035 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA07347.tsv (4 / 16)
00:17:29.190 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA11843.tsv (5 / 16)
00:17:29.344 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA12878.tsv (6 / 16)
00:17:29.499 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18959.tsv (7 / 16)
00:17:29.654 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18960.tsv (8 / 16)
00:17:29.808 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18537.tsv (9 / 16)
00:17:29.954 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18976.tsv (10 / 16)
00:17:30.093 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA19971.tsv (11 / 16)
00:17:30.231 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA19920.tsv (12 / 16)
00:17:30.370 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG00240.tsv (13 / 16)
00:17:30.509 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02154.tsv (14 / 16)
00:17:30.646 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02277.tsv (15 / 16)
00:17:30.786 INFO GermlineCNVCaller - Aggregating read-count file /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02555.tsv (16 / 16)
00:17:30.930 DEBUG ScriptExecutor - Executing:
00:17:30.930 DEBUG ScriptExecutor - python
00:17:30.930 DEBUG ScriptExecutor - /tmp/cohort_denoising_calling.11918075781700808988.py
00:17:30.930 DEBUG ScriptExecutor - --ploidy_calls_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/ploidy/ploidy-calls
00:17:30.930 DEBUG ScriptExecutor - --output_calls_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-calls
00:17:30.930 DEBUG ScriptExecutor - --output_tracking_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-tracking
00:17:30.930 DEBUG ScriptExecutor - --random_seed=1984
00:17:30.930 DEBUG ScriptExecutor - --modeling_interval_list=/tmp/intervals4455187968449947169.tsv
00:17:30.930 DEBUG ScriptExecutor - --output_model_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-model
00:17:30.930 DEBUG ScriptExecutor - --enable_explicit_gc_bias_modeling=True
00:17:30.930 DEBUG ScriptExecutor - --read_count_tsv_files
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA06986.rc9044433718998481504.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA06989.rc12452120697434499065.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA07051.rc1885589103072221940.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA07347.rc1279844474304484703.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA11843.rc12301560452012374721.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA12878.rc11872029483327323378.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA18959.rc9459339676102199183.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA18960.rc6256338621902382719.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA18537.rc16791841972370257769.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA18976.rc6581421593900877568.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA19971.rc2228030607602018757.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/NA19920.rc12576022814788442980.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/HG00240.rc9196753230300445104.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/HG02154.rc14097840262549730733.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/HG02277.rc17532145266885259301.tsv
00:17:30.930 DEBUG ScriptExecutor - /tmp/HG02555.rc311237558638496068.tsv
00:17:30.930 DEBUG ScriptExecutor - --psi_s_scale=1.000000e-04
00:17:30.930 DEBUG ScriptExecutor - --mapping_error_rate=1.000000e-02
00:17:30.930 DEBUG ScriptExecutor - --depth_correction_tau=1.000000e+04
00:17:30.930 DEBUG ScriptExecutor - --q_c_expectation_mode=hybrid
00:17:30.930 DEBUG ScriptExecutor - --num_samples_copy_ratio_approx=200
00:17:30.930 DEBUG ScriptExecutor - --max_bias_factors=5
00:17:30.930 DEBUG ScriptExecutor - --psi_t_scale=1.000000e-03
00:17:30.930 DEBUG ScriptExecutor - --log_mean_bias_std=1.000000e-01
00:17:30.930 DEBUG ScriptExecutor - --init_ard_rel_unexplained_variance=1.000000e-01
00:17:30.930 DEBUG ScriptExecutor - --num_gc_bins=20
00:17:30.930 DEBUG ScriptExecutor - --gc_curve_sd=1.000000e+00
00:17:30.930 DEBUG ScriptExecutor - --active_class_padding_hybrid_mode=50000
00:17:30.930 DEBUG ScriptExecutor - --enable_bias_factors=True
00:17:30.930 DEBUG ScriptExecutor - --disable_bias_factors_in_active_class=False
00:17:30.930 DEBUG ScriptExecutor - --p_alt=1.000000e-06
00:17:30.931 DEBUG ScriptExecutor - --cnv_coherence_length=1.000000e+04
00:17:30.931 DEBUG ScriptExecutor - --max_copy_number=5
00:17:30.931 DEBUG ScriptExecutor - --p_active=0.010000
00:17:30.931 DEBUG ScriptExecutor - --class_coherence_length=10000.000000
00:17:30.931 DEBUG ScriptExecutor - --learning_rate=1.000000e-02
00:17:30.931 DEBUG ScriptExecutor - --adamax_beta1=9.000000e-01
00:17:30.931 DEBUG ScriptExecutor - --adamax_beta2=9.900000e-01
00:17:30.931 DEBUG ScriptExecutor - --log_emission_samples_per_round=50
00:17:30.931 DEBUG ScriptExecutor - --log_emission_sampling_rounds=10
00:17:30.931 DEBUG ScriptExecutor - --log_emission_sampling_median_rel_error=5.000000e-03
00:17:30.931 DEBUG ScriptExecutor - --max_advi_iter_first_epoch=5000
00:17:30.931 DEBUG ScriptExecutor - --max_advi_iter_subsequent_epochs=200
00:17:30.931 DEBUG ScriptExecutor - --min_training_epochs=10
00:17:30.931 DEBUG ScriptExecutor - --max_training_epochs=50
00:17:30.931 DEBUG ScriptExecutor - --initial_temperature=1.500000e+00
00:17:30.931 DEBUG ScriptExecutor - --num_thermal_advi_iters=2500
00:17:30.931 DEBUG ScriptExecutor - --convergence_snr_averaging_window=500
00:17:30.931 DEBUG ScriptExecutor - --convergence_snr_trigger_threshold=1.000000e-01
00:17:30.931 DEBUG ScriptExecutor - --convergence_snr_countdown_window=10
00:17:30.931 DEBUG ScriptExecutor - --max_calling_iters=10
00:17:30.931 DEBUG ScriptExecutor - --caller_update_convergence_threshold=1.000000e-03
00:17:30.931 DEBUG ScriptExecutor - --caller_internal_admixing_rate=7.500000e-01
00:17:30.931 DEBUG ScriptExecutor - --caller_external_admixing_rate=1.000000e+00
00:17:30.931 DEBUG ScriptExecutor - --disable_caller=false
00:17:30.931 DEBUG ScriptExecutor - --disable_sampler=false
00:17:30.931 DEBUG ScriptExecutor - --disable_annealing=false
INFO (theano.gof.compilelock): Waiting for existing lock by process '52732' (I am process '19052')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '51120' (I am process '19052')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '42396' (I am process '19052')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '42396' (I am process '19052')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
INFO (theano.gof.compilelock): Waiting for existing lock by process '31334' (I am process '19052')
INFO (theano.gof.compilelock): To manually release the lock, delete /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/lock_dir
00:18:40.679 INFO cohort_denoising_calling - THEANO_FLAGS environment variable has been set to: device=cpu,floatX=float64,optimizer=fast_run,compute_test_value=ignore,openmp=true,blas.ldflags=-lmkl_rt,openmp_elemwise_minsize=10
00:18:40.696 INFO gcnvkernel.io.io_intervals_and_counts - The given interval list provides the following interval annotations: {'GC_CONTENT'}
00:18:40.701 INFO cohort_denoising_calling - Loading 16 read counts file(s)...
00:18:40.839 INFO gcnvkernel.io.io_metadata - Loading germline contig ploidy and global read depth metadata...
00:18:40.853 WARNING gcnvkernel.structs.metadata - Sample HG02154 has an anomalous karyotype ({'Y': 0, 'X': 3}). The presence of unmasked PAR regions and regions of low mappability in the coverage metadata can result in unreliable ploidy designations. It is recommended that the user verifies this designation by orthogonal methods.
00:18:40.865 WARNING gcnvkernel.structs.metadata - Sample HG02277 has an anomalous karyotype ({'Y': 1, 'X': 2}). The presence of unmasked PAR regions and regions of low mappability in the coverage metadata can result in unreliable ploidy designations. It is recommended that the user verifies this designation by orthogonal methods.
00:18:40.875 WARNING gcnvkernel.structs.metadata - Sample NA07347 has an anomalous karyotype ({'Y': 1, 'X': 2}). The presence of unmasked PAR regions and regions of low mappability in the coverage metadata can result in unreliable ploidy designations. It is recommended that the user verifies this designation by orthogonal methods.
00:18:40.901 WARNING gcnvkernel.structs.metadata - Sample NA19971 has an anomalous karyotype ({'Y': 1, 'X': 2}). The presence of unmasked PAR regions and regions of low mappability in the coverage metadata can result in unreliable ploidy designations. It is recommended that the user verifies this designation by orthogonal methods.
00:18:41.053 INFO gcnvkernel.tasks.task_cohort_denoising_calling - Instantiating the denoising model (warm-up)...
00:18:52.038 INFO gcnvkernel.models.fancy_model - Global model variables: {'ard_u_log__', 'log_mean_bias_t', 'W_tu', 'psi_t_log__'}
00:18:52.038 INFO gcnvkernel.models.fancy_model - Sample-specific model variables: {'z_su', 'read_depth_s_log__', 'psi_s_log__', 'z_sg'}
00:18:52.038 WARNING gcnvkernel.tasks.inference_task_base - No log emission sampler given; skipping the sampling step
00:18:52.038 WARNING gcnvkernel.tasks.inference_task_base - No caller given; skipping the calling step
00:18:52.038 INFO gcnvkernel.tasks.inference_task_base - Instantiating the convergence tracker...
00:18:52.038 INFO gcnvkernel.tasks.inference_task_base - Setting up DA-ADVI...
00:19:04.688 INFO gcnvkernel.tasks.inference_task_base - (denoising (warm-up)) starting...: 0%| | 0/5000 [00:00<?, ?it/s]
00:19:04.804 INFO gcnvkernel.tasks.inference_task_base - (denoising (warm-up) epoch 1) ELBO: -157.564 +/- 0.000, SNR: 53855932356337432.0, T: 1.50: 0%| | 2/5000 [00:00<04:48, 17.31it/s]
00:19:04.906 INFO gcnvkernel.tasks.inference_task_base - (denoising (warm-up) epoch 1) ELBO: -122.394 +/- 8.569, SNR: 64.1, T: 1.50: 0%| | 7/5000 [00:00<02:23, 34.82it/s]
00:19:05.023 INFO gcnvkernel.tasks.inference_task_base - (denoising (warm-up) epoch 1) ELBO: -116.761 +/- 11.498, SNR: 7.1, T: 1.50: 0%| | 13/5000 [00:00<01:56, 42.73it/s]
...
...
100%|##########| 5000/5000 [01:31<00:00, 57.47it/s]
00:20:36.660 INFO gcnvkernel.tasks.inference_task_base - (denoising (warm-up) epoch 1) ELBO: -5.256 +/- 0.002, SNR: 1.7, T: 1.00: 100%|##########| 5000/5000 [01:31<00:00, 54.36it/s]
00:20:36.660 WARNING gcnvkernel.tasks.inference_task_base - Inference task completed successfully but convergence not achieved.
00:20:36.660 INFO gcnvkernel.tasks.task_cohort_denoising_calling - Instantiating the denoising model (main)...
Traceback (most recent call last):
File "/tmp/cohort_denoising_calling.11918075781700808988.py", line 190, in
shared_workspace, initial_params_supplier, warm_up_task)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/gcnvkernel/tasks/task_cohort_denoising_calling.py", line 162, in init
denoising_config, shared_workspace, initial_param_supplier)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/pymc3/model.py", line 197, in call
instance.init(*args, **kwargs)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/gcnvkernel/models/model_denoising_calling.py", line 872, in init
observed=shared_workspace.n_st)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/pymc3/distributions/distribution.py", line 39, in new
return model.Var(name, dist, data, total_size)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/pymc3/model.py", line 545, in Var
total_size=total_size, model=self)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/pymc3/model.py", line 970, in init
self.logp_elemwiset = distribution.logp(data)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/gcnvkernel/models/model_denoising_calling.py", line 838, in _copy_number_emission_logp
* shared_workspace.copy_number_values_c.dimshuffle('x', 'x', 0)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/tensor/var.py", line 570, in getitem
return self.take(args[axis], axis)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/tensor/var.py", line 614, in take
return theano.tensor.subtensor.take(self, indices, axis, mode)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/tensor/subtensor.py", line 2459, in take
a.dimshuffle(shuffle), indices).dimshuffle(shuffle)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/tensor/var.py", line 356, in dimshuffle
return op(self)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/op.py", line 670, in call
no_recycling=[])
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/op.py", line 955, in make_thunk
no_recycling)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/op.py", line 858, in make_c_thunk
output_storage=node_output_storage)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cc.py", line 1217, in make_thunk
keep_lock=keep_lock)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cc.py", line 1157, in compile
keep_lock=keep_lock)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cc.py", line 1624, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1155, in module_from_key
module = self._get_from_hash(module_hash, key, keep_lock=keep_lock)
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1055, in _get_from_hash
key_data.add_key(key, save_pkl=bool(key[0]))
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cmodule.py", line 519, in add_key
self.save_pkl()
File "/gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/lib/python3.6/site-packages/theano/gof/cmodule.py", line 540, in save_pkl
with open(self.key_pkl, 'wb') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/tmpd9btuv16/key.pkl'
WARNING (theano.gof.cmodule): A module that was loaded by this ModuleCache can no longer be read from file /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/tmpd9btuv16/m885ff006a95d626dac547a7bdfdb471bbf058622ece2b4435e42316c4012ea56.so... this could lead to problems.
00:20:37.313 WARNING theano.gof.cmodule - A module that was loaded by this ModuleCache can no longer be read from file /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/tmpd9btuv16/m885ff006a95d626dac547a7bdfdb471bbf058622ece2b4435e42316c4012ea56.so... this could lead to problems.
WARNING (theano.gof.cmodule): Removing key file /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/tmpd9btuv16/key.pkl because the corresponding module is gone from the file system.
00:20:37.321 WARNING theano.gof.cmodule - Removing key file /gpfs/hpc/home/lijc/xiangxud/.theano/compiledir_Linux-3.10-el7.x86_64-x86_64-with-centos-7.9.2009-Core-x86_64-3.6.10-64/tmpd9btuv16/key.pkl because the corresponding module is gone from the file system.
00:20:38.301 INFO GermlineCNVCaller - Shutting down engine
[December 12, 2024 at 12:20:38 AM CST] org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller done. Elapsed time: 3.81 minutes.
Runtime.totalMemory()=2147483648
org.broadinstitute.hellbender.utils.python.PythonScriptExecutorException:
python exited with 1
Command Line: python /tmp/cohort_denoising_calling.11918075781700808988.py --ploidy_calls_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/ploidy/ploidy-calls --output_calls_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-calls --output_tracking_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-tracking --random_seed=1984 --modeling_interval_list=/tmp/intervals4455187968449947169.tsv --output_model_path=/gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all/cohort_42-model --enable_explicit_gc_bias_modeling=True --read_count_tsv_files /tmp/NA06986.rc9044433718998481504.tsv /tmp/NA06989.rc12452120697434499065.tsv /tmp/NA07051.rc1885589103072221940.tsv /tmp/NA07347.rc1279844474304484703.tsv /tmp/NA11843.rc12301560452012374721.tsv /tmp/NA12878.rc11872029483327323378.tsv /tmp/NA18959.rc9459339676102199183.tsv /tmp/NA18960.rc6256338621902382719.tsv /tmp/NA18537.rc16791841972370257769.tsv /tmp/NA18976.rc6581421593900877568.tsv /tmp/NA19971.rc2228030607602018757.tsv /tmp/NA19920.rc12576022814788442980.tsv /tmp/HG00240.rc9196753230300445104.tsv /tmp/HG02154.rc14097840262549730733.tsv /tmp/HG02277.rc17532145266885259301.tsv /tmp/HG02555.rc311237558638496068.tsv --psi_s_scale=1.000000e-04 --mapping_error_rate=1.000000e-02 --depth_correction_tau=1.000000e+04 --q_c_expectation_mode=hybrid --num_samples_copy_ratio_approx=200 --max_bias_factors=5 --psi_t_scale=1.000000e-03 --log_mean_bias_std=1.000000e-01 --init_ard_rel_unexplained_variance=1.000000e-01 --num_gc_bins=20 --gc_curve_sd=1.000000e+00 --active_class_padding_hybrid_mode=50000 --enable_bias_factors=True --disable_bias_factors_in_active_class=False --p_alt=1.000000e-06 --cnv_coherence_length=1.000000e+04 --max_copy_number=5 --p_active=0.010000 --class_coherence_length=10000.000000 --learning_rate=1.000000e-02 --adamax_beta1=9.000000e-01 --adamax_beta2=9.900000e-01 --log_emission_samples_per_round=50 --log_emission_sampling_rounds=10 --log_emission_sampling_median_rel_error=5.000000e-03 --max_advi_iter_first_epoch=5000 --max_advi_iter_subsequent_epochs=200 --min_training_epochs=10 --max_training_epochs=50 --initial_temperature=1.500000e+00 --num_thermal_advi_iters=2500 --convergence_snr_averaging_window=500 --convergence_snr_trigger_threshold=1.000000e-01 --convergence_snr_countdown_window=10 --max_calling_iters=10 --caller_update_convergence_threshold=1.000000e-03 --caller_internal_admixing_rate=7.500000e-01 --caller_external_admixing_rate=1.000000e+00 --disable_caller=false --disable_sampler=false --disable_annealing=false
at org.broadinstitute.hellbender.utils.python.PythonExecutorBase.getScriptException(PythonExecutorBase.java:75)
at org.broadinstitute.hellbender.tools.copynumber.GermlineCNVCaller.doWork(GermlineCNVCaller.java:351)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:140)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:192)
at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:211)
at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
at org.broadinstitute.hellbender.Main.main(Main.java:289)
Using GATK jar /gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/share/gatk4-4.3.0.0-0/gatk-package-4.3.0.0-local.jar
Running:
java -Dsamjdk.use_async_io_read_samtools=false -Dsamjdk.use_async_io_write_samtools=true -Dsamjdk.use_async_io_write_tribble=false -Dsamjdk.compression_level=2 -jar /gpfs/hpc/home/lijc/xiangxud/software/miniconda3/envs/gatk4/share/gatk4-4.3.0.0-0/gatk-package-4.3.0.0-local.jar GermlineCNVCaller --run-mode COHORT -L /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/scatter/temp_0042_of_45/scattered.interval_list -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA06986.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA06989.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA07051.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA07347.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA11843.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA12878.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18959.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18960.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18537.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA18976.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA19971.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/NA19920.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG00240.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02154.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02277.tsv -I /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/read_counts/HG02555.tsv --contig-ploidy-calls /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/ploidy/ploidy-calls --annotated-intervals /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/preprocessed.annotated.tsv --interval-merging-rule OVERLAPPING_ONLY --output /gpfs/hpc/home/lijc/xiangxud/project/result/NGS_WES/3_tools_vcf/gatk4/info/cohort_all --verbosity DEBUG --output-prefix cohort_42
`

Before running GermlineCaller, I have manually specified the theano compilation directory for each of scattered jobs to avoid the compilelock, but there is another error after doing that.

[lovelycatZ]

according to #7411