Nextflow crash on AWS Batch with transition error

[hkarakurt8742]

Hello everyone, I am having a problem with AWS Batch.

executor >  awsbatch (57)
[d6/1f982d] process > BWA_ALIGNMENT (Alignment on R187-POOL-25_S25_L001)          [100%] 6 of 6, failed: 1
[c6/f78a3c] process > FASTQC (FASTQC on R187-POOL-25_S25_L001)                    [100%] 24 of 24
[-        ] process > MULTIQC                                                     -
[-        ] process > MARKDUPLICATES (MarkDuplicates on R187-POOL-01_S1_L001.bam) -
[-        ] process > BASERECALIBRATOR                                            -
[-        ] process > APPLYBQSR                                                   -
[-        ] process > POOL_SPLITTER                                               -
[-        ] process > VARIANT_CALLING                                             -
[-        ] process > VCF_ANNOTATION                                              -
[-        ] process > VCF2TSV                                                     -
ERROR ~ Error executing process > 'BWA_ALIGNMENT (Alignment on R187-POOL-16_S16_L001)'

Caused by:
  Essential container in task exited - CannotInspectContainerError: Could not transition to inspecting; timed out after waiting 30s

Command executed:

  bwa mem -M -R '@RG\tID:Sample\tLB:lib\tSM:Sample\tPL:illumina\tCN:Sample' -t 12 hg19_v0_Homo_sapiens_assembly19.fasta R187-POOL-16_S16_L001_R1_001.fastq.gz R187-POOL-16_S16_L001_R2_001.fastq.gz  | samtools sort -@ 12 -o R187-POOL-16_S16_L001.bam -

Command exit status:
  0

Command output:
  (empty)

Command error:
  [M::bwa_idx_load_from_disk] read 0 ALT contigs
  [M::process] read 659908 sequences (74763087 bp)...
  [M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (57, 86100, 105, 49)
  [M::mem_pestat] analyzing insert size distribution for orientation FF...
  [M::mem_pestat] (25, 50, 75) percentile: (101, 214, 492)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1274)
  [M::mem_pestat] mean and std.dev: (287.94, 254.84)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 1665)
  [M::mem_pestat] analyzing insert size distribution for orientation FR...
  [M::mem_pestat] (25, 50, 75) percentile: (63, 97, 144)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 306)
  [M::mem_pestat] mean and std.dev: (108.09, 57.59)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 387)
  [M::mem_pestat] analyzing insert size distribution for orientation RF...
  [M::mem_pestat] (25, 50, 75) percentile: (86, 184, 859)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2405)
  [M::mem_pestat] mean and std.dev: (288.90, 419.84)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 3178)
  [M::mem_pestat] analyzing insert size distribution for orientation RR...
  [M::mem_pestat] (25, 50, 75) percentile: (80, 206, 523)
  [M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1409)
  [M::mem_pestat] mean and std.dev: (266.86, 277.91)
  [M::mem_pestat] low and high boundaries for proper pairs: (1, 1852)
  [M::mem_pestat] skip orientation FF
  [M::mem_pestat] skip orientation RF
  [M::mem_pestat] skip orientation RR
  [M::mem_process_seqs] Processed 659908 reads in 235.092 CPU sec, 19.759 real sec
  [main] Version: 0.7.17-r1188
  [main] CMD: bwa mem -M -R @RG\tID:Sample\tLB:lib\tSM:Sample\tPL:illumina\tCN:Sample -t 12 hg19_v0_Homo_sapiens_assembly19.fasta R187-POOL-16_S16_L001_R1_001.fastq.gz R187-POOL-16_S16_L001_R2_001.fastq.gz
  [main] Real time: 25.762 sec; CPU: 240.056 sec
  [bam_sort_core] merging from 0 files and 12 in-memory blocks...

Work dir:
  s3://gene2var-nextflow/test_folder/78/7180148c2cb524b4379b8c0ba18063

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run`

 -- Check '.nextflow.log' file for details

[AWS BATCH] Waiting jobs reaper to complete (20 jobs to be terminated)

Sometimes I am having the same issue in another steps but I do not know how to solve it. I am using an AWS Linux ECS-Optimized AMI with 8 GB root and 142 GB EBS volume. What might be the problem here?

My config:

params {
    reads = "s3://R187/*R{1,2}_001.fastq.gz"
    genome_file = "s3:///hg19_v0_Homo_sapiens_assembly19.fasta"
    genome_dict = "s3:///hg19_v0_Homo_sapiens_assembly19.dict"
    genome_file_idx = "s3:///hg19_v0_Homo_sapiens_assembly19.fasta.fai"
    genome_loc = "s3://"
    known_sites = "s3://references/dbsnp/hg19_v0_Homo_sapiens_assembly19.dbsnp138.vcf.gz"
    known_sites_idx = "s3://references/dbsnp/hg19_v0_Homo_sapiens_assembly19.dbsnp138.vcf.gz.tbi"
    outdir = "s3://test_folder/R187_results"
    pool_info = "s3://BED_Files/R187_pool_info.txt"
    bed_location = "s3://BED_Files/R187_bed_files/"
    vep_cache = "s3://vep_data"
    vep_plugin = "s3://vep_data/Plugins"
    cpus = "12"
}

profile {
aws {
    accessKey = '***********'
    secretKey = '******************************'
    docker.enabled = true
    region = 'eu-west-3'
    }
}


aws.batch.maxParallelTransfers = 5

process.executor = 'awsbatch'

process.container = 'aligner:latest'

process.container = 'annotation:latest'

process.container = 'variant_caller:latest'

process.queue = 'batch_queue'

aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'

workDir = 's3://test_folder/'

Why I am having this error?

[bentsherman]

I have usually seen this error when the task runs out of disk space. Maybe 150 GB is not enough to hold the container image + input data?

You can also try Fusion. It will lazy load the input data instead of downloading everything at once, and it will respect the disk space that you give it.