We've moved!
This site is now read-only. You can find our new documentation site and support forum for posting questions here.
Be sure to read our welcome blog!

StructuralVariationDiscoveryPipelineSpark Erro: CouldNotReadImageException

I am running the following script with GATK 4.0.0.1.1.

gatk --java-options "-Djava.io.tmpdir=tmp" StructuralVariationDiscoveryPipelineSpark \
-R $REF \
--aligner-index-image hdfs:///user/farrell/ref/reference.img \
--kmers-to-ignore GRCh38_ignored_kmers.txt \
--contig-sam-file $SAMPLE.contig-sam-file\
-I $CRAM_DIR/$SAMPLE.cram \
-O hdfs:///user/farrell/$CENTER/$SAMPLE.vcf.gz -- --spark-runner SPARK --spark-master yarn --executor-memory=96G --driver-memory 24g --driver-cores 4 --num-executors=8 --executor-cores=8

The during state 12, the run is getting an error:
org.broadinstitute.hellbender.utils.bwa.CouldNotReadImageException: cannot read the image file

The image file is present and there were no error messages when generating it...

hadoop fs -ls hdfs:///user/farrell/ref/reference.img
-rw-r--r-- 3 farrell casa 5630892103 2018-02-21 20:19 hdfs:///user/farrell/ref/reference.img

Here is the log from stage 12 when the error occurs. If there really is a problem with the image, it would be nice to know before stage 12. if there is a problem with the image file, it would point to the BwaMemIndexImageCreator tool. Any suggestions or is this a bug somewhere?

18/02/22 08:01:20 INFO scheduler.TaskSetManager: Starting task 156.0 in stage 12.0 (TID 3180, scc-q15.scc.bu.edu, executor 3, partition 156, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:20 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 12.0 (TID 3024) in 2994 ms on scc-q15.scc.bu.edu (executor 3) (93/24528)
18/02/22 08:01:30 INFO scheduler.TaskSetManager: Starting task 157.0 in stage 12.0 (TID 3181, scc-q09.scc.bu.edu, executor 7, partition 157, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:30 WARN scheduler.TaskSetManager: Lost task 100.0 in stage 12.0 (TID 3124, scc-q09.scc.bu.edu, executor 7): org.broadinstitute.hellbender.utils.bwa.CouldNotReadImageException: cannot read the image file 'hdfs:///user/farrell/ref/reference.img': is empty or is not readable
at org.broadinstitute.hellbender.utils.bwa.BwaMemIndex.assertNonEmptyReadableImageFile(BwaMemIndex.java:355)
at org.broadinstitute.hellbender.utils.bwa.BwaMemIndex.(BwaMemIndex.java:331)
at org.broadinstitute.hellbender.utils.bwa.BwaMemIndexCache.getInstance(BwaMemIndexCache.java:24)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.FermiLiteAssemblyHandler.apply(FermiLiteAssemblyHandler.java:68)
at org.broadinstitute.hellbender.tools.spark.sv.evidence.FermiLiteAssemblyHandler.apply(FermiLiteAssemblyHandler.java:21)
at org.apache.spark.api.java.JavaPairRDD$$anonfun$toScalaFunction$1.apply(JavaPairRDD.scala:1040)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at scala.collection.AbstractIterator.to(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:935)
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:935)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1944)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

18/02/22 08:01:31 INFO scheduler.TaskSetManager: Starting task 100.1 in stage 12.0 (TID 3182, scc-q09.scc.bu.edu, executor 7, partition 100, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:31 INFO scheduler.TaskSetManager: Lost task 99.0 in stage 12.0 (TID 3123) on scc-q09.scc.bu.edu, executor 7: org.broadinstitute.hellbender.utils.bwa.CouldNotReadImageException (cannot read the image file 'hdfs:///user/farrell/ref/reference.img': is empty or is not readable) [duplicate 1]
18/02/22 08:01:32 INFO scheduler.TaskSetManager: Starting task 99.1 in stage 12.0 (TID 3183, scc-q15.scc.bu.edu, executor 3, partition 99, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:32 INFO scheduler.TaskSetManager: Lost task 145.0 in stage 12.0 (TID 3169) on scc-q15.scc.bu.edu, executor 3: org.broadinstitute.hellbender.utils.bwa.CouldNotReadImageException (cannot read the image file 'hdfs:///user/farrell/ref/reference.img': is empty or is not readable) [duplicate 2]
18/02/22 08:01:33 INFO scheduler.TaskSetManager: Starting task 145.1 in stage 12.0 (TID 3184, scc-q09.scc.bu.edu, executor 7, partition 145, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:33 INFO scheduler.TaskSetManager: Lost task 111.0 in stage 12.0 (TID 3135) on scc-q09.scc.bu.edu, executor 7: org.broadinstitute.hellbender.utils.bwa.CouldNotReadImageException (cannot read the image file 'hdfs:///user/farrell/ref/reference.img': is empty or is not readable) [duplicate 3]
18/02/22 08:01:34 INFO scheduler.TaskSetManager: Starting task 111.1 in stage 12.0 (TID 3185, scc-q15.scc.bu.edu, executor 3, partition 111, PROCESS_LOCAL, 5834 bytes)
18/02/22 08:01:34 INFO scheduler.TaskSetManager: Lost task 151.0 in

Best Answer

  • jfarrelljfarrell ✭✭
    Accepted Answer

    After playing with this, I eventually found my own solution and the job finished to completion.

    The image file was fine. The image file justed needed to be uploaded to the spark application containers to avoid triggering the exception at stage 12. This is done with the --files command. The --aligner-index-image does not need the full path since the file will now be in the working directory of the pipeline.

    The result is the tasks avoid reading the file over the network as it will be found on local hdfs storage. My guess is there were a very large number of tasks trying to read the img file. This resulted in some timing out and returning an empty/truncated file which in turn triggered the CouldNotReadImageException.

    REF=file:///restricted/projectnb/casa/ref/GRCh38_full_analysis_set_plus_decoy_hla.fa
    gatk --java-options "-Djava.io.tmpdir=tmp" StructuralVariationDiscoveryPipelineSpark \
    -R $REF \
    --aligner-index-image GRCh38_full_analysis_set_plus_decoy_hla.fa.img \
    --kmers-to-ignore GRCh38_ignored_kmers.txt \
    --contig-sam-file $SAMPLE.contig-sam-file\
    -I $CRAM_DIR/$SAMPLE.cram \
    -O hdfs:///user/farrell/$CENTER/sv/$SAMPLE.sv.vcf.gz -- --spark-runner SPARK --spark-master yarn --executor-memory=28G --driver-memory 16g --driver-cores 4 --executor-cores=5 --num-executors 8 --files $REF.img

Answers

  • jfarrelljfarrell Member ✭✭
    Accepted Answer

    After playing with this, I eventually found my own solution and the job finished to completion.

    The image file was fine. The image file justed needed to be uploaded to the spark application containers to avoid triggering the exception at stage 12. This is done with the --files command. The --aligner-index-image does not need the full path since the file will now be in the working directory of the pipeline.

    The result is the tasks avoid reading the file over the network as it will be found on local hdfs storage. My guess is there were a very large number of tasks trying to read the img file. This resulted in some timing out and returning an empty/truncated file which in turn triggered the CouldNotReadImageException.

    REF=file:///restricted/projectnb/casa/ref/GRCh38_full_analysis_set_plus_decoy_hla.fa
    gatk --java-options "-Djava.io.tmpdir=tmp" StructuralVariationDiscoveryPipelineSpark \
    -R $REF \
    --aligner-index-image GRCh38_full_analysis_set_plus_decoy_hla.fa.img \
    --kmers-to-ignore GRCh38_ignored_kmers.txt \
    --contig-sam-file $SAMPLE.contig-sam-file\
    -I $CRAM_DIR/$SAMPLE.cram \
    -O hdfs:///user/farrell/$CENTER/sv/$SAMPLE.sv.vcf.gz -- --spark-runner SPARK --spark-master yarn --executor-memory=28G --driver-memory 16g --driver-cores 4 --executor-cores=5 --num-executors 8 --files $REF.img

  • SheilaSheila Broad InstituteMember, Broadie ✭✭✭✭✭

    @jfarrell
    Hi,

    Thanks for posting this.

    -Sheila

Sign In or Register to comment.