There is no space left on the device

Hi,

with GATK 3.3-0 I am confronted with an error that was present in a much older version, but seemed resolved about a year ago:

ERROR MESSAGE: There is no space left on the device, so writing failed

There is 8TB left on the drive, no user limit. Sometimes re-running the exact same job works, sometimes not. Some jobs keep failing despite asking for an insane amount of memory on the cluster, given these are RNAseq bam files, the largest one being less than 7GB.

For example:

qsub -b y -cwd -N step3_145 -o step3_145.o -e step3_145.e -V -l h_vmem=40G /share/apps/java/oracle/1.8.0_11/bin/java -Xmx35G -jar /data/home/hhx037/GATK-3.3.0/GenomeAnalysisTK.jar -T SplitNCigarReads -R Homo_sapiens.GRCh37.75.dna.1-22XYMT.fa -I Analyses/file_dedup.bam -o Analyses/file_splittedcigar.bam -rf ReassignOneMappingQuality -RMQF 255 -RMQT 60 -U ALLOW_N_CIGAR_READS

Here is the log:

INFO 10:50:51,568 HelpFormatter - Executing as [email protected] on Linux 2.6.32-431.1.2.el6.x86_64 amd64; Java HotSpot(TM) 64-Bit Server VM 1.8.0_11-b12.
INFO 10:50:51,571 HelpFormatter - Date/Time: 2015/02/13 10:50:51
INFO 10:50:51,571 HelpFormatter - --------------------------------------------------------------------------------
INFO 10:50:51,576 HelpFormatter - --------------------------------------------------------------------------------
INFO 10:50:52,503 GenomeAnalysisEngine - Strictness is SILENT
INFO 10:50:52,827 GenomeAnalysisEngine - Downsampling Settings: No downsampling
INFO 10:50:52,861 SAMDataSource$SAMReaders - Initializing SAMRecords in serial
INFO 10:50:52,876 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.01
INFO 10:50:53,021 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files
INFO 10:50:53,027 GenomeAnalysisEngine - Done preparing for traversal
INFO 10:50:53,030 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING]
INFO 10:50:53,030 ProgressMeter - | processed | time | per 1M | | total | remaining
INFO 10:50:53,030 ProgressMeter - Location | reads | elapsed | reads | completed | runtime | runtime
INFO 10:50:53,047 ReadShardBalancer$1 - Loading BAM index data
INFO 10:50:53,050 ReadShardBalancer$1 - Done loading BAM index data
INFO 10:51:23,404 ProgressMeter - 1:1477348 702953.0 30.0 s 43.0 s 0.0% 17.5 h 17.5 h
INFO 10:52:32,660 ProgressMeter - 1:16909108 1202983.0 99.0 s 82.0 s 0.5% 5.0 h 5.0 h
INFO 10:53:09,769 ProgressMeter - 1:21069702 1302985.0 2.3 m 104.0 s 0.7% 5.6 h 5.5 h
INFO 10:53:49,083 ProgressMeter - 1:27951393 1803181.0 2.9 m 97.0 s 0.9% 5.4 h 5.4 h
INFO 10:54:29,275 ProgressMeter - 1:32739969 2103299.0 3.6 m 102.0 s 1.1% 5.7 h 5.6 h
INFO 10:55:09,177 ProgressMeter - 1:36643589 2203300.0 4.3 m 116.0 s 1.2% 6.0 h 5.9 h
INFO 10:55:45,643 ProgressMeter - 1:39854010 2303302.0 4.9 m 2.1 m 1.3% 6.3 h 6.2 h
INFO 10:56:25,147 ProgressMeter - 1:40542516 2403303.0 5.5 m 2.3 m 1.3% 7.0 h 6.9 h
INFO 10:57:10,934 ProgressMeter - 1:40654849 2503322.0 6.3 m 2.5 m 1.3% 8.0 h 7.9 h
INFO 10:57:54,084 ProgressMeter - 1:43162895 2503322.0 7.0 m 2.8 m 1.4% 8.4 h 8.3 h
INFO 10:58:24,149 ProgressMeter - 1:45244391 2703426.0 7.5 m 2.8 m 1.5% 8.6 h 8.4 h
INFO 10:58:56,749 ProgressMeter - 1:53716450 2803427.0 8.1 m 2.9 m 1.7% 7.7 h 7.6 h
INFO 10:59:38,928 ProgressMeter - 1:86821106 3103432.0 8.8 m 2.8 m 2.8% 5.2 h 5.1 h
INFO 11:00:11,337 ProgressMeter - 1:93301870 3303437.0 9.3 m 2.8 m 3.0% 5.1 h 5.0 h
INFO 11:01:13,113 ProgressMeter - 1:115252321 3803590.0 10.3 m 2.7 m 3.7% 4.6 h 4.5 h
INFO 11:02:02,172 ProgressMeter - 1:145441389 4303778.0 11.2 m 2.6 m 4.7% 4.0 h 3.8 h
INFO 11:02:38,237 ProgressMeter - 1:150547232 4703871.0 11.8 m 2.5 m 4.9% 4.0 h 3.8 h
INFO 11:03:09,693 ProgressMeter - 1:153362937 5003904.0 12.3 m 2.5 m 5.0% 4.1 h 3.9 h
INFO 11:03:39,934 ProgressMeter - 1:155984762 5403968.0 12.8 m 2.4 m 5.0% 4.2 h 4.0 h
INFO 11:04:05,477 GATKRunReport - Uploaded run statistics report to AWS S3

ERROR ------------------------------------------------------------------------------------------
ERROR A USER ERROR has occurred (version 3.3-0-g37228af):
ERROR
ERROR This means that one or more arguments or inputs in your command are incorrect.
ERROR The error message below tells you what is the problem.
ERROR
ERROR If the problem is an invalid argument, please check the online documentation guide
ERROR (or rerun your command with --help) to view allowable command-line arguments for this tool.
ERROR
ERROR Visit our website and forum for extensive documentation and answers to
ERROR commonly asked questions http://www.broadinstitute.org/gatk
ERROR
ERROR Please do NOT post this error to the GATK forum unless you have really tried to fix it yourself.
ERROR
ERROR MESSAGE: There is no space left on the device, so writing failed
ERROR ------------------------------------------------------------------------------------------

I understand temporary files may be large, but not that large. Are the temporary files written in the working directory (as I believe should be the case), or are they written in GATK installation directory?

Also, note I never run into this problem with the previous version.

Any idea?

Cheers,

Stephane

Tagged:

Best Answer

Answers

  • tommycarstensentommycarstensen United KingdomMember

    I am not an expert on java, but you can set your tmp directory location like this:

    java -Djava.io.tmpdir=tmp
    

    I believe the default location is /tmp in the root, which can fill up quickly, when on a shared system.

  • sbourgeoissbourgeois London, UKMember

    Hi Tommy,

    thanks a lot, that's a useful thing to know.

    Unfortunately I don't think that's the issue in this case; while my jobs were running, a colleague suddenly found herself unable to use our storage space, not even to copy a small file, with errors signalling her the drive was full (while a df -kh clearly indicates there is lot of free space). Now that no jobs are running, I've copied a 6GB bam file without any issue. Yet, I'm still getting the "no space left" error from GATK, even when attempting to run a single job.

    I've contacted our IT department, hopefully they can show me a way to monitor our storage I/O while I'm attempting to run GATK, so I can see what's creating the issue.

    Thanks again,

    Stephane

  • medhatmedhat PolandMember

    Hi,
    Did you find any solution cause I am facing the same issue

    Issue · Github
    by Sheila

    Issue Number
    867
    State
    closed
    Last Updated
    Assignee
    Array
    Milestone
    Array
    Closed By
    vdauwera
Sign In or Register to comment.