Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

Unable to create temporary file for stub

Hi there!

I'm currently trying to run the Unified Genotyper on 99 whole-exome sampels however after ~12h I received the message:

ERROR MESSAGE: Failure working with the tmp directory /localscratch2/. Override with -Djava.io.tmpdir=X on the command line to a bigger/better file system. Exact error was Unable to create temporary file for stub: org.broadinstitute.sting.gatk.io.stubs.VariantContextWriterStub

First I thought my tmp dir is too small, so I increased it to 4,4 TB. However, I received the same error after about 12h again (So it seems to be independent of the tmp dir size).
Do you have any suggestion what the problem could be ? Maybe the FileHandles ? (Although I did not see any error message complaining about the number of opened files.)

Thanks a lot!

Answers

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @‌max

    Hi,

    This could be an issue with your user permissions on the filesystem.

    What do you mean by "increased it to 4,4 TB"? And, have you checked that your user account allows you to actually use that much space?

    Thanks.

    -Sheila

  • MaxMax Member

    Hi Sheila,

    thanks for your answer!

    Well, I specified a tmp dir with 4.4 TB space using the -Djava.io.tmpdir option. Everything works fine until about chr 7, which is reached after about 12 hours. Then this error occurs.

    There should be no quota that limits the amount of used space for my account. So I should have full permissions to write as much as required .

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @Max

    Hi,

    Please try running "quota -v" to check the quotas, just to be sure.

    Thanks,
    Sheila

  • MaxMax Member

    Hi Sheila,

    "quota -v" gives no output. I also talked to our SysAdmin. He confirmed that there should be no limit for /localscratch2/ (which is the tmp dir).

    Is there any possibility to produce a more detailed error log, with more information about the exact reason ?

  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @Max‌

    Hi,

    You can try -l DEBUG, but I'm not sure if that will help as this seems to be a general java issue, not GATK-specific.

    You can also try running a non-GATK java program like Picard (but make sure it's one that writes to tmp) to test that.

    -Sheila

  • MaxMax Member

    Hi ,

    thanks for your advice!

    However, I've now splitted the data set to avoid the problem. I will try to fix the problem later on and will let you know if I'm able to solve it.

  • teamcoopteamcoop Ontario, CanadaMember

    Just wanted to note that I had a similar error:

    ERROR MESSAGE: Failure working with the tmp directory tmpdir/. Override with -Djava.io.tmpdir=X on the command line to a bigger/better file system. Exact error was Unable to create directory

    when running a number of jobs in parallel. I was already using -Djava.io.tmpdir=tmpdir/ on the command line, so was a bit confused. However, I was able to get rid of the error by delaying a portion of the jobs (i.e. adding a 'sleep 10m' at the beginning of those scripts).

  • pdexheimerpdexheimer Member ✭✭✭✭

    You may want to talk to your infrastructure people - this sounds like a failure of the underlying file system. It looks like you're trying to create a number of directories simultaneously, and the file system throws up its hands and fails a subset of them. Someone who's familiar with your storage system should be able to confirm/deny that hypothesis, and if it's true give you some guidance on how many simultaneous jobs you can run against it.

    Another thought - if you're running into overload on something as simple as creating directories, you might cause bigger problems with large-scale read/writes. Before we upgraded our storage to a proper, massively-parallel system, I occasionally brought down entire racks by over-parallelizing. (I could reliably do it with Tophat, GATK's a bit smarter in its file access strategies)

Sign In or Register to comment.