Code Exception when running UG with multiple threads

vasya

Hi --

I'm running the UG on a subset of 1000G data (100 samples subset to chr20), and I would love to throw some more threads at the problem. However, using the -nt flag throws the following code exception:

ERROR ------------------------------------------------------------------------------------------

ERROR stack trace

java.lang.InternalError at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:815) at sun.misc.URLClassPath.getResource(URLClassPath.java:195) at sun.misc.URLClassPath.getResource(URLClassPath.java:247) at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1298) at java.lang.ClassLoader.getResource(ClassLoader.java:1135) at java.lang.ClassLoader.getResource(ClassLoader.java:1133) at java.lang.ClassLoader.getSystemResource(ClassLoader.java:1260) at java.lang.ClassLoader.getSystemResourceAsStream(ClassLoader.java:1363) at java.lang.Class.getResourceAsStream(Class.java:2045) at javax.xml.stream.SecuritySupport$4.run(SecuritySupport.java:92) at java.security.AccessController.doPrivileged(Native Method) at javax.xml.stream.SecuritySupport.getResourceAsStream(SecuritySupport.java:87) at javax.xml.stream.FactoryFinder.findJarServiceProvider(FactoryFinder.java:285) at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:253) at javax.xml.stream.FactoryFinder.find(FactoryFinder.java:177) at javax.xml.stream.XMLInputFactory.newInstance(XMLInputFactory.java:153) at org.simpleframework.xml.stream.NodeBuilder.(NodeBuilder.java:48) at org.simpleframework.xml.core.Persister.write(Persister.java:1000) at org.simpleframework.xml.core.Persister.write(Persister.java:982) at org.simpleframework.xml.core.Persister.write(Persister.java:963) at org.broadinstitute.sting.gatk.phonehome.GATKRunReport.postReportToStream(GATKRunReport.java:236) at org.broadinstitute.sting.gatk.phonehome.GATKRunReport.postReportToAWSS3(GATKRunReport.java:333) at org.broadinstitute.sting.gatk.phonehome.GATKRunReport.postReport(GATKRunReport.java:215) at org.broadinstitute.sting.gatk.CommandLineExecutable.generateGATKRunReport(CommandLineExecutable.java:157) at org.broadinstitute.sting.gatk.CommandLineExecutable.execute(CommandLineExecutable.java:116) at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:236) at org.broadinstitute.sting.commandline.CommandLineProgram.start(CommandLineProgram.java:146) at org.broadinstitute.sting.gatk.CommandLineGATK.main(CommandLineGATK.java:93) Caused by: java.io.FileNotFoundException: /usr/java/jdk1.7.0_03/jre/lib/resources.jar (Too many open files) at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.(ZipFile.java:214) at java.util.zip.ZipFile.(ZipFile.java:144) at java.util.jar.JarFile.(JarFile.java:152) at java.util.jar.JarFile.(JarFile.java:89) at sun.misc.URLClassPath$JarLoader.getJarFile(URLClassPath.java:706) at sun.misc.URLClassPath$JarLoader.access$600(URLClassPath.java:587) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:667) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:660) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:659) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:813) ... 27 more

ERROR ------------------------------------------------------------------------------------------

ERROR A GATK RUNTIME ERROR has occurred (version 2.1-13-g1706365):

ERROR

ERROR Please visit the wiki to see if this is a known problem

ERROR If not, please post the error, with stack trace, to the GATK forum

ERROR Visit our website and forum for extensive documentation and answers to

ERROR commonly asked questions http://www.broadinstitute.org/gatk

ERROR

ERROR MESSAGE: Code exception (see stack trace for error itself)

ERROR ------------------------------------------------------------------------------------------

The code is called with the following code. Without the -nt flag it executes normally.

java -jar /userhome/vasya/apps/GenomeAnalysisTK-2.1-13/GenomeAnalysisTK.jar \ -T UnifiedGenotyper \ $(echo $bamFiles) \ -R ${fasta} \ -nt 2 \ -o ${ug_vcf_root}

Are there any known issues with the threading options in the UG for GATK2?

Any suggestions for where to look next would be much appreciated! Its hard to know where to start with this.

Geraldine_VdAuwera

Hi there,

This could be a limitation of your system. If you're having trouble getting -nt to work for you, as an alternative you can look into using Queue with scatter-gather.

ebanks

If you look at the error message you will see that you have "Too many open files", a problem covered in other threads.