(howto) Test your Queue installation

Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin
edited June 2013 in Tutorials

Objective

Test that Queue is correctly installed, and that the supporting tools like Java are in your path.

Prerequisites

  • Basic familiarity with the command-line environment
  • Understand what is a PATH variable
  • GATK installed
  • Queue downloaded and placed on path

Steps

  1. Invoke the Queue usage/help message
  2. Troubleshooting

1. Invoke the Queue usage/help message

The command we're going to run is a very simple command that asks Queue to print out a list of available command-line arguments and options. It is so simple that it will ALWAYS work if your Queue package is installed correctly.

Note that this command is also helpful when you're trying to remember something like the right spelling or short name for an argument and for whatever reason you don't have access to the web-based documentation.

Action

Type the following command:

java -jar <path to Queue.jar> --help

replacing the <path to Queue.jar> bit with the path you have set up in your command-line environment.

Expected Result

You should see usage output similar to the following:

usage: java -jar Queue.jar -S <script> [-jobPrefix <job_name_prefix>] [-jobQueue <job_queue>] [-jobProject <job_project>]
       [-jobSGDir <job_scatter_gather_directory>] [-memLimit <default_memory_limit>] [-runDir <run_directory>] [-tempDir
       <temp_directory>] [-emailHost <emailSmtpHost>] [-emailPort <emailSmtpPort>] [-emailTLS] [-emailSSL] [-emailUser
       <emailUsername>] [-emailPass <emailPassword>] [-emailPassFile <emailPasswordFile>] [-bsub] [-run] [-dot <dot_graph>]
       [-expandedDot <expanded_dot_graph>] [-startFromScratch] [-status] [-statusFrom <status_email_from>] [-statusTo
       <status_email_to>] [-keepIntermediates] [-retry <retry_failed>] [-l <logging_level>] [-log <log_to_file>] [-quiet]
       [-debug] [-h]

 -S,--script <script>                                                      QScript scala file
 -jobPrefix,--job_name_prefix <job_name_prefix>                            Default name prefix for compute farm jobs.
 -jobQueue,--job_queue <job_queue>                                         Default queue for compute farm jobs.
 -jobProject,--job_project <job_project>                                   Default project for compute farm jobs.
 -jobSGDir,--job_scatter_gather_directory <job_scatter_gather_directory>   Default directory to place scatter gather
                                                                           output for compute farm jobs.
 -memLimit,--default_memory_limit <default_memory_limit>                   Default memory limit for jobs, in gigabytes.
 -runDir,--run_directory <run_directory>                                   Root directory to run functions from.
 -tempDir,--temp_directory <temp_directory>                                Temp directory to pass to functions.
 -emailHost,--emailSmtpHost <emailSmtpHost>                                Email SMTP host. Defaults to localhost.
 -emailPort,--emailSmtpPort <emailSmtpPort>                                Email SMTP port. Defaults to 465 for ssl,
                                                                           otherwise 25.
 -emailTLS,--emailUseTLS                                                   Email should use TLS. Defaults to false.
 -emailSSL,--emailUseSSL                                                   Email should use SSL. Defaults to false.
 -emailUser,--emailUsername <emailUsername>                                Email SMTP username. Defaults to none.
 -emailPass,--emailPassword <emailPassword>                                Email SMTP password. Defaults to none. Not
                                                                           secure! See emailPassFile.
 -emailPassFile,--emailPasswordFile <emailPasswordFile>                    Email SMTP password file. Defaults to none.
 -bsub,--bsub_all_jobs                                                     Use bsub to submit jobs
 -run,--run_scripts                                                        Run QScripts.  Without this flag set only
                                                                           performs a dry run.
 -dot,--dot_graph <dot_graph>                                              Outputs the queue graph to a .dot file.  See:
                                                                           http://en.wikipedia.org/wiki/DOT_language
 -expandedDot,--expanded_dot_graph <expanded_dot_graph>                    Outputs the queue graph of scatter gather to
                                                                           a .dot file.  Otherwise overwrites the
                                                                           dot_graph
 -startFromScratch,--start_from_scratch                                    Runs all command line functions even if the
                                                                           outputs were previously output successfully.
 -status,--status                                                          Get status of jobs for the qscript
 -statusFrom,--status_email_from <status_email_from>                       Email address to send emails from upon
                                                                           completion or on error.
 -statusTo,--status_email_to <status_email_to>                             Email address to send emails to upon
                                                                           completion or on error.
 -keepIntermediates,--keep_intermediate_outputs                            After a successful run keep the outputs of
                                                                           any Function marked as intermediate.
 -retry,--retry_failed <retry_failed>                                      Retry the specified number of times after a
                                                                           command fails.  Defaults to no retries.
 -l,--logging_level <logging_level>                                        Set the minimum level of logging, i.e.
                                                                           setting INFO get's you INFO up to FATAL,
                                                                           setting ERROR gets you ERROR and FATAL level
                                                                           logging.
 -log,--log_to_file <log_to_file>                                          Set the logging location
 -quiet,--quiet_output_mode                                                Set the logging to quiet mode, no output to
                                                                           stdout
 -debug,--debug_mode                                                       Set the logging file string to include a lot
                                                                           of debugging information (SLOW!)
 -h,--help                                                                 Generate this help message

If you see this message, your Queue installation is ok. You're good to go! If you don't see this message, and instead get an error message, proceed to the next section on troubleshooting.


2. Troubleshooting

Let's try to figure out what's not working.

Action

First, make sure that your Java version is at least 1.6, by typing the following command:

java -version

Expected Result

You should see something similar to the following text:

java version "1.6.0_12"
Java(TM) SE Runtime Environment (build 1.6.0_12-b04)
Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode)  

Remedial actions

If the version is less then 1.6, install the newest version of Java onto the system. If you instead see something like

java: Command not found  

make sure that java is installed on your machine, and that your PATH variable contains the path to the java executables.

On a Mac running OS X 10.5+, you may need to run /Applications/Utilities/Java Preferences.app and drag Java SE 6 to the top to make your machine run version 1.6, even if it has been installed.

Post edited by Geraldine_VdAuwera on

Geraldine Van der Auwera, PhD

Comments

  • jphekmanjphekman University of IllinoisPosts: 5Member

    I got the error message:

    Caused by: java.net.UnknownHostException: ansci-5ln0zv1.ansci.illinois.edu: ansci-5ln0zv1.ansci.illinois.edu: Name or service not known

    It appears that Queue is attempting to phone home, and being flummoxed by the fact that name service for this machine doesn't work outside of our firewall? I would be happy to give it the machine's IP to use instead, but am not sure how.

    I also tried adding "-et NO_ET", but that did not make a difference.

    I'd love to know how to get this working -- I want to parallelize some SNP calls so I don't have to wait two weeks for the job to finish running!

    Thanks very much, Jessica

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin

    Hi Jessica,

    If you are working behind a firewall, you'll be better off just deactivating the Phone Home feature. You'll need to request a deactivation key for the NO_ET setting to work. See http://www.broadinstitute.org/gatk/request-key for details.

    Geraldine Van der Auwera, PhD

  • jphekmanjphekman University of IllinoisPosts: 5Member

    Thanks! I have submitted my request. I appreciate the quick response!

  • jphekmanjphekman University of IllinoisPosts: 5Member

    I got the key, but am unable to get Queue to work. The command line is:

    java -jar /usr/local/Queue/Queue.jar -et NO_ET -K hekman2_illinois.edu.key -h

    And I'm getting the same error message as before, so I conclude that phone home is still enabled. The key file is correctly named and in the current working directory.

    Thanks!

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin

    The problem here is that the key is a GATK argument, not a Queue argument. You can't actually pass GATK arguments directly to Queue from the command line; you need to write a scala script, and within that, you either define arguments to pass in the key at command line, or you hard-code it into the script for convenience.

    I'm thinking your original issue wasn't a phone home problem since Queue itself doesn't phone home. I had assumed you ran a script that called GATK, but if not, then something else is wrong. Can you please post the full error message you got?

    Geraldine Van der Auwera, PhD

  • jphekmanjphekman University of IllinoisPosts: 5Member

    Here 'tis:

    [hekman2@ansci-5ln0zv1 ~]$ java -jar /usr/local/Queue/Queue.jar -et NO_ET -K hekman2_illinois.edu.key -h Exception in thread "main" java.lang.ExceptionInInitializerError at org.broadinstitute.sting.queue.engine.QGraphSettings.(QGraphSettings.scala:65) at org.broadinstitute.sting.queue.QCommandLine.(QCommandLine.scala:86) at org.broadinstitute.sting.queue.QCommandLine$.main(QCommandLine.scala:50) at org.broadinstitute.sting.queue.QCommandLine.main(QCommandLine.scala) Caused by: java.net.UnknownHostException: ansci-5ln0zv1.ansci.illinois.edu: ansci-5ln0zv1.ansci.illinois.edu: Name or service not known at java.net.InetAddress.getLocalHost(InetAddress.java:1473) at org.broadinstitute.sting.queue.util.SystemUtils$.(SystemUtils.scala:36) at org.broadinstitute.sting.queue.util.SystemUtils$.(SystemUtils.scala) ... 4 more Caused by: java.net.UnknownHostException: ansci-5ln0zv1.ansci.illinois.edu: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293) at java.net.InetAddress.getLocalHost(InetAddress.java:1469) ... 6 more

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin

    Oh yeah that's something else -- but it's not anything I recognize, sorry. I would recommend contacting your IT department to ask for help with your local environment.

    Geraldine Van der Auwera, PhD

  • jphekmanjphekman University of IllinoisPosts: 5Member

    Phew! So as it turns out, the problems with Queue were the first sign that our lab computer was dying. We ended up getting an entirely new machine. I just tried Queue out on it, and it works perfectly (well, at least so far as giving the help message without complaint). So the problem does seem to have been on our end. Thanks for all the help!

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin

    Sorry to hear about your lab computer, @jphekman; may it rest in piece(s). But glad to hear that Queue is behaving properly on the new machine !

    Geraldine Van der Auwera, PhD

  • sorrywmsorrywm Posts: 9Member

    I am getting an ExceptionInInitializerError when I try to run Queue (v. 3.1.1), as follows: Exception in thread "main" java.lang.ExceptionInInitializerError at org.broadinstitute.sting.queue.engine.QGraph.(QGraph.scala:71) at org.broadinstitute.sting.queue.QCommandLine.(QCommandLine.scala:89) at org.broadinstitute.sting.queue.QCommandLine$.main(QCommandLine.scala:50) at org.broadinstitute.sting.queue.QCommandLine.main(QCommandLine.scala) Caused by: java.lang.NullPointerException at org.reflections.Reflections.scan(Reflections.java:220) at org.reflections.Reflections.scan(Reflections.java:166) at org.reflections.Reflections.(Reflections.java:94) at org.broadinstitute.sting.utils.classloader.PluginManager.(PluginManager.java:79)

    I am running java version 1.8, on a Mac. Please let me know if I can provide any other information for troubleshooting. Thank you!

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 6,462Administrator, GATK Developer admin

    @sorrywm‌ That might be a java version issue. We have not tested the software with java 1.8. I can only help you if this also happens when you run on java 1.7.

    Geraldine Van der Auwera, PhD

Sign In or Register to comment.