GATK licensing moves to direct-through-Broad model -- read about it on the GATK blog

Queue with IntelliJ IDEA

Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin
edited February 2014 in Pipelining with Queue

We have found it that Queue works best with IntelliJ IDEA Community Edition (free) or Ultimate Edition installed with the Scala Plugin enabled. Once you have downloaded IntelliJ IDEA, follow the instructions below to setup a Sting project with Queue and the Scala Plugin.

[[File:sting_project_libraries.png|300px|thumb|right|Project Libraries]]
[[File:sting_module_sources.png|300px|thumb|right|Module Sources]]
[[File:sting_module_dependencies.png|300px|thumb|right|Module Dependencies]]
[[File:sting_module_scala_facet.png|300px|thumb|right|Scala Facet]]

1. Build Queue on the Command Line

Build Queue from source from the command line with ant queue, so that:
- The lib folder is initialized including the scala jars
- The queue-extensions for the GATK are generated to the build folder

2. Add the scala plugin

  • In IntelliJ, open the menu File > Settings
  • Under the IDE Settings in the left navigation list select Plugins
  • Click on the Available tab under plugins
  • Scroll down in the list of available plugins and install the scala plugin
  • If asked to retrieve dependencies, click No. The correct scala libraries and compiler are already available in the lib folder from when you built Queue from the command line
  • Restart IntelliJ to load the scala plugin

3. Creating a new Sting Project including Queue

  • Select the menu File... > New Project...

  • On the first page of "New Project" select
    Create project from scratch
    Click Next >

  • On the second page of "New Project" select
    Set the project Name: to Sting
    Set the Project files location: to the directory where you checked out the Sting git repository, for example /Users/jamie/src/Sting
    Uncheck Create Module
    Click Finish

  • The "Project Structure" window should open. If not open it via the menu File > Project Structure

  • Under the Project Settings in the left panel of "Project Structure" select Project
    Make sure that Project SDK is set to a build of 1.6
    If the Project SDK only lists <No SDK> add a New > JSDK pointing to /System/Library/Frameworks/JavaVM.framework/Versions/1.6

  • Under the Project Settings in the left panel of "Project Structure" select Libraries
    Click the plus (+) to create a new Project Library
    Set the Name: to Sting/lib
    Select Attach Jar Directories
    Select the path to lib folder under your SVN checkout

  • Under the Project Settings in the left panel of "Project Structure" select Modules

  • Click on the + box to add a new module

  • On the first page of "Add Module" select
    Create module from scratch
    Click Next \>

  • On the second page of "Add Module" select
    Set the module Name: to Sting
    Change the Content root to: <directory where you checked out the Sting SVN repository>
    Click Next \>

  • On the third page
    Uncheck all of the other source directories only leaving the java/src directory checked
    Click Next \>

  • On fourth page click Finish

  • Back in the Project Structure window, under the Module 'Sting', on the Sources tab make sure the following folders are selected

    • Source Folders (in blue):
      public/java/src
      public/scala/src
      private/java/src (Broad only)
      private/scala/src (Broad only)
      build/queue-extensions/src
    • Test Source Folders (in green):
      public/java/test
      public/scala/test
      private/java/test (Broad only)
      private/scala/test (Broad only)
  • In the Project Structure window, under the Module 'Sting', on the Module Dependencies tab select
    Click on the button Add...
    Select the popup menu Library...
    Select the Sting/lib library
    Click Add selected

  • Refresh the Project Structure window so that it becomes aware of the Scala library in Sting/lib
    Click the OK button
    Reopen Project Structure via the menu File > Project Structure

  • In the second panel, click on the Sting module
    Click on the plus (+) button above the second panel module
    In the popup menu under Facet select Scala
    On the right under Facet 'Scala' set the Compiler library: to Sting/lib
    Click OK

4. Enable annotation processing

  • Open the menu File > Settings
  • Under Project Settings [Sting] in the left navigation list select Compiler then Annotation Processors
  • Click to enable the checkbox Enable annotation processing
  • Leave the radio button obtain processors from the classpath selected
  • Click OK

5. Debugging Queue

Adding a Remote Configuration

[[File:queue_debug.png|300px|thumb|right|Queue Remote Debug]]

  • In IntelliJ 10 open the menu Run > Edit Configurations.

  • Click the gold [+] button at the upper left to open the Add New Configuration popup menu.

  • Select Remote from the popup menu.

  • With the new configuration selected on the left, change the configuration name from 'Unnamed' to something like 'Queue Remote Debug'.

  • Set the Host to the hostname of your server, and the Port to an unused port. You can try the default port of 5005.

  • From the Use the following command line arguments for running remote JVM, copy the argument string.

  • On the server, paste / modify your command line to run with the previously copied text, for example java -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 Queue.jar -S myscript.scala ....

  • If you would like the program to wait for you to attach the debugger before running, change suspend=n to suspend=y.

  • Back in IntelliJ, click OK to save your changes.

Running with the Remote Configuration

  • Ensure Queue Remote Debug is selected via the configuration drop down or Run > Edit Configurations.
  • Set your breakpoints as you normally would in IntelliJ.
  • Start your program by running the full java path (with the above -Xdebug -Xrunjdwp ...) on the server.
  • In IntelliJ go to the Run > Debug.

6. Binding javadocs and source

From Stack overflow:

Add javadocs:

Point IntelliJ to http://download.oracle.com/javase/6/docs/api/.
Go to File -> Project Structure -> SDKs -> Apple 1.x -> DocumentationPaths, and the click specify URL.

Add sources:

In IntelliJ, open File -> Project Structure.
Click on "SDKs" under "Platform Settings".
Add the following path under the Sourcepath tab:
/Library/Java/JavaVirtualMachines/1.6.0_29-b11-402.jdk/Contents/Home/src.jar!/src

Post edited by Geraldine_VdAuwera on

Geraldine Van der Auwera, PhD

Comments

  • TimHughesTimHughes Posts: 61Member

    I use Eclipse and see that it has a scala plugin. So I was wondering about when you say "We have found it that Queue works best with IntelliJ IDEA Community Edition (free) or Ultimate Edition installed with the Scala Plugin enabled." >> does this mean that you tried different ways of doing things and it was tricky but you got it working with IntelliJ? It would be nice to know so that I can evaluate whether it makes sense to try getting it working on eclipse or whether I should just use intelliJ for qscripts.

  • Johan_DahlbergJohan_Dahlberg Posts: 89Member ✭✭✭

    Just a comment about running Eclipse as IDE - which is what I'm doing. I think that eclipse works fine for developing java and scala for Queue. I've not tried IntelliJ, mostly because I've been so comfortable with Eclipse.

    However, in addition to installing the Scala plugin, make sure that you have Apache IvyDE installed for the dependency management, otherwise it won't build. As a side note I normally just use Eclipse as a glorified editor, and then build with ant on the command line, since I find that easier and more flexible.

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    The GSA team mostly works with IntelliJ, but it may be more a matter of personal preference. Eclipse does work fine, given a few tweaks (nothing major). I would recommend picking a single IDE to work on both java and scala, otherwise it becomes a pain to switch.

    FYI, Typesafe.com has a Scala-ready version of Eclipse available for download here:
    http://typesafe.com/stack/scala_ide_download

    And Johan's point is very good: you do need Ivy set up for dependency management.

    Geraldine Van der Auwera, PhD

  • dklevebringdklevebring Posts: 76Member

    I'm trying to setup IntelliJ IDEA CE to write scala scripts for Queue. However, it cant seem to find import org.broadinstitute.sting.queue.function.CommandLineFunction after importing it. Before importing it, it suggests to do so, and adds it to this list of imports. It immediately becomes gray ("unused import") and CommandLineFunction remains red (not found).

    Screenshot 2013-11-02 20.40.44.png
    535 x 260 - 29K
  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Hmm, sounds like an issue with your IntelliJ project setup. Make sure you have the sources/modules settings set up to include the scala sources, maybe?

    Geraldine Van der Auwera, PhD

  • CarlosBorrotoCarlosBorroto Posts: 38Member

    Hi,

    Would it by possible to update this document to the new structure in 3.0?

    Thanks,
    Carlos

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Hi Carlos,

    Yes, it's on my to-do for next week when I get back from vacation.

    Geraldine Van der Auwera, PhD

  • CarlosBorrotoCarlosBorroto Posts: 38Member

    Any updates? I saw you have an updated guide for setting GATK 3+ and Maven, but no update on how to setup the environment for easy Queue/Scala development :-(.

    Best,
    Carlos

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Ack, sorry Carlos, I forgot! I'll ask @kshakir what needs to be updated here.

    Geraldine Van der Auwera, PhD

  • CarlosBorrotoCarlosBorroto Posts: 38Member

    Hi @Geraldine_VdAuwera‌ , this would be great. Using the updated information I got gatk to build and imported into IntelliJ 13.1.3. My main problem is I don't know how to setup this project so I can create a module to write a Queue script in Scala. Before I was able to get autocompletion for class attributes and methods. Quite convenient.

    See how IntelliJ doesn't recognize my imports of even scala syntax.

    Thanks,
    Carlos

    Screen Shot 2014-06-10 at 10.09.09 AM.png
    2526 x 1412 - 408K
  • pdexheimerpdexheimer Posts: 454Member, GSA Collaborator ✭✭✭✭

    @CarlosBorroto‌ -

    I think I encountered this same issue when I set up an IntelliJ project. There was a directory somewhere in the tree that needs to be set as a "source directory" in IntelliJ - I think it was the auto-generated Queue extensions for GATK. Also, make sure you run "mvn test-compile" once from the command line so that the extensions are generated

  • CarlosBorrotoCarlosBorroto Posts: 38Member

    Hi @pdexheimer‌ , @Geraldine_VdAuwera‌ ,

    I got it working!.

    @pdexheimer‌ I couldn't add the "source directories" I found could be the ones to my module(Scala) directly. Every source directory I tried was already part of a different module already in the project and IntelliJ wouldn't let me add them. What I was able to do instead was add "Dependencies" to my modules.

    For the particular Queue script I'm writing it was enough to add this list:

    • scala-library (installed using homebrew in '/usr/local/Cellar/scala/2.11.1/libexec')
    • queue-framework (gatk project module)
    • gatk-framework (gatk project module)
    • gatk-protected (gatk project module)
    • net.sf.sam (already added by Maven to the local repository cache)

    Thanks for the pointers!,
    Carlos

    Screen Shot 2014-06-11 at 12.52.18 PM.png
    1060 x 855 - 145K
  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Glad to hear that, Carlos! Thanks for reporting your solution. I'll try to get the Queue docs updated sometime this century, though I have to confess it's pretty low on the priority list right now.

    Geraldine Van der Auwera, PhD

  • yoskeriyoskeri tokyo,japanPosts: 6Member

    I'm sorry for asking a very silly question.
    When I try to do " ant queue" in my Sting directory,i got the error message," Buildfile: build.xml does not exist!"
    Please teach me how to fix that problem.

    Yoskeri

  • pdexheimerpdexheimer Posts: 454Member, GSA Collaborator ✭✭✭✭

    @yoskeri‌ - This document is a bit dated. The GATK switched from ant to maven a few versions ago, the directions for setting up GATK in IntelliJ should also get you going with Queue

  • yoskeriyoskeri tokyo,japanPosts: 6Member

    @pdexheimer‌ - Thank you for your guidance. I'm totally new to maven, but i try.

  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Sorry about the outdated docs -- we'll get around to updating them eventually, it's just been hard to make this a priority.

    Geraldine Van der Auwera, PhD

  • jamjam Posts: 12Member
    edited January 5

    I need to write a jobRunner for Queue on UGE that doesn't rely on sync ( considered harmful by our cluster admins ), and think I've learned enough scala to do this but have a few issues with getting started. the page on setting up IntelliJ with maven was very helpful but when it comes to Queue I still manage to get lost.

    Should note, i'm vi born and bred, haven't used an IDE since turbo c++, am easily confused by all the graphical wonderment.

    where should I be adding my files to /src/main/scala/org/broadinstitute/gatk/queue/engine/gridengine ?

    I couldn't get methods recognized for imported queue thingies ( modules? ) though I could get most of IntelliJ working and the rest of GATK recognized following your nice maven instructions. If it helps this is the start of a script where command completion/definition part falls apart

    import org.broadinstitute.sting.queue.QScript
    import org.broadinstitute.sting.queue.extensions.gatk._
    
    class HaplotypeCaller extends QScript {
            qscript =>
            HaplotypeCaller  <-- highlights as unrecognized.
    

    Couldn't make much sense of carlos borroto's instructions, probably a combination of the move to intellij 13 and my lack of IDE savvy.

    Post edited by Geraldine_VdAuwera on
  • Geraldine_VdAuweraGeraldine_VdAuwera Posts: 7,565Administrator, GATK Developer admin

    Have a look at the presentations on Queue from last year's workshop: https://www.broadinstitute.org/gatk/guide/presentations?id=3391

    See bottom of the page -- the Queue 101 presentation will give you the basic syntax you need to use in your script.

    Geraldine Van der Auwera, PhD

Sign In or Register to comment.