Plans for Queue

konuvakonuva NetherlandsMember

Dear GATK Team,

I am wondering about future plans for the Queue framework. I find it a useful framework to write and run pipelines in computing clusters. However, I found myself often wanting to use Queue in pipelines without any GATK walkers at all. Are there any plans in the future to release Queue as its own, GATK-independent package?

I know that internally there are some shared classes (e.g. the command line parser), and refactoring them so that Queue can be GATK-free may require a little more work. But I'm just interested to know if there are already plans to do this (or perhaps even already ongoing).

Cheers,
konuva

Tagged:

Best Answer

Answers

  • konuvakonuva NetherlandsMember

    Hi Sheila,

    That is encouraging to hear :). Queue seems to have a lot of potential and releasing it as a library separate from GATK looks like the right thing to do.

    I'll be waiting for the announcement soon :).

  • pjvthofpjvthof NLMember

    Is there maybe any update on this already?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Work on this is ongoing; unfortunately we can't give you an ETA at this time. In the meantime, check out our public github repo for the core GATK+Queue framework, which contains the core GATK engine and all the Queue code, without the GATK analysis tools. You can use that to use Queue without too much GATK-specific baggage. It's open-sourced under the MIT license.

  • pjvthofpjvthof NLMember

    I understand. Already using this public repo to generate extensive pipelines, so from fastq->clipping->trimming-> mapping -> sorting -> dedup -> gatk best practises (maybe more in the future). Having Queue as a puplic maven library would really help a lot, right now I use Queue as a local maven library.

    Queue will stay at MIT license? We want to use it for more pipelines but we want to be sure we also can use Queue for this in the future.

    For now Queue already works great, so please keep us up to date :).

  • Johan_DahlbergJohan_Dahlberg Member ✭✭✭

    I just want to second @pjvanthof‌, having both Queue (and GATK) in separate maven repos would be awesome!

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    It will happen, and yes Queue will stay MIT-licensed for-eee-veeeeeeer... :)

  • LeonLeon Leiden University Medical CenterMember

    Hi Geraldine,

    If possible, I like to know some update on the plan of moving Queue into a separate maven repo. At LUMC, we have been building our pipeline framework based on Queue and adding additional Queue modules wrapping all common NGS tools. It has been working very well and we are really satisfied with the Queue support on our HPC. Having Queue more clearly separate from the rest applications of GATK will help a lot with regard to sustaining our pipeline framework in future.

    If the GATK team and others are interested, we might even think about a kind of hackathon for people interested in working together around the core of Queue.

    Thanks,
    Leon

    Issue · Github
    by Geraldine_VdAuwera

    Issue Number
    960
    State
    open
    Last Updated
    Milestone
    Array
  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hi @Leon,

    Last I heard we had a couple of blockers that were preventing us from finalizing the move to full mavenization. I'll check with the devs what is the status on that -- I think it might have been put on hold due to other priorities.

    A Queue hackathon would be awesome :)

    Issue · Github
    by Geraldine_VdAuwera

    Issue Number
    958
    State
    open
    Last Updated
    Milestone
    Array
  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Update on the mavenization plan: the last obstacles that stand in our way are some third party dependencies that need to be cleaned out before we can push the GATK and Queue artifacts to Central. These are not very difficult tasks but we haven't been able to devote resources to them yet, partly due to lack of awareness that these are outstanding tasks. I'm now putting in some issue tickets to give these tasks visibility for the engineering team, so hopefully we can push this through sooner rather than later.

    Issue · Github
    by Geraldine_VdAuwera

    Issue Number
    959
    State
    open
    Last Updated
    Assignee
    Array
    Milestone
    Array
  • LeonLeon Leiden University Medical CenterMember

    Hi Geraldine,

    Thanks for the follow-ups! I will speak to my colleagues when some of them are back from holiday on what we might contribute.

    Cheers,
    Leon

  • LeonLeon Leiden University Medical CenterMember

    Hi Geraldine,

    Sorry to push on this again. Is there any update on separating Queue?

    For you or some other people who might be interested, we now pushed a copy of our work based on Queue to https://github.com/biopet/biopet. You can see having Queue jar in a Maven repo will greatly simplify our dependency setting.

    Thanks!

    Leon

    Issue · Github
    by Sheila

    Issue Number
    166
    State
    open
    Last Updated
    Assignee
    Array
    Milestone
    Array
  • SheilaSheila Broad InstituteMember, Broadie, Moderator admin

    @Leon
    Hi Leon,

    I hate to tell you this, but the developers are not focusing on this right now. They are certain it will be useful, but they just cannot devote the time to this right now.

    Sorry,
    Sheila

Sign In or Register to comment.