Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

Job aborting for a month

cgdiascgdias Member
in Ask the FireCloud Team

Hi,
I have a job that has been aborting for a month. I don't know how to handle this bug. I tried to delete the associated bucket, but nothing happened. Any tips?
Also, I won't be billed one month of computational analysis because of that, right? :smile:

Thanks!

· Share on Twitter Share on Google+

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    In most cases, it's just an error with the UI not updating properly. Could you share your workspace in FireCloud with [email protected]? I will also need the name of your workspace, as well as the submission ID for the job in question.

    · Share on Twitter Share on Google+
  • cgdiascgdias Member

    I just sent an email to the address you mention and now waiting for a reply. Thanks for your help Kate :)

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @cgdias would you mind sharing the name of your workspace? I'll look into the details.

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @cgdias I found your email with the details, sorry about the delayed response.

    There are 2 workflows inside the submission 1c9460a0-d3e7-45c1-8fbc-de973bcd0734 that have aRunning status. Looking further, it seems like the jobs inside those workflows both finished on May 17th:
    WF a0844776-45b6-4fd3-b41e-40484570737e -- Job finished at 2018-05-17T14:02:10Z
    WF e05e8cee-a94a-4da3-b152-ced43ffd6895 -- Job finished at 2018-05-17T14:02:05Z

    It seems like there was a system restart around the time that the jobs finished, and the workflow status update was lost, even though the jobs finished running and the workflow completed.

    For the future -- deleting the bucket doesn't affect jobs from starting/running. However, if you're trying to confirm if workflow is actually still running/costing you money, then its possible to check on the google cloud console if you're project owner.

    1. Go to this link (after adding in your project name): https://console.cloud.google.com/compute/instances?project=
    2. In the search bar, filter for instances labelled with the workflow uuid you're investigating. The syntax for the filter looks like Labels:cromwell-workflow-id:cromwell-<workflow-uuid>

    Hope this helps, let me know if you have further questions.

    · Share on Twitter Share on Google+
  • cgdiascgdias Member

    Hi,
    Thank you for your help. It's worse now, the workspace shows that a task is running (on the Firecloud homepage), but when I browse the 'monitor' section of the workspace, nothing is showing up as 'running'. I thus can't retrieve Labels:cromwell-workflow-id:cromwell- :neutral:
    Can you help me with that? I just sent you the workspace name by private message.
    Thank you again @Ruchi !

    · Share on Twitter Share on Google+
  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    This conversation has been taken off the forum for followup. @Ruchi, would you mind posting a summary when a solution is reached? Thank you.

    · Share on Twitter Share on Google+
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @KateN, I've not heard back for about 10 days, but I just want to share again the fact that no jobs last in Google for more than 7 days. It's quite possible the FireCloud shows a task as "Running" for an extended period -- but Google wont' allow for it to run for a month, so the job should actually be dead. I've encouraged @cgdias to share a screenshot of the instructions I sent as a confirmation to list any running jobs.

    Let me know if there's more I can do!

    · Share on Twitter Share on Google+
Sign In or Register to comment.