Jobs marked as 'Running' but not executing

For example:
Submission ID: 7973aa29-1d26-4868-aa68-8e20ba50956e
Workflow ID: 5827b887-fbf1-4029-89dd-1013ef008e71

Answers

  • francois_afrancois_a Member, Broadie ✭✭

    The jobs were waiting for quota, but I should have had plenty of quota to run these. Is there another reason why jobs might get stuck in this state? It would be helpful if the status reflected this, instead of marking the jobs as running.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Administrator, Broadie, Moderator admin

    Hi @francois_a , it appears you have hit your quota on persistent disk. Quotas are set by Google at the project level. If other workspaces aligned to the project have hit the cap, it will affect yours.

    Are you asking why jobs "get stuck" waiting for quota?

    In addition, you mention an improvement- "It would be helpful if the status reflected this, instead of marking the jobs as running." Do you want the call/task status to show a separate status for when it is waiting for quota? Currently, there are four statuses of a task: not started, starting, running, and done.

  • dvoetdvoet Member ✭✭

    Francois,

    The quota in billing project nci-francois-bi-org is 204,800 GB of persistent disk. There are in the neighborhood of 1200 VMs running in the project each with either a 150GB or 250GB (mostly 150) disk plus a 10GB boot disk.

    If you are an owner of this project in FireCloud, you should be able to view your quotas
    https://console.cloud.google.com/iam-admin/quotas?project=nci-francois-bi-org&organizationId=386193000800

    -Doug

  • francois_afrancois_a Member, Broadie ✭✭

    @Tiffany_at_Broad, @dvoet,
    Thanks both -- I hadn't realized that the quotas on this billing account/project were different from those on the account I typically use. Sorry for missing this.

    As for having a separate status explicitly mentioning 'waiting for quota', that'd be very useful.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Simply because this wasn't specifically mentioned on this thread, I wanted to follow up. Tiffany did file a ticket to add a "waiting for quota" status to FireCloud, and we will let you know when it is implemented. Thank you for your suggestion!

Sign In or Register to comment.