The front line support team will be unavailable to answer questions until May 27th 2019 as we are celebrating Memorial Day. We will be back soon after. Thank you for your patience and we apologize for any inconvenience!
Release Notes can be found here.
Quota on GPUs per billing account leads to job failures rather than queueing
Hi Firecloud team,
I am running a gpu task on ~1300 samples and have started running into this error: The job was stopped before the command finished. PAPI error code 2. failed to insert instance: googleapi: Error 403: Quota 'NVIDIA_K80_GPUS' exceeded. Limit: 64.0 in region us-central1., quotaExceeded
1) Can I get a higher limit on my billing project?
2) I believe this is a bug. Firecloud should wait for resources to become available rather than failing.
My current work around for this is to use Dalmatian to monitor the number of active jobs and submit a new job when the number of running jobs is below 64 which is basically doing the load management.