Quota on GPUs per billing account leads to job failures rather than queueing
Hi Firecloud team,
I am running a gpu task on ~1300 samples and have started running into this error: The job was stopped before the command finished. PAPI error code 2. failed to insert instance: googleapi: Error 403: Quota 'NVIDIA_K80_GPUS' exceeded. Limit: 64.0 in region us-central1., quotaExceeded
1) Can I get a higher limit on my billing project?
2) I believe this is a bug. Firecloud should wait for resources to become available rather than failing.
My current work around for this is to use Dalmatian to monitor the number of active jobs and submit a new job when the number of running jobs is below 64 which is basically doing the load management.