Billing Alert - 100% of Budget Exceeded

ChipChip 415M 4053Member, Broadie


Today I'm getting an email from [email protected] warning that billing project <not sure billing id's are ok to list on the forum> had reached it's monthly limit of $5K. I've never seen an email like this until today. This is the rebc-oct16 project billing account.

The only jobs I see that were run using this project in June were three failed jobs that ran for 2 minutes with a "Total Run Cost" estimate that says <$0.01 each. I should have run some more but I've been a bit distracted.

From the Google Cloud Platform page for rebc-oct16 billing page there is a "view detailed charges" button to get to "Charges this month" which makes a list of VMs and storage expenses:

Product Resource Usage Amount
Compute Engine Highcpu Intel N1 32 VCPU running in Americas 2,948.27 Hour $3,344.52
Cloud Storage Multi-Regional Storage US 30,539.89 Gibibyte-month $794.04
Compute Engine Standard Intel N1 32 VCPU running in Americas 262.92 Hour $399.64
Compute Engine Storage PD Capacity 4,550.91 Gibibyte-month $181.80
Compute Engine Highcpu Intel N1 4 VCPU running in Americas 198.81 Hour $28.19
Compute Engine Highmem Intel N1 4 VCPU running in Americas 2,167.25 Minute $8.55

The top job(s?) appears to have been running for almost 3000 hours (more than 4 months) on a beefy VM(s), adding up to >$3k. I don't see any such job from the Firecloud monitor tab in any of the rebc-oct16 workspaces and I was under the impression that Google would pull the plug on any job after a week of running.

From the VM instances page there are three running VMs:

I don't know what these VMs are doing since there are no jobs running in FC, but they all indicate they were started either yesterday or today so they are probably not the huge $3.344K eating VM.

Is this a real problem that should be tracked down - or is this one of those things we should ignore? If this should be tracked down, how should someone like me track this down?



  • ChipChip 415M 4053Member, Broadie

    Update: A hard-working fast-spending collaborator has been submitting lots of compute-intense jobs this week and that explains the billing warning. Doesn't explain how I missed these jobs when I spent an hour yesterday hunting for them, but whatever it was - it probably my fault.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    I'm glad to hear you were able to find the source of the charges. If you encounter billing issues like this again, you are welcome to post here and we can certainly follow up off the forum to maintain privacy of billing account numbers.

Sign In or Register to comment.