Heads up:
We’re moving the GATK website, docs and forum to a new platform. Read the full story and breakdown of key changes on this blog.

Cleaning Failed Runs / Temporary Data

jasongallant1jasongallant1 East Lansing, MIMember

Hi All-

Curious if there is a straightforward way to trim away failed runs and temporary files in FireCloud once we are satisfied with the output on a method run level? The storage costs proliferate pretty quickly for a lot of the "junk".... Obviously this is possible using the google bucket, but I'm wondering if it would be more straightforward using the firecloud UI?

Thanks!

Tagged:

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    This isn't currently available via the UI, unfortunately. Our field engineering team wrote a script a while back that you can find here. However, I warn you that this will delete whole workflow directories not linked to the data model.

    If you have specific parts you'd like to delete, I'd recommend emailing [email protected] with that info. This will link you to our Field Engineering team, and they can assist you with deleting bits and pieces.

    I will put in an additional note on the feature request to put this in the UI that you are interested in seeing it completed as well. It always helps us to know what you want to see as we improve the product.

  • jasongallant1jasongallant1 East Lansing, MIMember

    Hi @KateN ,

    Thanks for this tip. Trying to run this in FC now, however it requires a svcActKeyJson? I tried to create one, however I'm working with a project that is associated with trial credits, and I don't have permission to create one? Any insights into how to do this, or instructions to follow?

    Thanks,
    Jason

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Alright, I have consulted with the author of the script, and he says that you can use the following code to get your key:

    curl -H "Authorization:Bearer $(gcloud auth print-access-token)" -X GET  'https://sam.dsde-prod.broadinstitute.org/api/google/user/petServiceAccount/<the project where this workspace is they want to run the WDL within>/key' > svcAcctKey.json
    

    You should then place your key in a bucket only you have access to (which could be your workspace's bucket if your workspace is private to you), and your key will last 48-60 days before expiring. At that point you will just need to get a new key.

Sign In or Register to comment.