Charges for failed runs

Hi,
As we all are aware there have been some upgrade issues, patches, and fixes on FC over the past weeks.
I had several runs (of sets) that failed (several times over) and generated a whole bunch of unusable intermediate sep file and folders that basically ate away all my GATK-FC free credit and my google cloud credits. With all the overage it basically disconnected the workspace from both my fccredits-terbium account and my google cloud account. The only way I can now access it is through my saved bookmark.

The failure obviously happened between FC and Google cloud storage (of the broad institute), since it is not able to read the reference files (see below, BTW all references are from 5$ germline method).

Q1. Is there a way to recover these credits and complete my run to actually get usable data?
Q2. Should this happen again, how do we avoid/clean-up the large "intermediate step data" dump?
Q3. Is it possible to get my workspace to be reconnected to my fccredits-terbium account
Q4. Is it possible to have some kind of support when such things happen (chat would be great) ? It is quite terrifying to see all reds alters <!> all over the account and not be able to see the workspace all of a sudden.

Any help is much appreciated.

Call #18:
Operation:
Status:
Failed
Cache Result:Miss
Started:August 4, 2018, 1:46 AM (3 days ago)
Ended:August 4, 2018, 1:46 AM (3 days ago)
Inputs:Show
Outputs:None
stdout:stdout
stderr:stderr
JES log:BaseRecalibrator-17.log
Failures:Show

"message: Either ["gs://broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz","gs://broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz"] exists on a filesystem not supported by this instance of Cromwell, or a failure occurred while building an actionable path from it. Supported filesystems are: Google Cloud Storage. Failures: Google Cloud Storage: The specified GCS path '["gs://broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz","gs://broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz"]' does not parse as a URI. Illegal character in scheme name at index 0: %5B%22gs://broad-references/hg38/v0/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz%22,%22gs://broad-references/hg38/v0/Homo_sapiens_assembly38.known_indels.vcf.gz%22%5D Please refer to the documentation for more information on how to configure filesystems: http://cromwell.readthedocs.io/en/develop/backends/HPC/#filesystems"

Answers

  • Tiffany_at_BroadTiffany_at_Broad admin Cambridge, MAMember, Administrator, Broadie, Moderator admin

    Hi @shivish I am super sorry to hear about your negative experience with the platform. Let me speak with the vendor now and figure out a way to get these free credits back to this account. I'll respond to the remaining questions shortly, as well.

  • Hi Tiffany,

    Sorry I was not intending to be negative about the system. I still like it and it offers a great tool for non-programing novices like to analyse the data using proven "gold standard" protocols. It is just that I experimented with a smaller data set, made a lot of mistakes and finally when I felt a little comfort I ran the bigger data set and was not expecting this outcome. Thank so much for offering to talk to the vendor for me. Let me know if you need any additional information from my end.
    Regards
    Shiv

  • Tiffany_at_BroadTiffany_at_Broad admin Cambridge, MAMember, Administrator, Broadie, Moderator admin
    edited August 2018

    Hi @shivish your free credit billing project was terminated on August 2nd, 2018 due to the trial period ending, not because you went over the credit amount. See the Terms and Conditions to read more about the trial period.

    Q1: The quickest solution is to use another Google account you have access to, to start a new trial. Register that account with FireCloud and you will be issued a new billing project. From there you can work under your original FireCloud account by adding this account to the new billing project. Then clone the workspace and set the billing project equal to the new one, so that the costs accrue there. You can also work within your newly signed up Google account by sharing the workspace with it and following the cloning steps mentioned above.

    Q2: This sounds like a feature request that may already exist. I will take a look and point you to it if so.
    Q3: No, the billing project is terminated.
    Q4: We are considering in-app chat support in future iterations of the platform. We will be providing this service one day!

    Post edited by Tiffany_at_Broad on
  • Tiffany_at_BroadTiffany_at_Broad admin Cambridge, MAMember, Administrator, Broadie, Moderator admin
    edited August 2018

    Re: Q2: I took a quick peek and didn't see a feature request. If this is something you want to be automated on the FireCloud end, let us know there.

    You can go inside your Google bucket from the Summary tab and delete the intermediate files /folders that you don't want.

    Hope this helps!

  • Tiffany_at_BroadTiffany_at_Broad admin Cambridge, MAMember, Administrator, Broadie, Moderator admin

    Lastly, regarding the error you saw when running the $5 pipeline method, this part of the message Illegal character in scheme name at index 0 means that you may have hit a current bug that happens when copying and pasting gs://paths from the user interface. Hidden characters are getting copied over giving you this message. You can read more about this error here. Do you recall if this is what you were doing?

Sign In or Register to comment.