We've moved!
You can find our new documentation site and support forum for posting questions here.

Task takes 2 hours in FireCloud, seconds on my laptop

In my workflow https://portal.firecloud.org/#workspaces/jianshu-projects/181217_JB_scATAC_v1/monitor/2fd7591f-964f-4b06-a504-41de4122156d/62acb6bd-298e-4972-852e-d2998af594d6, I scatter running bowtie many times with the same reference and different fastq files (e.g. operations/EKWPisr9LBjW0d3G2LT_q-kBIKGQqLm8ECoPcHJvZHVjdGlvblF1ZXVl). On my laptop, once the files are localized, the task takes seconds, however it takes ~2 hours in FireCloud. Do you know why? Thanks.

Answers

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @jgould I can double-check to make sure but the extra time on FireCloud is probably due to time spent waiting for quotas in different parts of the process. Since many users are running workflows at the same time, our systems have limits on how many tasks can be pushed to our execution engine, Cromwell, and how many tasks can be pushed to Google Pipelines API etc. Sometimes the load increases if a higher than usual number of users submit workflows at the same time, further adding to the time.

  • jgouldjgould GouldMember ✭✭

    Thanks. Would you also be able to help me understand the workflow cost so that we can optimize the pipeline? Thanks.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    Hi @jgould - Payment is based on Google services used such as compute, storage and egress - FireCloud, itself, will not cost you anything! Since the calculation of the three google services is not based on a single calculation, we have written up a document that goes into each of the three services and how you would be able to get a general idea of what your cost might be for your pipeline. Please let me know if this is helpful to you!

  • jgouldjgould GouldMember ✭✭

    Thanks for your reply. I'm aware of how cost is computed, but am unable to figure out the cost details for my run. For example, how much did it cost per task? Was the bowtie2 task that takes seconds to run on my laptop, but took 2 hours to run on FireCloud billed for the full 2 hours or does the 2 hour running time include time where a VM is not provisioned? Thanks.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @jgould Apologies, I misunderstood your question! This is the correct document - it contains step-by-step instructions on how to calculate cost using BigQuery. There is currently no FireCloud UI method to retrieve the details but the document should aid you in getting the data.

  • jgouldjgould GouldMember ✭✭

    Thanks for your help-I have my billing data in BigQuery. Is there a field that shows the FireCloud operation? I have hundreds of the same task with seemingly similar sized inputs. The cost for this operation varies several thousand fold and I want to identify the inputs used for the most expensive run. Thanks.

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @jgould Sorry for the delay - I am looking into this and will get back to you!

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    @jgould Since a team member has reached out to work with you directly regarding this issue, we are going to close this ticket but should you require any further help please feel free to reach out!

Sign In or Register to comment.