Latest Release: 12/4/18
Release Notes can be found here.

Significant slow down of Firecloud

lelaginalelagina Member, Broadie
edited July 8 in Ask the FireCloud Team

Hello FIrecloud team,

Since the latest release I see significant slowing down of Firecloud. It takes around 16-17 seconds to just load method configuration in the workspace and then another 20 seconds to launch analysis (after selecting sample).

I was wondering if anyone else experienced this. And if there is a reason for this.

Thank you for your help!

Answers

  • lelaginalelagina Member, Broadie
    edited July 9

    Just did another test:

    Changing snapshot ID (in method configuration in the workspace) takes 1 MINUTE!!!

  • mleventhalmleventhal Cambridge, MAMember, Broadie ✭✭
    edited July 9

    I have been experiencing similar slow-downs when launching analyses as well as filtering which entities/samples I want to run my analyses

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Thank you for this report; the level of slowness you're experiencing is certainly frustrating and not something we intend. I am investigating this with the team and will be back once I have more information.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    @lelagina and @mleventhal
    Would you be able to share a workspace where this issue is occurring using the Share button in FireCloud with [email protected].

    In this thread, I will need the name of the Workspace and the name of a method config that is particularly slow to load. Our developers would like to take a look to determine whether its a specific factor in your workspaces making it slow, or if it is a more general issue (i.e. any workspace would be slow). They can do this by exploring the an example of the issue in your workspace(s).

  • mleventhalmleventhal Cambridge, MAMember, Broadie ✭✭

    The workspace is ebert-fc/exac_maf_aggregation and it is already shared with [email protected]

  • mleventhalmleventhal Cambridge, MAMember, Broadie ✭✭

    My issues are more with handling the data model either from a method config or directly in the "data" tab

  • lelaginalelagina Member, Broadie

    Hello,

    I shared the broad-firecloud-wuclonal/neovax_pipeline_testing workspace with [email protected]

    For example changing NeoVax_Clinical_Pipeline method's snapshot takes about a minute and just trying to change method configurations is a bit slow.

    Thank you,
    Luda.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    Hi @lelagina and @mleventhal - Kate is out of the office, so I am following up. We noticed many folks reporting issues with lag over the last 24 hours too and have been investigating. I will see what the development team says about these instances and report back. Thanks.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin
    edited July 23

    @lelagina - We tried to reproduce the issue in dev with 400+ snapshots and saw no slowdown. Are you still experiencing this? If yes, could you share the method with mbemis @ broadinstitute.org?

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    @mleventhal we were not able to reproduce and suspect it was related to the slowdown we say again last week and put a fix out for Friday. Please let me know if you are still experiencing this behavior.

  • lelaginalelagina Member, Broadie

    Hello Tiffany,

    I am experiencing this extreme slow down with one particular method. This method has large number of inputs and outputs as well as large number of tasks run within this workflow.

    Thank you,
    Luda.

  • Tiffany_at_BroadTiffany_at_Broad Cambridge, MAMember, Broadie, Moderator admin

    @lelagina did you share the method, as requested above? @mbemis and I will take a look.

  • lelaginalelagina Member, Broadie

    Yes, I shared the NeoVax_Clinical_Pipeline method with [email protected]

    Let me know if you find the cause of this slow down. I tested and number of snapshots is not an issue here. The wdl script is more than 3000 lines. I suspect parsing this large wdl takes so long.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Hi Tiffany,

    I am using Luda's method (lelagina/CGA_Production_Analysis_Pipeline_WES, Snapshot ID: 2) and am seeing the same slow response time. It takes over 35 seconds to load a method config that links to this pipeline. This is in a workspace with no data, so I'm pretty sure it is a function of the method's WDL, and not the contents of the workspace data model. I added a second method config to the test workspace that references a very simple WDL, and the method config comes up in a couple of seconds. When FireCloud brings up (i.e., "loads") a method config in a workspace, does it parse the method WDL? Could this be the source of the slowdown? If so, is there a way of determining why it takes FireCloud so long to parse the CGA_Production_Analysis_Pipeline_WES wdl?

    -Chet

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Hello @lelagina and @birger. Our developers have discovered an issue that caused slowness & freezing (a lot like you are describing) when we run workshops to teach folks to use FireCloud. A fix has been implemented, tested, and will be released tomorrow. This fix should at least partially address the slowness you are seeing when loading a method configuration, and speaking from experience with some manual testing I was doing today, the load times of the method configuration section have been vastly improved.

    Some work remains for improving and optimizing the backend database for FireCloud, and there is a ticket in the developer's queue for addressing that. I will check back in after tomorrow's release to see if your particular observations of slowness have been improved.

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    @lelagina and @birger Further investigation identified that the particular slowness you see with your workflow is also due to certain inefficiencies on how the WDL was being parsed. We have a release coming out today, and we've tested our changes with your particular workflow to make sure that the inefficiencies are addressed. If you could check on loading a Method Config and/or try submitting your workflow anytime after tomorrow -- it would be a good validation to ensure that the release has addressed your concerns.

    Thanks!

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    Will do.
    -Chet

Sign In or Register to comment.