Latest Release: 03/12/19
Release Notes can be found here.

Workflow which takes ~11 mins on laptop took ~11 hours on FireCloud

mnoblemnoble Broad Institute of MIT & HarvardMember, Broadie

We think this is an apples to apples comparison. I've attached the workflow log from both local and Firecloud execution, as well as the WDL (which has 4 tasks).

The firecloud workflow execution ID was 2f2d0fd6-4935-4bb3-bfe0-eb0f19774920, invoked from the broad-firecloud-gdac/dev workspace (via the monitor URI https://portal.firecloud.org/#workspaces/broad-firecloud-gdac:dev/Monitor/f8022611-fd8f-4b79-a4df-23fed28ad363)

Thanks,
Mike

Answers

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    We should take a look at the actual processing time as reported by JES. We know that Cromwell has issues recognizing tasks are completed in a timely fashion. @mnoble , when you say it takes 11 minutes on a laptop, is that running cromwell locally on your laptop, or running a non-WDL version of the pipeline?

  • abaumannabaumann Broad DSDEMember, Broadie ✭✭✭

    We will be upgrading Cromwell soon (waiting for the v 25 release to be cut), and that version should resolve this issue. Not sure if this is what you did, but correct apples to apples comparison would be to run Cromwell in server mode locally against a JES backend. That would be equivalent to what we do. When this is working properly there should be a small reasonable delay in Cromwell updating status after JES updates its status, nothing on the order of hours as you show.

  • mnoblemnoble Broad Institute of MIT & HarvardMember, Broadie

    Thanks for the updates, Alex. Well, to your first question: yes, we were running Cromwell locally on the laptop (the attached local_execution.log does show this), but AFAIK JES backend was not run--i.e. the compute was also local on laptop.

Sign In or Register to comment.