Latest Release: 05/01/19
Release Notes can be found here.

Failed Analysis on Broad_MutationCalling_Mutect_Workflow_BestPractice_OA

Hello,

First, I was able to successfully run 2 submissions within the workspace MiniMutationCalling_V1_Tutorial. One on each of the HCC pairs and they both succeeded.

As a next step, I cloned the workspace Broad_MutationCalling_Mutect_Workflow_BestPractice_OA. It already had the method
MutationCalling_Mutect_v1-2_BETA_cfg built-in, as well as the 2 HCC pairs under the data tab. I selected a single pair, HCC1954_100_gene_pair
and launched the analysis, But it failed rather quickly (after 5 min).
Trying to debug what the problem is, since I didn't change anything in the workspace. I just ran it as is.
In the Monitor page, the status of the submission says Failed, but the message field is empty.

I also went to fetch the workflow.log, but I don't see any error message there either.
I'm attaching screenshots and the workflow.log, and would appreciate guidance on how to debug further.

Thanks
Severine

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Hi @severinec. Thank you for your forum post.

    Regarding MutationCalling_Mutect_v1-2_BETA, unfortunately that WDL used to work, but then upon update to cromwell24, it stopped working. I have made a new WDL MutationCalling_Mutect_v1-3_BETA to address the problem. In the workspace Broad_MutationCalling_Mutect_Workflow_BestPractice_OA where you cloned from, I replaced the old version (1-2) with the new version (1-3). If you go into that workspace you should find the update. You could go there and re-clone. Alternatively, I've also published a config MutationCalling_Mutect_v1-3_BETA_cfg which you can import into your existing workspace and that should work as well.

  • Thank you for your answer. I will give this a try and let you know.

    Severine

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Great @severinec . Let me know whether you encounter success or failure?

  • Hello

    here is a follow-up:

    I tried both suggestions:

    1) Import MutationCalling_Mutect_v1-3_BETA_cfg from my exisiting workspace. When I selected the pairID for the input, it asked me to select by participant ID instead of pair_ID. I did, and then could launch the run, but it failed shortly after with the following error:

    CallingGroup_Workflow.oncoDBTarBall - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.mutectForcecallIntervals - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.refFastaIdx - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.dbSNPVCF - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.tumorBam - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.mutectIntervals - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.fracContam - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.refFasta - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.normalBam - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.readGroupBlackList - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.oncoVEPDiskGB - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.preemptible - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.oncoMemoryGB - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.VEPMemoryGB - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.cosmicVCF - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.pairName - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.downsampleToCoverage - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.tumorBamIdx - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.refFastaDict - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.mutectDiskGB - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.VEP_File - Failed at line 1, column 1: workspace.' expected but end of source found CallingGroup_Workflow.normalBamIdx - Failed at line 1, column 1: workspace.' expected but end of source found
    CallingGroup_Workflow.normalPanel - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.nWay - Failed at line 1, co

    2) Clone the updated Broad_MutationCalling_Mutect_Workflow_BestPractice_OA into a new workspace.
    This time, I could select a pair ID as input in this case, and it is still running. I launched it 19 hours ago. Is this expected??

    Thanks
    Severine

  • re: 2) how can I know whether the pipeline is actually still running, or hanging?
    Is there a progress bar that I can consult?

    I tried to go see the workflow.log, but I saw an error there as well. attached screenshot.

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Hi @severinec

    1) Import MutationCalling_Mutect_v1-3_BETA_cfg from my exisiting workspace. When I selected the pairID for the input, it asked me to select by participant ID instead of pair_ID. I did, and then could launch the run, but it failed shortly after with the following error:

    Make sure that the config root entity type is set to "pair". To do so : 1) go to the workspace and hit the "Method Configurations" tab, 2) select "pair" as the root entity type.

    2) workspace.' expected but end of source found (and similar errors)

    In the method configuration, an entry of the form "workspace.ITEM" means to lookup from the workspace attributes (in the summary tab) the key "ITEM" and find its corresponding VALUE and then use that as as the evaluated expression. I am guessing that in your workspace, you have no attributes specified and so that the lookup fails. For that configuration go to the workspace summary tab and see if it is empty?

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Hi @severinec

    You wrote

    2) Clone the updated Broad_MutationCalling_Mutect_Workflow_BestPractice_OA into a new workspace. This time, I could select a pair ID as input in this case, and it is still running. I launched it 19 hours ago. Is this expected??

    The workflow should not take that long. Was it on either of the HCC-1954 pair? or 1143?

    @severinec feel free to share with me the workspace by making me a "READER". If you want to do so, go to the "Summary" tab, then click on "Sharing..." then add me "[email protected]" as a READER. With that permission, I can glance and see if I can offer any further information

  • Hi

    I would love to share it, but it says "running" so I can't see any "sharing" under the summary tab.
    Should I abort it? maybe we cannot share a workspace while it's still running?

    looks like the VEP task has been running for 10.5 hours. see attached screenshot
    Severine

  • oops sorry.. just saw the sharing link. doing it now

  • ok, just shared the workspace that's been running for 20 hours

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @severinec I received the shared workspace.

    You are correct that it has been "Running" for 20 hours. It should take maybe 15 min or so however.

    If you glance into the workflow status you will see that the VEP task is still marked as "Running". However, inspection into the bucket shows delocalized files. Moreover, inspection in the JES log shows that the task pulled the image, localized the files, ran the docker, delocalized files too.

    wm8b1-75c:workflows esalinas$ gsutil cat gs://fc-4fcf2e73-ea1f-4f08-86a3-204d2b9fb763/49fa8c67-cea3-474a-af1d-741a8415eeab/CallingGroup_Workflow/73e6cc62-529a-4b9f-bc38-187bd3428012/call-VEP_Task/VEP_Task.log|grep -i 'switch'
    2017/02/24 09:18:27 I: Switching to status: pulling-image
    2017/02/24 09:19:20 I: Switching to status: localizing-files
    2017/02/24 09:20:20 I: Switching to status: running-docker
    2017/02/24 09:21:04 I: Switching to status: delocalizing-files
    2017/02/24 09:21:05 I: Switching to status: copied 1 file(s) to "gs://fc-4fcf2e73-ea1f-4f08-86a3-204d2b9fb763/49fa8c67-cea3-474a-af1d-741a8415eeab/CallingGroup_Workflow/73e6cc62-529a-4b9f-bc38-187bd3428012/call-VEP_Task/variant_effect_output.txt_summary.html"
    2017/02/24 09:21:06 I: Switching to status: copied 1 file(s) to "gs://fc-4fcf2e73-ea1f-4f08-86a3-204d2b9fb763/49fa8c67-cea3-474a-af1d-741a8415eeab/CallingGroup_Workflow/73e6cc62-529a-4b9f-bc38-187bd3428012/call-VEP_Task/VEP_Task-rc.txt"
    2017/02/24 09:21:07 I: Switching to status: copied 1 file(s) to "gs://fc-4fcf2e73-ea1f-4f08-86a3-204d2b9fb763/49fa8c67-cea3-474a-af1d-741a8415eeab/CallingGroup_Workflow/73e6cc62-529a-4b9f-bc38-187bd3428012/call-VEP_Task/variant_effect_output.txt"
    wm8b1-75c:workflows esalinas$ 
    

    Please know that FC workflow submissions may currently be experiencing technical difficulties. For example see this post

    http://gatkforums.broadinstitute.org/firecloud/discussion/9072/job-is-done-but-cromwell-thinks-it-is-still-running#latest

  • I also shared the workspace where I imported the cfg,

    the workspace attributes are not empty.
    but when I launch the analysis, I get this:

    Error 400: MethodConfiguration broadinstitute_cga/MutationCalling_Mutect_v1-3_BETA is missing definitions for these inputs: CallingGroup_Workflow.oncoDBTarBall, CallingGroup_Workflow.mutectForcecallIntervals, CallingGroup_Workflow.refFastaIdx, CallingGroup_Workflow.dbSNPVCF, CallingGroup_Workflow.tumorBam, CallingGroup_Workflow.mutectIntervals, CallingGroup_Workflow.fracContam, CallingGroup_Workflow.refFasta, CallingGroup_Workflow.normalBam, CallingGroup_Workflow.readGroupBlackList, CallingGroup_Workflow.oncoVEPDiskGB, CallingGroup_Workflow.preemptible, CallingGroup_Workflow.oncoMemoryGB, CallingGroup_Workflow.VEPMemoryGB, CallingGroup_Workflow.cosmicVCF, CallingGroup_Workflow.pairName, CallingGroup_Workflow.downsampleToCoverage, CallingGroup_Workflow.tumorBamIdx, CallingGroup_Workflow.refFastaDict, CallingGroup_Workflow.mutectDiskGB, CallingGroup_Workflow.VEP_File, CallingGroup_Workflow.normalBamIdx, CallingGroup_Workflow.normalPanel, CallingGroup_Workflow.nWay

  • so is the workflow that has been running for 20 hours actually done? can i see the outputs?
    Or should I just restart it and hope it will finish in 15 min?

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited February 2017

    hi @severinec I got a workspace nci-severine-catreux-gmail-com/Broad_MutationCalling_Mutect_Workflow_BestPractice_OA_v1 shared from you. I more closely examined the errors

    CallingGroup_Workflow.oncoDBTarBall - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.mutectForcecallIntervals - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.refFastaIdx - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.dbSNPVCF - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.tumorBam - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.mutectIntervals - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.fracContam - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.refFasta - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.normalBam - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.readGroupBlackList - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.oncoVEPDiskGB - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.preemptible - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.oncoMemoryGB - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.VEPMemoryGB - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.cosmicVCF - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.pairName - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.downsampleToCoverage - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.tumorBamIdx - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.refFastaDict - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.mutectDiskGB - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.VEP_File - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.normalBamIdx - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.normalPanel - Failed at line 1, column 1: `workspace.' expected but end of source found
    CallingGroup_Workflow.nWay - Failed at line 1, column 1: `workspace.' expected but end of source found
    

    I think what you did is you took the WDL method MutationCalling_Mutect_v1-3_BETA from the method repo and created a new configuration from it and that you should have instead imported into your workspace an existing configuration :
    broadinstitute_cga/MutationCalling_Mutect_v1-3_BETA_cfg#2

    The configuration has all the inputs pre-filled in.
    The WDL in contrast has nothing filled in and so the errors resulted.

    Post edited by esalinas on
Sign In or Register to comment.