Where to find error log for failed analysis?

An analysis performed with a new method failed. The workspace in which it was run is highlighted in red in my list of workspaces.

I can browse into that workspace to the Monitor tab and see the list of Method Configurations. Then clicking on a Method Configuration shows a list of runs and a status of "Failed!". But I do not see a link where I can find an error message to know where and why it failed.

Thank you.

Best Answer

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Hi Dave.

    You can find the stdout, stderr, and JES logs by drilling down.
    If you watch this youtube (starting at 6:51) you can see the drill-down "route" in the UI

    Also, if you go to the summary page you can click on the link to the google bucket.
    There you can find a folder for the submission. Then in that folder look for a subfolder called "worfklow.logs" . for the corresponding workflow you can find the log. See attached figure for example....gs://workspacebucket/submission/ID/workflow.logs is the pattern.

    -eddie

  • daveMdaveM BostonMember

    Hi Eddie,

    Both methods work fine on the jobs that I ran from tutorials, which completed successfully. But when I am attempting to develop a new method and the job fails, the google bucket is empty except for input files. Also, there is no link to drill down to. See the attached images.

    Dave

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2016

    Hi Dave. These figures you've posted are helpful to understand things. Based on the "Analysis-Failed" link it looks like there's an error about getting input to the workflow - specifically the inputBam. The workflow never started because the input never got to it. Something about the method config or a referenced data entity or workspace attribute is set so that input is not properly brought in.

    If you can post the error you see in the "Analysis-failed" figure that could be helpful.

    Also if you post a figure of your method config that could be helpful.

    Also if you go to the data tab and download a TSV for the samples that could be helpful too. I say samples because I think your root entity type here is a sample.

    -eddie

    Post edited by esalinas on
  • daveMdaveM BostonMember

    Oh! I see the error message now. I was looking for a link, There are no links but it's in the Messages column on the Monitor tab:
    SomaticVariantCaller.bamUtilTask.input_bam - Expected single value for workflow input, but evaluated result set was empty

    Attached is an image of the Method Config and also the sample data file. Yes, sample is the root entity. The input bam file for this method is in the "bam" column of the sample data. I've taken the URL for the bam file by looking at the bucket and right-clicking to grab the url for the bam. It seems to be correct because I can paste the URL into my browser and it starts to download the bam.

    One thing I'm not sure of in the Method Config is the way that I've handled the output filename. It's an input string, but then a file in the output section. That seemed like what was done in one of the tutorials.

    Dave

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2016

    Hi Dave,

    I think I see the issue now. The method config inputs are case-sensitive.

    In your config you have this.bam ("bam" is lowercase)

    In your sample entity you have a column called "BAM" (all uppercase)

    Try either changing "this.bam" to "this.BAM" or alternatively edit the TSV you posted in the forum and change "BAM" to "bam" then upload it.

    -eddie

    Post edited by esalinas on
  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2016

    Referring to your output, based on my review of your WDL, the column header and the file name will be the same which is fine, BUT it could possibly be confusing.

  • daveMdaveM BostonMember

    OK, yes, the problem was that the cases didn't match. You answer to my original question (location of the error message) above, so I noted that this question has been answered. My job is still not running successfully, but it's going further. There is a folder for the execution in the Google bucket. I can drill down to a call-bamUtilTask folder and find a script, exec.sh, that looks like this:

    !/bin/bash

    export _JAVA_OPTIONS=-Djava.io.tmpdir=/cromwell_root/tmp
    export TMPDIR=/cromwell_root/tmp
    cd /cromwell_root

    bam trimBam https://storage.cloud.google.com/fc-64f9579e-7fdd-42cb-a45a-ff72c4ee1a50/HCC1143-21.bam?_ga=1.51112169.1101352068.1471635948 trimmed.bam -L 30

    echo $? > /cromwell_root/bamUtilTask-rc.txt

    There is no stdout or stderr in the bucket.

    When I ran the job using Cromwell locally, I could drill down in the Cromwell-executions folder to find stdout, stderr, and a few other files.

    On the Monitor tab in the GUI, I can see the input and there are links to output logs. But clicking on those links gives a message like:

    Google Bucket: fc-64f9579e-7fdd-42cb-a45a-ff72c4ee1a50
    Object: deacfab1-0f10-42e7-9f37-603b7d3a1642/SomaticVariantCaller/588e68a1-3d94-4430-8aa7-313ec61a16ad/call-bamUtilTask/bamUtilTask-stdout.log
    Error! This file was not found.
    {
    "error": {
    "errors": [{
    "domain": "global",
    "reason": "notFound",
    "message": "Not Found"
    }],
    "code": 404,
    "message": "Not Found"
    },
    "timestamp": 1475178858279
    }

    Hide detail

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    When the job is submitted to google, note that it can take a few moments for the VM to be provisioned, boot up, for the files to be downloaded from the bucket to the VM, etc. before the job starts running. Be sure to check back in a few minutes to see if any output has then.

  • esalinasesalinas BroadMember, Broadie ✭✭✭
    edited September 2016

    the other thing I see is that in the sample.txt file you sent, that the URL is not a GS url. For example : gs://fc-64f9579e-7fdd-42cb-a45a-ff72c4ee1a50/HCC1143-21.bam . The kind of URL you have it looks like you copied from the browser GUI may be causing an issue. I've never seen that work or tried it when it's in the data entity model. You could edit the TSV and make it use GS urls instead, then upload it and retry and see if that works.

  • daveMdaveM BostonMember

    Yes, that was it. I needed to use a GS url to specify the input bam file. It runs successfully now.

    Thank you very much!

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    You're welcome!

Sign In or Register to comment.