Cromwell status failed during WDL script running bwa mem

dbeckerdbecker MunichMember

Hi!

I try to build a workflow in WDL step by step. Since it is based on tasks it is quite easy to extract the step you want to establish and insert it into your main workflow afterwards.
I'm still at the very begininning, which is (after parsing some information from my files via embeded python code) the alignemnt using bwa mem.
The data here is a small example. In the real workflow it should work with scatter gather over all 8 fastq files per sample (4 lanes) and many samples. But since it always throws the error below I build this as a minimal example. I tried to google the error and read the source code of cromwell, but I could not find something making sense to me. Most people talk about the config file of cromwell but this wouldn't make any difference in my opinion since the problem is not related to multithreading and resources. I run the Workflow on a local Linux server with 64Gb of RAM.

My workflow validates without problems:

workflow run_wdl_gatk {
    File give_referenceGenome
    String give_inDirectory 
    String PL = "illumina"      # platform  - illumina
    String CN = "MVZ"           # sequencing center 

    call mapping {
        input:
            inDirectory = "../../../../../161104_NS500396_0127_AHGKGLAFXX/",
            outDirectory = "../../../../../161104_NS500396_0127_AHGKGLAFXX/_gatk/",
            bamFile = "1643-0547-Kardio",
            ID = "NS500396",
            PL = PL,
            DT = "2016-11-04",
            CN = CN,
            PU = "AHGKGLAFXX"
    }
}
task mapping {
    String inDirectory
    String outDirectory
    String bamFile
    String ID
    String PL
    String CN
    String DT
    String PU

    command <<<
        bwa mem \
            -M \
            -R '@RG\tID:${ID}.L001\tSM:${bamFile}\tPL:${PL}\tCN:${CN}\tDT:${DT}\tPU:${PU}.L001\tLB:LB-${bamFile}' \
            hg19.fa \
            ${inDirectory}${bamFile}_L001_R1_001.fastq.gz \
            ${inDirectory}${bamFile}_L001_R2_001.fastq.gz > ${outDirectory}${bamFile}_L001.sam
    >>>
}

My Error:

[2016-12-22 09:09:38,09] [info] RUN sub-command
[2016-12-22 09:09:38,11] [info]   WDL file: /srv/samba/bioinf/GATK/test_bwa.wdl
[2016-12-22 09:09:38,11] [info]   Inputs: /srv/samba/bioinf/GATK/inputs.json
[INFO] [12/22/2016 09:09:38.144] [cromwell-system-akka.actor.default-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Submitting workflow
[2016-12-22 09:09:38,80] [info] Running with database db.url = jdbc:hsqldb:mem:267e4bc4-6ca8-4853-9548-1e746515acd8;shutdown=false;hsqldb.tx=mvcc
[INFO] [12/22/2016 09:09:42.665] [cromwell-system-akka.actor.default-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor/ServiceRegistryActor/MetadataService] Metadata summary refreshing every 2 seconds.
[INFO] [12/22/2016 09:09:42.887] [cromwell-system-akka.actor.default-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowStoreActor] Workflow e537a4c6-136b-4d23-a52a-73e6ec3bba83 submitted.
[INFO] [12/22/2016 09:09:42.887] [cromwell-system-akka.actor.default-dispatcher-3] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor: Workflow submitted UUID(e537a4c6-136b-4d23-a52a-73e6ec3bba83)
[INFO] [12/22/2016 09:09:42.994] [cromwell-system-akka.actor.default-dispatcher-3] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowStoreActor] 1 new workflows fetched
[INFO] [12/22/2016 09:09:42.995] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor] WorkflowManagerActor Starting workflow UUID(e537a4c6-136b-4d23-a52a-73e6ec3bba83)
[INFO] [12/22/2016 09:09:43.000] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor] WorkflowManagerActor Successfully started WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83
[INFO] [12/22/2016 09:09:43.001] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor] Retrieved 1 workflows from the WorkflowStoreActor
[INFO] [12/22/2016 09:09:43.165] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/MaterializeWorkflowDescriptorActor] MaterializeWorkflowDescriptorActor [UUID(e537a4c6)]: Call-to-Backend assignments: run_wdl_gatk.mapping -> Local
[INFO] [12/22/2016 09:09:43.293] [cromwell-system-akka.dispatchers.engine-dispatcher-30] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83] WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83 [UUID(e537a4c6)]: Starting calls: run_wdl_gatk.mapping:NA:1
[INFO] [12/22/2016 09:09:43.421] [cromwell-system-akka.dispatchers.backend-dispatcher-35] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/e537a4c6-136b-4d23-a52a-73e6ec3bba83-EngineJobExecutionActor-run_wdl_gatk.mapping:NA:1/e537a4c6-136b-4d23-a52a-73e6ec3bba83-BackendJobExecutionActor-e537a4c6:run_wdl_gatk.mapping:-1:1/SharedFileSystemAsyncJobExecutionActor] SharedFileSystemAsyncJobExecutionActor [UUID(e537a4c6)run_wdl_gatk.mapping:NA:1]: `bwa mem \
        -M \
        -R '@RG\tID:NS500396.L001\tSM:1643-0547-Kardio\tPL:illumina\tCN:MVZ\tDT:2016-11-04\tPU:AHGKGLAFXX.L001\tLB:LB-1643-0547-Kardio' \
        hg19.fa \
        ../../../../../161104_NS500396_0127_AHGKGLAFXX/1643-0547-Kardio_L001_R1_001.fastq.gz \
        ../../../../../161104_NS500396_0127_AHGKGLAFXX/1643-0547-Kardio_L001_R2_001.fastq.gz > ../../../../../161104_NS500396_0127_AHGKGLAFXX/_gatk/1643-0547-Kardio_L001.sam`
[INFO] [12/22/2016 09:09:43.429] [cromwell-system-akka.dispatchers.backend-dispatcher-35] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/e537a4c6-136b-4d23-a52a-73e6ec3bba83-EngineJobExecutionActor-run_wdl_gatk.mapping:NA:1/e537a4c6-136b-4d23-a52a-73e6ec3bba83-BackendJobExecutionActor-e537a4c6:run_wdl_gatk.mapping:-1:1/SharedFileSystemAsyncJobExecutionActor] SharedFileSystemAsyncJobExecutionActor [UUID(e537a4c6)run_wdl_gatk.mapping:NA:1]: executing: /bin/bash /srv/samba/bioinf/GATK/cromwell-executions/run_wdl_gatk/e537a4c6-136b-4d23-a52a-73e6ec3bba83/call-mapping/execution/script
[INFO] [12/22/2016 09:09:43.430] [cromwell-system-akka.dispatchers.backend-dispatcher-35] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/e537a4c6-136b-4d23-a52a-73e6ec3bba83-EngineJobExecutionActor-run_wdl_gatk.mapping:NA:1/e537a4c6-136b-4d23-a52a-73e6ec3bba83-BackendJobExecutionActor-e537a4c6:run_wdl_gatk.mapping:-1:1/SharedFileSystemAsyncJobExecutionActor] SharedFileSystemAsyncJobExecutionActor [UUID(e537a4c6)run_wdl_gatk.mapping:NA:1]: command: "/bin/bash" "/srv/samba/bioinf/GATK/cromwell-executions/run_wdl_gatk/e537a4c6-136b-4d23-a52a-73e6ec3bba83/call-mapping/execution/script.submit"
[INFO] [12/22/2016 09:09:43.446] [cromwell-system-akka.dispatchers.backend-dispatcher-35] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/WorkflowExecutionActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83/e537a4c6-136b-4d23-a52a-73e6ec3bba83-EngineJobExecutionActor-run_wdl_gatk.mapping:NA:1/e537a4c6-136b-4d23-a52a-73e6ec3bba83-BackendJobExecutionActor-e537a4c6:run_wdl_gatk.mapping:-1:1/SharedFileSystemAsyncJobExecutionActor] SharedFileSystemAsyncJobExecutionActor [UUID(e537a4c6)run_wdl_gatk.mapping:NA:1]: job id: 411
[ERROR] [12/22/2016 09:09:44.440] [cromwell-system-akka.dispatchers.engine-dispatcher-34] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor] WorkflowManagerActor Workflow e537a4c6-136b-4d23-a52a-73e6ec3bba83 failed (during ExecutingWorkflowState): java.lang.Exception: Call run_wdl_gatk.mapping: return code was 1
[INFO] [12/22/2016 09:09:44.441] [cromwell-system-akka.dispatchers.engine-dispatcher-34] [akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor] WorkflowManagerActor WorkflowActor-e537a4c6-136b-4d23-a52a-73e6ec3bba83 is in a terminal state: WorkflowFailedState
[INFO] [12/22/2016 09:09:45.824] [cromwell-system-akka.actor.default-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[ERROR] [12/22/2016 09:09:45.825] [cromwell-system-akka.actor.default-dispatcher-5] [akka://cromwell-system/user/SingleWorkflowRunnerActor] Workflow e537a4c6-136b-4d23-a52a-73e6ec3bba83 transitioned to state Failed
java.lang.RuntimeException: Workflow e537a4c6-136b-4d23-a52a-73e6ec3bba83 transitioned to state Failed
        at cromwell.engine.workflow.SingleWorkflowRunnerActor$RunnerData.addFailure(SingleWorkflowRunnerActor.scala:53)
        at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:132)
        at cromwell.engine.workflow.SingleWorkflowRunnerActor$$anonfun$2.applyOrElse(SingleWorkflowRunnerActor.scala:110)
        at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
        at akka.actor.FSM$class.processEvent(FSM.scala:663)
        at cromwell.engine.workflow.SingleWorkflowRunnerActor.akka$actor$LoggingFSM$$super$processEvent(SingleWorkflowRunnerActor.scala:70)
        at akka.actor.LoggingFSM$class.processEvent(FSM.scala:799)
        at cromwell.engine.workflow.SingleWorkflowRunnerActor.processEvent(SingleWorkflowRunnerActor.scala:70)
        at akka.actor.FSM$class.akka$actor$FSM$$processMsg(FSM.scala:657)
        at akka.actor.FSM$$anonfun$receive$1.applyOrElse(FSM.scala:651)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:484)
        at cromwell.server.CromwellRootActor.aroundReceive(CromwellRootActor.scala:27)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
        at akka.actor.ActorCell.invoke(ActorCell.scala:495)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
        at akka.dispatch.Mailbox.run(Mailbox.scala:224)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Workflow e537a4c6-136b-4d23-a52a-73e6ec3bba83 transitioned to state Failed

I don't really know how to format my post correctly, so please excuse my layout.

Thanks in advance.

Daniel

Post edited by KateN on

Best Answer

Answers

  • EADGEADG KielMember ✭✭✭

    Hi @dbecker,

    did you allready check the stdout and stderr for your mapping task?

    Greetings EADG

  • dbeckerdbecker MunichMember

    Thanks for the help and the formatting.

    I had a wrong path in my code because I had forgotten that cromwell runs the process in some sub directory. I had to go a few directories up to find my index file.

    btw... is there a guide on how to format code properly for the next time?

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin
    You can use either markdown or HTML. We don't have a guide but the simplest markdown for when you have multiple lines of code is to type 4 backticks, go to a new line, paste in your code then close with another new line and 4 backticks.
  • EADGEADG KielMember ✭✭✭

    Hi @dbecker,

    you can start your code block with 3 times Tilde(~) , add your code lines and close it with 3 times Tilde. GATK/WDL is running vanilla as forum software, so you can easily google for specific formating options.

    Greetings EADG

Sign In or Register to comment.