workflow: SingleWorkflowRunnerActor workflow finished with status 'Failed'.

edited September 2018 in Ask the FireCloud Team

i am running the example workflow locally following instrctions here:
https://gatkforums.broadinstitute.org/gatk/discussion/12521/how-to-execute-workflows-from-the-gatk-workflows-git-organization

I got an error. what went wrong? Is this due to the bam file itself? thanks

[email protected] gatk-workflows]$ /gsc/software/linux-x86_64/jre1.8.0_66/bin/java -jar cromwell-33.1.jar run ./seq-format-validation/validate-bam.wdl --inputs ./seq-format-validation/validate-bam.inputs.json[2018-09-24 16:53:11,97] [info] Running with database db.url = jdbc:hsqldb:mem:5a8b3c34-4e77-4260-b284-156e4ff20c55;shutdown=false;hsqldb.tx=mvcc
[2018-09-24 16:53:21,55] [info] Running migration RenameWorkflowOptionsInMetadata with a read batch size of 100000 and a write batch size of 100000
[2018-09-24 16:53:21,57] [info] [RenameWorkflowOptionsInMetadata] 100%
[2018-09-24 16:53:21,67] [info] Running with database db.url = jdbc:hsqldb:mem:3e6a191d-dc8a-4990-b523-dec612287786;shutdown=false;hsqldb.tx=mvcc
[2018-09-24 16:53:22,19] [info] Slf4jLogger started
[2018-09-24 16:53:22,61] [info] Workflow heartbeat configuration:
{
  "cromwellId" : "cromid-d687857",
  "heartbeatInterval" : "2 minutes",
  "ttl" : "10 minutes",
  "writeBatchSize" : 10000,
  "writeThreshold" : 10000
}
[2018-09-24 16:53:22,68] [info] Metadata summary refreshing every 2 seconds.
[2018-09-24 16:53:22,72] [info] WriteMetadataActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-24 16:53:22,72] [info] KvWriteActor configured to flush with batch size 200 and process rate 5 seconds.
[2018-09-24 16:53:22,72] [info] CallCacheWriteActor configured to flush with batch size 100 and process rate 3 seconds.
[2018-09-24 16:53:24,02] [info] JobExecutionTokenDispenser - Distribution rate: 50 per 1 seconds.
[2018-09-24 16:53:24,04] [info] SingleWorkflowRunnerActor: Submitting workflow
[2018-09-24 16:53:24,12] [info] Unspecified type (Unspecified version) workflow 92883a94-765a-414d-a0aa-9fa3f5a2e6af submitted
[2018-09-24 16:53:24,19] [info] SingleWorkflowRunnerActor: Workflow submitted 92883a94-765a-414d-a0aa-9fa3f5a2e6af
[2018-09-24 16:53:24,19] [info] 1 new workflows fetched
[2018-09-24 16:53:24,19] [info] WorkflowManagerActor Starting workflow 92883a94-765a-414d-a0aa-9fa3f5a2e6af
[2018-09-24 16:53:24,20] [info] WorkflowManagerActor Successfully started WorkflowActor-92883a94-765a-414d-a0aa-9fa3f5a2e6af
[2018-09-24 16:53:24,20] [info] Retrieved 1 workflows from the WorkflowStoreActor
[2018-09-24 16:53:24,21] [warn] SingleWorkflowRunnerActor: received unexpected message: Done in state RunningSwraData
[2018-09-24 16:53:24,23] [warn] Couldn't find a suitable DSN, defaulting to a Noop one.
[2018-09-24 16:53:24,24] [info] Using noop to send events.
[2018-09-24 16:53:24,27] [info] WorkflowStoreHeartbeatWriteActor configured to flush with batch size 10000 and process rate 2 minutes.
[2018-09-24 16:53:24,30] [info] MaterializeWorkflowDescriptorActor [92883a94]: Parsing workflow as WDL draft-2
[2018-09-24 16:53:25,35] [info] MaterializeWorkflowDescriptorActor [92883a94]: Call-to-Backend assignments: ValidateBamsWf.ValidateBAM -> Local
[2018-09-24 16:53:25,50] [warn] Local [92883a94]: Key/s [memory, cpu, disks] is/are not supported by backend. Unsupported attributes will not be part of job executions.
[2018-09-24 16:53:31,01] [info] WorkflowExecutionActor-92883a94-765a-414d-a0aa-9fa3f5a2e6af [92883a94]: Starting ValidateBamsWf.ValidateBAM
[2018-09-24 16:53:32,48] [warn] BackgroundConfigAsyncJobExecutionActor [92883a94ValidateBamsWf.ValidateBAM:0:1]: Unrecognized runtime attribute keys: disks, cpu, memory
[2018-09-24 16:53:33,50] [info] BackgroundConfigAsyncJobExecutionActor [92883a94ValidateBamsWf.ValidateBAM:0:1]: /gatk/gatk \
  ValidateSamFile \
  --INPUT /cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/inputs/362257838/NA12878_24RG_small.hg38.bam \
  --OUTPUT NA12878_24RG_small.hg38.validation_SUMMARY.txt \
  --MODE SUMMARY
[2018-09-24 16:53:33,56] [info] BackgroundConfigAsyncJobExecutionActor [92883a94ValidateBamsWf.ValidateBAM:0:1]: executing: # make sure there is no preexisting Docker CID file
rm -f /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid
# run as in the original configuration without --rm flag (will remove later)
docker run \
  --cidfile /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid \
  -i \
   \
  --entrypoint /bin/bash \
  -v /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0:/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0 \
  broadinstitute/[email protected]:497185dbf683c3ae5deda0c0b42e641fad04d724d0cb887dd7d601fe30b96ca2 /cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/script

# get the return code (working even if the container was detached)
rc=$(docker wait cat /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid)

# remove the container after waiting
docker rm cat /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid

# return exit code
exit $rc
[2018-09-24 16:53:37,77] [info] BackgroundConfigAsyncJobExecutionActor [92883a94ValidateBamsWf.ValidateBAM:0:1]: job id: 123319
[2018-09-24 16:53:37,78] [info] BackgroundConfigAsyncJobExecutionActor [92883a94ValidateBamsWf.ValidateBAM:0:1]: Status change from - to Done
[2018-09-24 16:53:38,56] [error] WorkflowManagerActor Workflow 92883a94-765a-414d-a0aa-9fa3f5a2e6af failed (during ExecutingWorkflowState): Job ValidateBamsWf.ValidateBAM:0:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
Check the content of stderr for potential additional information: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr.
 Could not retrieve content: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr
[2018-09-24 16:53:38,56] [info] WorkflowManagerActor WorkflowActor-92883a94-765a-414d-a0aa-9fa3f5a2e6af is in a terminal state: WorkflowFailedState
[2018-09-24 16:53:45,29] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
[2018-09-24 16:53:47,74] [info] Workflow polling stopped
[2018-09-24 16:53:47,76] [info] Shutting down WorkflowStoreActor - Timeout = 5 seconds
[2018-09-24 16:53:47,76] [info] Shutting down WorkflowLogCopyRouter - Timeout = 5 seconds
[2018-09-24 16:53:47,77] [info] Shutting down JobExecutionTokenDispenser - Timeout = 5 seconds
[2018-09-24 16:53:47,77] [info] Aborting all running workflows.
[2018-09-24 16:53:47,77] [info] JobExecutionTokenDispenser stopped
[2018-09-24 16:53:47,77] [info] WorkflowStoreActor stopped
[2018-09-24 16:53:47,78] [info] WorkflowLogCopyRouter stopped
[2018-09-24 16:53:47,78] [info] Shutting down WorkflowManagerActor - Timeout = 3600 seconds
[2018-09-24 16:53:47,78] [info] WorkflowManagerActor All workflows finished
[2018-09-24 16:53:47,78] [info] WorkflowManagerActor stopped
[2018-09-24 16:53:47,78] [info] Connection pools shut down
[2018-09-24 16:53:47,79] [info] Shutting down SubWorkflowStoreActor - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] Shutting down JobStoreActor - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] Shutting down CallCacheWriteActor - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] SubWorkflowStoreActor stopped
[2018-09-24 16:53:47,79] [info] Shutting down ServiceRegistryActor - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] Shutting down DockerHashActor - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] Shutting down IoProxy - Timeout = 1800 seconds
[2018-09-24 16:53:47,79] [info] JobStoreActor stopped
[2018-09-24 16:53:47,79] [info] CallCacheWriteActor Shutting down: 0 queued messages to process
[2018-09-24 16:53:47,79] [info] KvWriteActor Shutting down: 0 queued messages to process
[2018-09-24 16:53:47,79] [info] WriteMetadataActor Shutting down: 0 queued messages to process
[2018-09-24 16:53:47,79] [info] CallCacheWriteActor stopped
[2018-09-24 16:53:47,79] [info] DockerHashActor stopped
[2018-09-24 16:53:47,79] [info] IoProxy stopped
[2018-09-24 16:53:47,80] [info] ServiceRegistryActor stopped
[2018-09-24 16:53:47,83] [info] Database closed
[2018-09-24 16:53:47,83] [info] Stream materializer shut down
Workflow 92883a94-765a-414d-a0aa-9fa3f5a2e6af transitioned to state Failed
[2018-09-24 16:53:47,89] [info] Automatic shutdown of the async connection
[2018-09-24 16:53:47,89] [info] Gracefully shutdown sentry threads.
[2018-09-24 16:53:47,89] [info] Shutdown finished.
Post edited by bshifaw on

Best Answers

Answers

  • bshifawbshifaw Member, Broadie, Moderator admin

    Hi @stuartzong1111

    [2018-09-24 16:53:38,56] [error] WorkflowManagerActor Workflow 92883a94-765a-414d-a0aa-9fa3f5a2e6af failed (during ExecutingWorkflowState): Job ValidateBamsWf.ValidateBAM:0:1 exited with return code -1 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.
    Check the content of stderr for potential additional information: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr.
     Could not retrieve content: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr
    

    As stated in the message above, check the standard error file to view additional details as to why your workflow failed. /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr

  • edited September 2018

    Hi bshifaw, thanks for looking into this. it seems that it is a docker problem. do you know if i could install docker without sudo previledge? thanks

    [[email protected] gatk-workflows]$ cat /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/stderr.background 
    /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/script.submit: line 5: docker: command not found
    cat: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid: No such file or directory
    /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/script.submit: line 14: docker: command not found
    cat: /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/docker_cid: No such file or directory
    /projects/da_workspace/software/gatk-workflows/cromwell-executions/ValidateBamsWf/92883a94-765a-414d-a0aa-9fa3f5a2e6af/call-ValidateBAM/shard-0/execution/script.submit: line 17: docker: command not found
    
    Post edited by bshifaw on
  • bshifawbshifaw Member, Broadie, Moderator admin
    edited September 2018

    This is outside my expertise, but the following links might be helpful:
    how-can-i-use-docker-without-sudo
    manage-docker-as-a-non-root-user

    Edit:
    @stuartzong1111 I may have misinterpreted your question, if your just trying to install docker and you don't have sudo access then that might be a problem your Sys Admin would need to handle.

    Post edited by bshifaw on
  • bshifawbshifaw Member, Broadie, Moderator admin

    Hi @stuartzong1111 ,

    Did you have any luck with installing Docker?

  • KateNKateN Cambridge, MAMember, Broadie, Moderator admin

    Hello @stuartzong1111,

    Unfortunately, as we have not yet heard back from you I will be closing this ticket. If your question was not fully answered, please do let us know.

  • freedogfreedog Member
    edited October 2018

    Hi @Tiffany_at_Broad, must I using Docker to run this workflow locally?

  • SChaluvadiSChaluvadi Member, Broadie, Moderator admin

    Hey there @freedog! Hope Tiffany_at_Broad's answer was helpful in answering your question. Since we haven't heard back from you, this ticket will be closed and her answer will be marked as "Accepted". If you have any follow-ups you can always reply back!

Sign In or Register to comment.