Latest Release: 05/01/19
Release Notes can be found here.

missing error message for workflows? missing from workflow log? missing from UI?

esalinasesalinas BroadMember, Broadie ✭✭✭

I made and ran workflows.

The workflows failed.

I'm pretty sure that one of the workflows failed because the file referenced does not exist.

If I go to the workflow input and click "Show" and click on the link for the file (a BAM), it says "Error! This file was not found."

An issue is that there are over 100 workflows and each failed. I can write a script that uses the FC API and/or gsutil (for listing and/or statting the bucket) to determine for each of the 100+ if the reason for failure is a file not found in the bucket. I mention scripting because clicking manually for so many samples would take a while.

I note that for the screen for the submission (which lists all the workflows) has a column called "Messages". I note that the fields for that column are always empty in the submission. I note that if I "drill" to a workflow there is no message either about the reason for a workflow's failure.

I note that the workflow log has 5 lines of text, none of which mention the reason for the workflow's failure and each of which is marked "INFO" (except for the line mentioning creating authentication file) :

wm8b1-75c:coad esalinas$ gsutil cat gs://fc-6df63763-a23e-4c5f-a9d6-c24e650f3dee/a5013c83-7052-43ad-9217-24b848bcec7b/workflow.logs/workflow.253bd08d-5a15-478c-ba28-9ab1f31fbcee.log|grep -in .
1:2017-01-11 17:44:38,125 INFO  - MaterializeWorkflowDescriptorActor [UUID(253bd08d)]: Call-to-Backend assignments: polysolverWorkflow.sizeCompute -> JES, polysolverWorkflow.polysolverType -> JES, polysolverWorkflow.polysolverMut -> JES, polysolverWorkflow.polysolverAnnot -> JES
2:2017-01-11 17:44:38,524 INFO  - JES [UUID(253bd08d)]: Creating authentication file for workflow 253bd08d-5a15-478c-ba28-9ab1f31fbcee at 
3: gs://cromwell-auth-broad-firecloud-testing/253bd08d-5a15-478c-ba28-9ab1f31fbcee_auth.json
4:2017-01-11 17:44:38,822 INFO  - WorkflowExecutionActor-253bd08d-5a15-478c-ba28-9ab1f31fbcee [UUID(253bd08d)]: Starting calls: polysolverWorkflow.sizeCompute:NA:1
5:2017-01-11 17:44:40,710 INFO  - $j [UUID(253bd08d)]: Copying workflow logs from /cromwell-workflow-logs/workflow.253bd08d-5a15-478c-ba28-9ab1f31fbcee.log to gs://fc-6df63763-a23e-4c5f-a9d6-c24e650f3dee/a5013c83-7052-43ad-9217-24b848bcec7b/workflow.logs/workflow.253bd08d-5a15-478c-ba28-9ab1f31fbcee.log
wm8b1-75c:coad esalinas$ 

I have never found the workflow log to be useful in debugging during usage of firecloud to find errors.

Should the workflow log have error messages in it? Should the "Messages" column in the UI be filled in with an error message? I have a firecloud wish for 100% error reporting that FC will report every error for time-saving purposes.


  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    Hi @esalinas,

    This is certainly an area where we need to make improvements. Part of the difficulty is identifying all the different types of errors and handling them appropriately; having a record of your experience will definitely help us with that.

    I think some of the lowest-hanging fruit lies in validating the inputs up front; that would get rid of what I expect will be a large proportion of reproducible user errors (as opposed to transient system errors). We have a story queued for this.

    For the rest, it seems that a lot of what needs to be done to improve error reporting is at the level of Cromwell, and the Cromwell team is working on some of that this quarter, I believe.

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Hi @Geraldine_VdAuwera ,

    Thank you for your forum post and the update!


Sign In or Register to comment.