Question on caching

Say I have 1 workflow that has 1000 scatter chunks, with caching pointing to a mysql database. The workflow failed when 100 scatter tasks were complete, but a further 400 was still running at the time of failure (an LSF system) and were able to successfully complete. Will the 400 tasks that finish after the main cromwell process failed still be recorded in the cache?

Best Answer

  • awacsawacs
    Accepted Answer

    This relates to a separate question I posted where the cromwell process itself dies because of a race condition related to the stderr file. On my setup (a LSF system), this means the cromwell process dies while the tasks keep on running to completion, but your answer seem to imply that since the cromwell process itself died, even though the "orphaned" tasks finished completely, since the main cromwell process, which is responsible for writing to cache, is not alive, the completed tasks will not be registered in the cache. Am I correct?

Answers

  • danbdanb Member, Broadie ✭✭

    The 400 that are still running will not be recorded in the cache by default, even if they finish successfully.

    However, using the workflow_failure_mode set to ContinueWhilePossible you will get the desired result. See the Cromwell github docs regarding workflow options for more information.

  • awacsawacs Member

    @danb said:
    The 400 that are still running will not be recorded in the cache by default, even if they finish successfully.

    However, using the workflow_failure_mode set to ContinueWhilePossible you will get the desired result. See the Cromwell github docs regarding workflow options for more information.

    Does that just keeps the main cromwell process alive? What if cromwell fails regardless? AKA is the main cromwell process the only one that is writing to cache so to speak? Or is the child process also writing to cache?

  • danbdanb Member, Broadie ✭✭

    Yes it keeps the main cromwell process "alive."

    I don't understand your query about failing "regardless," could you re-phrase or perhaps elaborate?

    Cromwell is writing to cache, not the task itself.

  • awacsawacs Member
    Accepted Answer

    This relates to a separate question I posted where the cromwell process itself dies because of a race condition related to the stderr file. On my setup (a LSF system), this means the cromwell process dies while the tasks keep on running to completion, but your answer seem to imply that since the cromwell process itself died, even though the "orphaned" tasks finished completely, since the main cromwell process, which is responsible for writing to cache, is not alive, the completed tasks will not be registered in the cache. Am I correct?

  • danbdanb Member, Broadie ✭✭

    You are correct.

Sign In or Register to comment.