Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/17/19
Release Notes can be found here.

submission timeouts related to number of tasks/jobs in WDL?

esalinasesalinas BroadMember, Broadie ✭✭✭

I made a "toy" workflow that seems to run fine. It calls a single task. It's a "tiny" hello world.

I redid it in "mega" style where the task is called 1000 times.

During attempted submission of the "mega" task I got timeouts but not from the regular "single" style task.

The only difference is in the number of calls using the WDL "as" keyword.

See the attached WDLs!

I note that the timeout issue I've experienced with as little as 7 tasks submissions from a single workflow!

The timeout issue seems related to the number of tasks being submitted?

-eddie

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    Checking on on the bucket contents via inspection shows that some of the "mega" tasks have run, but
    doing a listing 20 minutes later shows than none of the jobs have completed during that window.
    Does that indicate some issue? I can only speculate now....I can re-inspect the bucket contents tomorrow......

    -eddie

    wm8b1-75c:~ esalinas$ gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/|ggrep -Pi 'exec.sh$'
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/5bb1c0b1-bc0a-4081-b830-9d400d7ffd56/tinyworkflow/6ab7db3a-0404-4368-8d35-a68f3608272c/call-tinytask/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_165/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_202/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_232/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_274/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_288/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_298/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_306/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_308/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_342/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_346/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_371/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_401/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_415/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_451/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_475/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_496/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_498/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_512/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_533/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_55/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_554/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_608/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_637/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_652/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_663/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_666/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_715/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_735/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_756/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_805/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_812/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_883/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_971/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/321ab8ef-c9ec-4258-ade9-aef2fb8b286d/call-tiny_990/exec.sh
    gs://fc-703619e1-6de7-4acc-9546-6788c673684a/ade8ee21-5e20-4d02-9e90-2b229c9186a5/tinyworkflow/d81eed6c-66f4-443c-aad7-9871a28738f1/call-tiny_729/exec.sh
    wm8b1-75c:~ esalinas$ gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/
    |ggrep -Pi 'exec.sh$'|wc -l
    36
    wm8b1-75c:~ esalinas$ gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/|ggrep -Pi 'exec.sh$'|wc -l
    36
    wm8b1-75c:~ esalinas$ date && gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/
    |ggrep -Pi 'exec.sh$'|wc -l && sleep 120 && date && gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/|ggrep -Pi 'exec.sh$'|wc -l
    Mon Mar 14 12:32:38 EDT 2016
    36
    Mon Mar 14 12:34:42 EDT 2016
    36
    wm8b1-75c:~ esalinas$ date && gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/
    |ggrep -Pi 'exec.sh$'|wc -l && sleep 1200 && date && gsutil ls gs://fc-703619e1-6de7-4acc-9546-6788c673684a/**|ggrep -Pi 'exec.sh$'|wc -l
    Mon Mar 14 12:38:41 EDT 2016
    36
    Mon Mar 14 12:58:44 EDT 2016
    36
    wm8b1-75c:~ esalinas$

Sign In or Register to comment.