Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/17/19
Release Notes can be found here.

optional files not localized? bug?

esalinasesalinas BroadMember, Broadie ✭✭✭
edited February 2017 in Ask the FireCloud Team

I have an optional File input ("detailMetrics") in my WDL. (see WDL task opening with inputs including the optional input)

task generateMetrics_task {
    String ID
    File BAM
    File BAM_IDX
    File? detailMetrics
    File DBSNP
    File DBSNPIDX
    File REFERENCE
    Int diskGB

In the FireCloud UI under the submission for the task, the file is identified as an input. See the text copied/pasted from the UI here:

FilterWorkflow.generateMetrics_taskHide
Call #1:
ID:operations/ENni9pyiKxjRsY6Z2aKmhJcBIP3g3tG1AioPcHJvZHVjdGlvblF1ZXVl
Status:
warning
Failed
Started:02/09/2017 at 11:03:41 AM (2 hours ago)
Ended:02/09/2017 at 12:35:11 PM (an hour ago)
Inputs:Hide
BAM_IDX→gs://fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.bai
diskGB→750
ID→REBC-ACB7-TTP1-A-1-1-D-A49V-36
DBSNPIDX→gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf.idx
detailMetrics→gs://fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.pre_adapter_detail_metrics
REFERENCE→gs://firecloud-tcga-open-access/tutorial/reference/Homo_sapiens_assembly19.fasta
DBSNP→gs://firecloud-tcga-open-access/tutorial/reference/dbsnp_134_b37.leftAligned.vcf
BAM→gs://fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.bam
Outputs:None

In the exec.sh script, the file is referenced (see how GREP picks it from the exec.sh):

[email protected]:~/REBC/merge_work/docker$ gsutil cat gs://fc-7119068c-40f9-4960-9dee-1d73730e9c1c/e0faaf2a-a193-49b4-9ed0-17ddcc5dab86/FilterWorkflow/398c7660-97c6-4079-b98b-bbce037edda1/call-generateMetrics_task/exec.sh|grep -Pi 'metric'
#see if a metrics file was passed in and create one if not
METRICS_IN_LEN=`echo '/cromwell_root/fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.pre_adapter_detail_metrics'|tr -d "\n"|wc -c` ;
if [ "$METRICS_IN_LEN" -eq "0" ] ;
    /usr/local/bin/java  -Xmx3600M  -jar /usr/local/CollectSequencingArtifactMetrics/picard.1.895.jar CollectSequencingArtifactMetrics \
    echo "Passing the metrics file along!"
    mv -v /cromwell_root/fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.pre_adapter_detail_metrics REBC-ACB7-TTP1-A-1-1-D-A49V-36.pre_adapter_detail_metrics
echo $? > /cromwell_root/generateMetrics_task-rc.txt.tmp
mv /cromwell_root/generateMetrics_task-rc.txt.tmp /cromwell_root/generateMetrics_task-rc.txt
[email protected]:~/REBC/merge_work/docker$ 

However, the file was apparently NOT localized, because when the script ran, a file-not-found error was observed. If I grep the JES log in text before the "switching to status running docker" for the file there is no output.

[email protected]:~/REBC/merge_work/docker$ gsutil cat gs://fc-7119068c-40f9-4960-9dee-1d73730e9c1c/e0faaf2a-a193-49b4-9ed0-17ddcc5dab86/FilterWorkflow/398c7660-97c6-4079-b98b-bbce037edda1/call-generateMetrics_task/generateMetrics_task.log|grep -B 10000 -Pi 'status:.running'|grep -Pi 'pre_adapter_detail_metrics'

Is this is bug?

I note that in the method-config, I used a file literal. I enclosed it with double quotes. Also, I note that in the UI, the file was apparently picked up (see "detailMetrics→gs://fc-4520e968-446a-4f9f-85e2-8fcc901269dc/REBC-ACB7-TTP1-A-1-1-D-A49V-36.pre_adapter_detail_metrics" in the task inputs section above)

Attached if it may be helpful is a copy of the JES log.

Best Answer

Answers

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @Geraldine_VdAuwera Thank you for the update.

  • ChipChip 415M 4053Member, Broadie

    Hi @Geraldine_VdAuwera @eddieasalinas @gordon123 @abaumann @birger

    I've been re-implementing a workflow called "mutation validator" from Firehose to Firecloud, starting with testing the worfklow wdl on a google VM. The purpose of mutation validator is to scan for evidence of a mutation across many datatypes (WES, WGS, RNA, ...) and so it has lots of optional inputs which I've declared with "File?" in the wdl like this:

    File? WEXTUMOR
    File? WEXNORMAL
    File? WEXTUMORBAI
    File? WEXNORMALBAI
    File? WGSTUMOR
    File? WGSNORMAL
    File? WGSTUMORBAI
    

    ....

    Is this the correct wdl convention? In the json parameter file, I've left the fields for files that don't exist empty "". Cromwell 25 has failures attempting to localize the optional files. Should I use something other than an empty field for missing optional input files? Here are the cromwell 25 starting with the warning (then failures) about localizing missing files:

    [2017-03-24 03:45:51,16] [warn] Localization via hard link has failed: /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess -> /opt/MutationValidator/tasks/mutation_validator_preprocess: Operation not permitted
    [2017-03-24 03:45:51,18] [warn] Localization via copy has failed: /opt/MutationValidator/tasks/mutation_validator_preprocess/inputs
    [2017-03-24 03:45:51,23] [error] BackgroundConfigAsyncJobExecutionActor [356bc3b2mutation_validator_preprocess_workflow.mutation_validator_preprocess:NA:1]: Error attempting to Execute
    cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1$$anon$1: :
    :
    Could not localize -> /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess:
    doesn't exists
    /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess -> /opt/MutationValidator/tasks/mutation_validator_preprocess: Operation not permitted
    File not found /opt/MutationValidator/tasks/mutation_validator_preprocess/inputs
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1.applyOrElse(StandardAsyncExecutionActor.scala:106)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1.applyOrElse(StandardAsyncExecutionActor.scala:105)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at scala.util.Failure.recoverWith(Try.scala:203)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1.apply(StandardAsyncExecutionActor.scala:105)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1.apply(StandardAsyncExecutionActor.scala:105)
    at cromwell.backend.wdl.Command$.instantiate(Command.scala:27)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.instantiatedCommand(StandardAsyncExecutionActor.scala:198)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.commandScriptContents(StandardAsyncExecutionActor.scala:170)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor$class.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:136)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor$class.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor$class.execute(SharedFileSystemAsyncJobExecutionActor.scala:123)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$executeAsync$1.apply(StandardAsyncExecutionActor.scala:247)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$executeAsync$1.apply(StandardAsyncExecutionActor.scala:247)
    at scala.util.Try$.apply(Try.scala:192)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.executeAsync(StandardAsyncExecutionActor.scala:247)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.executeOrRecover(StandardAsyncExecutionActor.scala:512)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover$1.apply(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover$1.apply(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$class.withRetry(AsyncBackendJobExecutionActor.scala:52)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$class.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:80)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:112)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
    [2017-03-24 03:45:51,32] [error] WorkflowManagerActor Workflow 356bc3b2-d4bc-4126-b4e6-4144e0fd942a failed (during ExecutingWorkflowState): :
    :
    Could not localize -> /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess:
    doesn't exists
    /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess -> /opt/MutationValidator/tasks/mutation_validator_preprocess: Operation not permitted
    File not found /opt/MutationValidator/tasks/mutation_validator_preprocess/inputs
    cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1$$anon$1: :
    :
    Could not localize -> /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess:
    doesn't exists
    /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/mutation_validator_preprocess_workflow/356bc3b2-d4bc-4126-b4e6-4144e0fd942a/call-mutation_validator_preprocess/inputs/opt/MutationValidator/tasks/mutation_validator_preprocess -> /opt/MutationValidator/tasks/mutation_validator_preprocess: Operation not permitted
    File not found /opt/MutationValidator/tasks/mutation_validator_preprocess/inputs
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1.applyOrElse(StandardAsyncExecutionActor.scala:106)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1$$anonfun$apply$1.applyOrElse(StandardAsyncExecutionActor.scala:105)
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    at scala.util.Failure.recoverWith(Try.scala:203)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1.apply(StandardAsyncExecutionActor.scala:105)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$commandLinePreProcessor$1.apply(StandardAsyncExecutionActor.scala:105)
    at cromwell.backend.wdl.Command$.instantiate(Command.scala:27)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.instantiatedCommand(StandardAsyncExecutionActor.scala:198)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand$lzycompute(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.instantiatedCommand(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.commandScriptContents(StandardAsyncExecutionActor.scala:170)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.commandScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor$class.writeScriptContents(SharedFileSystemAsyncJobExecutionActor.scala:136)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.cromwell$backend$sfs$BackgroundAsyncJobExecutionActor$$super$writeScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.BackgroundAsyncJobExecutionActor$class.writeScriptContents(BackgroundAsyncJobExecutionActor.scala:11)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.writeScriptContents(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.sfs.SharedFileSystemAsyncJobExecutionActor$class.execute(SharedFileSystemAsyncJobExecutionActor.scala:123)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.execute(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$executeAsync$1.apply(StandardAsyncExecutionActor.scala:247)
    at cromwell.backend.standard.StandardAsyncExecutionActor$$anonfun$executeAsync$1.apply(StandardAsyncExecutionActor.scala:247)
    at scala.util.Try$.apply(Try.scala:192)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.executeAsync(StandardAsyncExecutionActor.scala:247)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeAsync(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.standard.StandardAsyncExecutionActor$class.executeOrRecover(StandardAsyncExecutionActor.scala:512)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.executeOrRecover(ConfigAsyncJobExecutionActor.scala:112)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover$1.apply(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover$1.apply(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.core.retry.Retry$.withRetry(Retry.scala:37)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$class.withRetry(AsyncBackendJobExecutionActor.scala:52)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$class.cromwell$backend$async$AsyncBackendJobExecutionActor$$robustExecuteOrRecover(AsyncBackendJobExecutionActor.scala:56)
    at cromwell.backend.async.AsyncBackendJobExecutionActor$$anonfun$receive$1.applyOrElse(AsyncBackendJobExecutionActor.scala:80)
    at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
    at akka.actor.Actor$class.aroundReceive(Actor.scala:496)
    at cromwell.backend.impl.sfs.config.BackgroundConfigAsyncJobExecutionActor.aroundReceive(ConfigAsyncJobExecutionActor.scala:112)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

    [2017-03-24 03:45:51,32] [info] WorkflowManagerActor WorkflowActor-356bc3b2-d4bc-4126-b4e6-4144e0fd942a is in a terminal state: WorkflowFailedState
    [2017-03-24 03:45:51,32] [info] Message [cromwell.subworkflowstore.SubWorkflowStoreActor$SubWorkflowStoreCompleteSuccess] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/$b#-30197880] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor/WorkflowActor-356bc3b2-d4bc-4126-b4e6-4144e0fd942a#954935932] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
    [2017-03-24 03:45:53,95] [info] SingleWorkflowRunnerActor workflow finished with status 'Failed'.
    [2017-03-24 03:45:54,06] [info] SingleWorkflowRunnerActor writing metadata to /opt/MutationValidator/tasks/mutation_validator_preprocess/cromwell-executions/run_metadata__2017-03-24__03-45-31.json
    Workflow 356bc3b2-d4bc-4126-b4e6-4144e0fd942a transitioned to state Failed
    [2017-03-24 03:45:54,11] [info] WorkflowManagerActor: Received shutdown signal.
    [2017-03-24 03:45:54,11] [info] Message [cromwell.engine.workflow.WorkflowManagerActor$AbortAllWorkflowsCommand$] from Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor#1813019509] to Actor[akka://cromwell-system/user/SingleWorkflowRunnerActor/WorkflowManagerActor#1813019509] was not delivered. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
    packet_write_wait: Connection to 104.196.26.129 port 22: Broken pipe
    ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
    See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.

  • birgerbirger Member, Broadie, CGA-mod ✭✭✭

    I discussed this with @esalinas and Gordon. We all agree you should try removing from the input json file the elements corresponding to the optional files that you are not providing as input.

  • ChipChip 415M 4053Member, Broadie

    That worked (at least something else failed later...).
    Thanks

  • esalinasesalinas BroadMember, Broadie ✭✭✭

    @Chip I'm glad it worked!!

Sign In or Register to comment.