Five dollar pipeline returning error 247

OggieMOggieM Member

Hi! I am running your latest commit of the Five dollar pipeline using Genomics Pipelines service, and I am getting consistent fails at the ValidateGVCF task:

[UUID(fd6eedf3)]: Starting BamToGvcf.ValidateGVCF, BamToGvcf.CollectGvcfCallingMetrics 2019-07-05 11:50:08,332 cromwell-system-akka.dispatchers.engine-dispatcher-1082 INFO - Assigned new job execution tokens to the following groups: 239cb578: 2 2019-07-05 11:50:16,682 cromwell-system-akka.dispatchers.backend-dispatcher-1545 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.CollectGvcfCallingMetrics:NA:1]: java -Xms2000m -jar /usr/gitc/picard.jar \ CollectVariantCallingMetrics \ INPUT=/cromwell_root/parkinson-test-bucket/work/WholeGenomeGermlineSingleSample/239cb578-fab3-4393-a2d8-318257f8bf1e/call-BamToGvcf/BamToGvcf/fd6eedf3-ba84-45d7-ac96-40fb5e1ab3c2/call-MergeVCFs/NA12878_PLUMBING.g.vcf.gz \ OUTPUT=NA12878_PLUMBING \ DBSNP=/cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf \ SEQUENCE_DICTIONARY=/cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.dict \ TARGET_INTERVALS=/cromwell_root/broad-references/hg38/v0/wgs_evaluation_regions.hg38.interval_list \ GVCF_INPUT=true 2019-07-05 11:50:31,702 cromwell-system-akka.dispatchers.backend-dispatcher-1695 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.ValidateGVCF:NA:1]: gatk --java-options -Xms3000m \ ValidateVariants \ -V /cromwell_root/parkinson-test-bucket/work/WholeGenomeGermlineSingleSample/239cb578-fab3-4393-a2d8-318257f8bf1e/call-BamToGvcf/BamToGvcf/fd6eedf3-ba84-45d7-ac96-40fb5e1ab3c2/call-MergeVCFs/NA12878_PLUMBING.g.vcf.gz \ -R /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.fasta \ -L /cromwell_root/broad-references/hg38/v0/wgs_calling_regions.hg38.interval_list \ -gvcf \ --validation-type-to-exclude ALLELES \ --dbsnp /cromwell_root/broad-references/hg38/v0/Homo_sapiens_assembly38.dbsnp138.vcf 2019-07-05 11:50:31,758 cromwell-system-akka.dispatchers.backend-dispatcher-1695 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.ValidateGVCF:NA:1]: To comply with GCE custom machine requirements, memory was adjusted from 3500 MB to 3584 MB 2019-07-05 11:50:38,925 cromwell-system-akka.dispatchers.backend-dispatcher-1545 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.ValidateGVCF:NA:1]: job id: projects/parkinson-clinical-trial/operations/3903307879607268101 2019-07-05 11:50:38,925 cromwell-system-akka.dispatchers.backend-dispatcher-1777 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.CollectGvcfCallingMetrics:NA:1]: job id: projects/parkinson-clinical-trial/operations/1813452767225742713 2019-07-05 11:51:09,021 cromwell-system-akka.dispatchers.backend-dispatcher-1699 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.ValidateGVCF:NA:1]: Status change from - to Running 2019-07-05 11:51:09,038 cromwell-system-akka.dispatchers.backend-dispatcher-1699 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.CollectGvcfCallingMetrics:NA:1]: Status change from - to Running 2019-07-05 12:00:08,138 cromwell-system-akka.dispatchers.backend-dispatcher-1945 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.ValidateGVCF:NA:1]: Status change from Running to Success 2019-07-05 12:10:10,801 cromwell-system-akka.dispatchers.backend-dispatcher-1777 INFO - PipelinesApiAsyncBackendJobExecutionActor [UUID(fd6eedf3)BamToGvcf.CollectGvcfCallingMetrics:NA:1]: Status change from Running to Success 2019-07-05 12:10:13,058 cromwell-system-akka.dispatchers.engine-dispatcher-1474 ERROR - WorkflowManagerActor Workflow 239cb578-fab3-4393-a2d8-318257f8bf1e failed (during ExecutingWorkflowState): Job BamToGvcf.ValidateGVCF:NA:1 exited with return code 247 which has not been declared as a valid return code. See 'continueOnReturnCode' runtime attribute for more details.

Seeing that this error can occur when the docker doesn't provide enough RAM, and I have been able to reproduce this by asking for -Xms larger than the Docker limit. But there is currently no limit imposed, and I see no reason why this should occur.

  1. Is it possible that this error is because the app is running out of RAM from a collision with another job? Looking at https://raw.githubusercontent.com/gatk-workflows/five-dollar-genome-analysis-pipeline/1.1.0/tasks/BamToGvcf.wdl and at logs both QC.ValidateGVCF and QC.CollectGvcfCallingMetrics seem to be called in parallel. Could they be at the same isntance and somehow the RAM is less then the sum of the two requirements?

  2. Can I somehow set a flag in google "generic.google-papi.options.json" file to put a minimum of say 8GB to an instance? I tried:

{ "read_from_cache":false, "default_runtime_attributes": { "zones": "us-central1-a us-central1-b us-central1-c us-central1-f", "maxRetries": 1, "memory": "8G" } }

I still see instances smaller than 8GB. Is it overridden by specific values or did I not create the config.json file properly? Is there a way to solve this without editing the files you are hosting online?

Thanks!

Sign In or Register to comment.