JES API worker actor unexpectedly terminated while conducting 1 polls

Hi,
I wrote a custom GTAK germline pipeline base on WDL script from GATK workflows GitHub. I tested the script successfully with subset fastq files. However when I used a original fastq or multiple fastqs, it failed. I also tried with different set of fastqs and returned the same error message (The JES API worker actor Actor[......] unexpectedly terminated while conducting 1 polls. Making a new one...) from non-specific task (depends on input files). Could someone help me to address this problem. Thank you.

2018-07-01 21:34:27,152 cromwell-system-akka.dispatchers.backend-dispatcher-74 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.MarkDuplicates:NA:1]: `java -Dsamjdk.compression_level=5 -Xms4000m -jar /app_tools/picard.jar MarkDuplicates \
  INPUT=/cromwell_root/ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MergeBamAlignment/shard-0/attempt-2/ERR1044158.aligned.unsorted.bam INPUT=/cromwell_root/ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MergeBamAlignment/shard-1/ERR1045260.aligned.unsorted.bam INPUT=/cromwell_root/ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MergeBamAlignment/shard-2/ERR1045261.aligned.unsorted.bam INPUT=/cromwell_root/ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MergeBamAlignment/shard-3/ERR1045262.aligned.unsorted.bam \
  OUTPUT=PRJEB11005.aligned.unsorted.duplicates_marked.bam \
  METRICS_FILE=PRJEB11005.duplicate_metrics \
  VALIDATION_STRINGENCY=SILENT \
  OPTICAL_DUPLICATE_PIXEL_DISTANCE=2500 \
  ASSUME_SORT_ORDER="queryname" \
  CREATE_MD5_FILE=true \
  REMOVE_DUPLICATES=false`
2018-07-01 21:34:35,181 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.MarkDuplicates:NA:1]: job id: operations/EM2sk77FLBjXrrfTzaWsorMBILDR4tGhGyoPcHJvZHVjdGlvblF1ZXVl
2018-07-01 21:34:46,247 cromwell-system-akka.dispatchers.backend-dispatcher-74 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.MarkDuplicates:NA:1]: Status change from - to Running
2018-07-02 00:08:54,842 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.MergeSortedBam:NA:1]: Status change from Running to Success
2018-07-02 03:25:16,548 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.MarkDuplicates:NA:1]: Status change from Running to Success
2018-07-02 03:25:19,137 cromwell-system-akka.dispatchers.engine-dispatcher-57 INFO  - 4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7-SubWorkflowActor-SubWorkflow-workflow_data_processing:-1:1 [UUID(4aa43b8e)]: Starting calls: main.SortAndFixTags:NA:1
2018-07-02 03:25:19,153 cromwell-system-akka.dispatchers.backend-dispatcher-74 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.SortAndFixTags:NA:1]: `java -Dsamjdk.compression_level=5 -Xms4000m -jar /app_tools/picard.jar SortSam \
  INPUT=/cromwell_root/ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MarkDuplicates/PRJEB11005.aligned.unsorted.duplicates_marked.bam \
  OUTPUT=PRJEB11005.aligned.duplicate_marked.sorted.bam \
  SORT_ORDER="coordinate" \
  CREATE_INDEX=true \
  CREATE_MD5_FILE=true \
  MAX_RECORDS_IN_RAM=300000`
2018-07-02 03:25:27,817 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.SortAndFixTags:NA:1]: job id: operations/ELammMjFLBjfhPGct9Gn-0AgsNHi0aEbKg9wcm9kdWN0aW9uUXVldWU
2018-07-02 03:25:38,899 cromwell-system-akka.dispatchers.backend-dispatcher-75 INFO  - JesAsyncBackendJobExecutionActor [UUID(4aa43b8e)main.SortAndFixTags:NA:1]: Status change from - to Running
2018-07-02 06:33:10,055 cromwell-system-akka.dispatchers.backend-dispatcher-80 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$a#661838551] unexpectedly terminated while conducting 1 polls. Making a new one...
2018-07-02 06:33:10,057 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$b#733808041]
2018-07-02 06:33:10,060 cromwell-system-akka.dispatchers.backend-dispatcher-80 ERROR - The JES API worker actor managed to unexpectedly terminate whilst doing absolutely nothing (Polling stopped itself unexpectedly). This is probably a programming error. Making a new one...
java.lang.RuntimeException: Polling stopped itself unexpectedly
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$$anonfun$receive$1.$anonfun$applyOrElse$1(JesApiQueryManager.scala:97)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.onFailure(JesApiQueryManager.scala:196)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$$anonfun$receive$1.applyOrElse(JesApiQueryManager.scala:97)
    at akka.actor.Actor.aroundReceive(Actor.scala:514)
    at akka.actor.Actor.aroundReceive$(Actor.scala:512)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.aroundReceive(JesApiQueryManager.scala:30)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.dungeon.DeathWatch.receivedTerminated(DeathWatch.scala:61)
    at akka.actor.dungeon.DeathWatch.receivedTerminated$(DeathWatch.scala:58)
    at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:370)
    at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:512)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2018-07-02 06:33:10,062 cromwell-system-akka.dispatchers.backend-dispatcher-80 INFO  - watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$c#1931476940]
2018-07-02 06:33:41,097 cromwell-system-akka.dispatchers.backend-dispatcher-87 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$b#733808041] unexpectedly terminated while conducting 1 polls. Making a new one...
2018-07-02 06:33:41,098 cromwell-system-akka.dispatchers.backend-dispatcher-87 INFO  - watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$d#-1462119984]
2018-07-02 06:33:41,099 cromwell-system-akka.dispatchers.backend-dispatcher-87 ERROR - The JES API worker actor managed to unexpectedly terminate whilst doing absolutely nothing (Polling stopped itself unexpectedly). This is probably a programming error. Making a new one...
java.lang.RuntimeException: Polling stopped itself unexpectedly
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$$anonfun$receive$1.$anonfun$applyOrElse$1(JesApiQueryManager.scala:97)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.onFailure(JesApiQueryManager.scala:196)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager$$anonfun$receive$1.applyOrElse(JesApiQueryManager.scala:97)
    at akka.actor.Actor.aroundReceive(Actor.scala:514)
    at akka.actor.Actor.aroundReceive$(Actor.scala:512)
    at cromwell.backend.impl.jes.statuspolling.JesApiQueryManager.aroundReceive(JesApiQueryManager.scala:30)
    at akka.actor.ActorCell.receiveMessage(ActorCell.scala:527)
    at akka.actor.dungeon.DeathWatch.receivedTerminated(DeathWatch.scala:61)
    at akka.actor.dungeon.DeathWatch.receivedTerminated$(DeathWatch.scala:58)
    at akka.actor.ActorCell.receivedTerminated(ActorCell.scala:370)
    at akka.actor.ActorCell.autoReceiveMessage(ActorCell.scala:512)
    at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Best,
Shaomin

Tagged:

Answers

  • whatsnewwhatsnew TWMember

    I found some of error messages are very similar to this issue.
    https://github.com/broadinstitute/cromwell/issues/1965

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hi Shaomin,

    To help debug, it would be helpful to get more details about the operation.
    operation id: operations/ELammMjFLBjfhPGct9Gn-0AgsNHi0aEbKg9wcm9kdWN0aW9uUXVldWU

    The best way to get that answer would be to download the gcloud sdk, login and run:
    gcloud alpha genomics operations describe operations/ELammMjFLBjfhPGct9Gn-0AgsNHi0aEbKg9wcm9kdWN0aW9uUXVldWU

    Would you mind pasting the results here?

    Thanks!
    Ruchi

  • ChrisLChrisL Cambridge, MAMember, Broadie, Moderator, Dev admin

    But remember to double check that there's nothing private/personal in the metadata response before posting :)

  • whatsnewwhatsnew TWMember

    Here's the log.
    Thank you!

    done: true
    metadata:
      '@type': type.googleapis.com/google.genomics.v1.OperationMetadata
      clientId: ''
      createTime: '2018-07-02T03:25:27Z'
      endTime: '2018-07-02T07:31:31Z'
      events:
      - description: start
        startTime: '2018-07-02T03:26:47.621783053Z'
      - description: pulling-image
        startTime: '2018-07-02T03:26:47.621870069Z'
      - description: localizing-files
        startTime: '2018-07-02T03:31:23.492765358Z'
      - description: running-docker
        startTime: '2018-07-02T03:44:12.362406555Z'
      - description: delocalizing-files
        startTime: '2018-07-02T07:06:08.961398700Z'
      - description: copied 1 file(s) to "gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/SortAndFixTags-rc.txt"
        startTime: '2018-07-02T07:06:12.924829970Z'
      - description: copied 1 file(s) to "gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bam"
        startTime: '2018-07-02T07:31:22.132681389Z'
      - description: copied 1 file(s) to "gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bai"
        startTime: '2018-07-02T07:31:26.529307998Z'
      - description: copied 1 file(s) to "gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bam.md5"
        startTime: '2018-07-02T07:31:28.147166101Z'
      - description: ok
        startTime: '2018-07-02T07:31:31.166340122Z'
      labels:
        cromwell-sub-workflow-name: main
        cromwell-workflow-id: cromwell-91444449-0c82-4eb7-914d-eabee58abb1d
        wdl-task-name: sortandfixtags
      projectId: ngs-dev-172107
      request:
        '@type': type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest
        ephemeralPipeline:
          description: ''
          docker:
            cmd: /bin/bash /cromwell_root/exec.sh
            imageName: gcr.io/ngs-dev-172107/[email protected]:4a05a99978307d9ec7bba1e29846cc8c1f6f849bbe528341cb4212e226cc9dcb
          inputParameters:
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MarkDuplicates/PRJEB11005.aligned.unsorted.duplicates_marked.bam
            name: main.SortAndFixTags.input_bam-0
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: exec.sh
            name: exec
          name: main
          outputParameters:
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: SortAndFixTags-rc.txt
            name: SortAndFixTags-rc.txt
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: PRJEB11005.aligned.duplicate_marked.sorted.bam
            name: PRJEB11005.aligned.duplicate_marked.sorted.bam
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: PRJEB11005.aligned.duplicate_marked.sorted.bai
            name: PRJEB11005.aligned.duplicate_marked.sorted.bai
          - defaultValue: ''
            description: ''
            localCopy:
              disk: local-disk
              path: PRJEB11005.aligned.duplicate_marked.sorted.bam.md5
            name: PRJEB11005.aligned.duplicate_marked.sorted.bam.md5
          pipelineId: ''
          projectId: ngs-dev-172107
          resources:
            acceleratorCount: '0'
            acceleratorType: ''
            bootDiskSizeGb: 100
            disks:
            - autoDelete: false
              mountPoint: /cromwell_root
              name: local-disk
              readOnly: false
              sizeGb: 400
              source: ''
              type: PERSISTENT_HDD
            minimumCpuCores: 1
            minimumRamGb: 5
            noAddress: false
            preemptible: false
            zones:
            - us-central1-a
            - us-central1-b
            - us-central1-c
            - us-central1-f
        pipelineArgs:
          clientId: ''
          inputs:
            exec: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/exec.sh
            main.SortAndFixTags.input_bam-0: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-MarkDuplicates/PRJEB11005.aligned.unsorted.duplicates_marked.bam
          labels:
            cromwell-sub-workflow-name: main
            cromwell-workflow-id: cromwell-91444449-0c82-4eb7-914d-eabee58abb1d
            wdl-task-name: sortandfixtags
          logging:
            gcsPath: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/SortAndFixTags.log
          outputs:
            PRJEB11005.aligned.duplicate_marked.sorted.bai: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bai
            PRJEB11005.aligned.duplicate_marked.sorted.bam: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bam
            PRJEB11005.aligned.duplicate_marked.sorted.bam.md5: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/PRJEB11005.aligned.duplicate_marked.sorted.bam.md5
            SortAndFixTags-rc.txt: gs://ngs-dev/rosser/final_run/wgs_germline_hg38_25x/work/main/91444449-0c82-4eb7-914d-eabee58abb1d/call-workflow_data_processing/main/4aa43b8e-6b07-4c7a-ad60-acd9dd570cf7/call-SortAndFixTags/SortAndFixTags-rc.txt
          projectId: ngs-dev-172107
          resources:
            acceleratorCount: '0'
            acceleratorType: ''
            bootDiskSizeGb: 100
            disks:
            - autoDelete: false
              mountPoint: ''
              name: local-disk
              readOnly: false
              sizeGb: 400
              source: ''
              type: PERSISTENT_HDD
            minimumCpuCores: 1
            minimumRamGb: 5
            noAddress: false
            preemptible: false
            zones:
            - us-central1-a
            - us-central1-b
            - us-central1-c
            - us-central1-f
          serviceAccount:
            email: default
            scopes:
            - https://www.googleapis.com/auth/genomics
            - https://www.googleapis.com/auth/compute
      runtimeMetadata:
        '@type': type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata
        computeEngine:
          diskNames:
          - local-disk-4681103184475472479
          instanceName: ggp-4681103184475472479
          machineType: us-central1-f/n1-standard-2
          zone: us-central1-f
      startTime: '2018-07-02T03:25:36Z'
    name: operations/ELammMjFLBjfhPGct9Gn-0AgsNHi0aEbKg9wcm9kdWN0aW9uUXVldWU
    
    
  • whatsnewwhatsnew TWMember

    Any updates? Thanks!

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @whatsnew, totally looks like all the expected outputs were produced and the google operation finished at 2018-07-02T07:31:31Z, but Cromwell had a polling glitch at 2018-07-02 06:33:41,097.

    • Is this error mode deterministic, have you tried re-running the pipeline?
    • Is the Cromwell configured to have Call Caching enabled?
    • May I know where are you running this Cromwell from (locally, GCE instance, etc) and how much memory it has?

    Thanks for your patience!

  • whatsnewwhatsnew TWMember

    Hi Ruchi,

    1 I have tried to re-run the pipeline. The error still existed. Sometimes pipeline can successfully finished if I used a "small fastq" input. But, for the large fastq input, 25x wgs, never success yet. This error is not specifically occurred at the task shown above, it could just occur at any task.

    2 I set read_from_cache to "false" and did not configure call caching. I use the wdl_runner from wdl github.

    3 The cromwell is running on GCE. The wdl_runner memory is 5GB. My configurations are based on https://cloud.google.com/genomics/docs/tutorials/gatk.
    Actually, I also tried using PairedEndSingleSampleWf.gatk4.0.wdl with 25x wgs inputs. But, this error still occurred. The below error messages was from PairedEndSingleSampleWf.gatk4.0.wdl with 25x wgs inputs run.

    Thank you.
    Shaomin.

    2018-07-06 06:30:28,161 cromwell-system-akka.dispatchers.backend-dispatcher-8359 INFO  - JesAsyncBackendJobExecutionActor [UUID(a43bb9a8)PairedEndSingleSampleWorkflow.HaplotypeCaller:27:1]: Status change from Running to Success
    2018-07-06 06:33:24,698 cromwell-system-akka.dispatchers.backend-dispatcher-8384 INFO  - JesAsyncBackendJobExecutionActor [UUID(a43bb9a8)PairedEndSingleSampleWorkflow.HaplotypeCaller:40:1]: Status change from Running to Success
    2018-07-06 06:35:01,865 cromwell-system-akka.dispatchers.backend-dispatcher-8374 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$a#557000009] unexpectedly terminated while conducting 2 polls. Making a new one...
    2018-07-06 06:35:01,875 cromwell-system-akka.dispatchers.backend-dispatcher-8374 INFO  - watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$b#-679864437]
    2018-07-06 06:35:32,900 cromwell-system-akka.dispatchers.backend-dispatcher-8384 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$b#-679864437] unexpectedly terminated while conducting 3 polls. Making a new one...
    2018-07-06 06:35:32,901 cromwell-system-akka.dispatchers.backend-dispatcher-8384 INFO  - watching Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$c#1066035156]
    2018-07-06 06:36:03,940 cromwell-system-akka.dispatchers.backend-dispatcher-8334 ERROR - The JES API worker actor Actor[akka://cromwell-system/user/cromwell-service/$b/$a/$c#1066035156] unexpectedly terminated while conducting 3 polls. Making a new one...
    
  • whatsnewwhatsnew TWMember

    HI, any updates? Thanks

  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @whatsnew,

    Since it seems like the workflow itself runs fine with smaller inputs, its possible the issue here is that Cromwell needs more memory. It may not be possible to give Cromwell more memory from the wdl_runner, but have you tried running Cromwell on a GCE node directly? If you're running the java command, you can give the process more memory as needed:
    https://software.broadinstitute.org/gatk/documentation/article?id=12521

Sign In or Register to comment.