connection reset by peer, defining memory is not working

afriedenafrieden Cambridge, MAMember

Hi WDL Team,
I am currently trying to run a pipeline using some small data. I have set the memory CLI to 5,10, and 20 but to no avail:

gcloud alpha genomics pipelines run \
--pipeline-file workflows/wdl_pipeline.yaml \
--zones us-central1-f \
--memory 20 \
--logging gs://ccgd-test/aggTest/logging \
--inputs-from-file WDL=/path/to/aggregationLevel.wdl \
--inputs-from-file WORKFLOW_INPUTS=/path/to/aggregationLevel.inputs.test.json \
--inputs-from-file WORKFLOW_OPTIONS=/path/to/PublicPairedSingleSampleWf_170412.options.json \
--inputs WORKSPACE=gs://ccgd-test/aggTest/workspace \
--inputs OUTPUTS=gs://ccgd-test/aggTest/outputs

Getting the following error:

error:
 code: 10
 message: |-
  11: Docker run failed: command failed: 2017-07-28 19:47:17,952 sys_util INFO: CROMWELL->/cromwell/cromwell.jar
  2017-07-28 19:47:17,952 sys_util INFO: CROMWELL_CONF->/cromwell/jes_template.conf
  2017-07-28 19:47:17,956 discovery INFO: URL being requested: GET https://www.googleapis.com/discovery/v1/apis/storage/v1/rest
  2017-07-28 19:47:17,994 discovery INFO: URL being requested: GET https://www.googleapis.com/storage/v1/b/ccgd-test/o?fields=nextPageToken%2Citems%28name%29&prefix=aggTest%2Foutputs&alt=json&maxResults=2
  2017-07-28 19:47:17,994 transport INFO: Attempting refresh to obtain initial access_token
  2017-07-28 19:47:18,089 cromwell_driver INFO: Started Cromwell
  2017-07-28 19:47:18,089 wdl_runner INFO: starting
  2017-07-28 19:47:24,145 cromwell_driver INFO: Failed to connect to Cromwell (attempt 1): ('Connection aborted.', error(104, 'Connection reset by peer'))
  ERROR: Job status from Cromwell was not 'Submitted', instead 'fail'
  . See logs at gs://ccgd-test/aggTest/logging

Logs don't seem to be any more useful. Any thoughts on what is going on?

Answers

  • Geraldine_VdAuweraGeraldine_VdAuwera Cambridge, MAMember, Administrator, Broadie admin

    @afrieden We'd have to see the logs to figure out what's going on here. Can you give us access to the bucket where they're located? Note that if you're at Broad on-site, there are office hours (Wednesdays 2-3pm in Yosemite) where you can talk to a member of the Cromwell/WDL development team, who may be able to help you debug.

  • shleeshlee CambridgeMember, Administrator, Broadie, Moderator admin
    edited October 2017

    I'm getting this same error. Any update on how to fix this? Logs are at gs://shlee-dev/newCNV/bifem_pon.

  • shleeshlee CambridgeMember, Administrator, Broadie, Moderator admin

    I've gone a different route, namely a compute engine VM, so this issue is no longer on my radar.

  • afriedenafrieden Cambridge, MAMember

    @shlee said:
    I've gone a different route, namely a compute engine VM, so this issue is no longer on my radar.

    So you are now moving away from the google genomics pipeline engine and just running cromwell straight on a VM?

  • shleeshlee CambridgeMember, Administrator, Broadie, Moderator admin

    Yes, I am running either WDLs or bash scripts straight up on a VM.

Sign In or Register to comment.