timing inconsistencies

esalinasesalinas ✭✭✭BroadMember, Broadie ✭✭✭

I have a WDL task that's the last in a scatter (of width 10) that ran on FC. The GUI shows the information copied/pasted here:

Call #10:
Operation:operations/ENn-3IvdKxjf_PKryqzyxb4BINj4y-ORBSoPcHJvZHVjdGlvblF1ZXVl
Status:Done
Cache Result:Miss
Started:August 11, 2017, 9:48 AM (26 minutes ago)
Ended:August 11, 2017, 9:48 AM (25 minutes ago)
Inputs:Show
Outputs:Show
stdout:Mutect1_Task-9-stdout.log
stderr:Mutect1_Task-9-stderr.log
JES log:Mutect1_Task-9.log

Note that the start/end times give a difference of 0 to 1 minutes .

Note also that the timings from the JES log do show an inconsistency in terms of difference. The end time in the JES log is near the same, but the start time is not. The JES logs have ":25" and ":46" for the start-end times as seen in the "head" and "tail" below. The ":25" (start) is not ":48" as shown in the GUI above. The time difference in the JES log is about 20 minutes (46-25), but the GUI is 0 minutes.

wm8b1-75c:~ esalinas$ gsutil cat gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9.log 2>/dev/null |head -1 2>/dev/null
2017/08/11 13:25:11 I: Switching to status: pulling-image
wm8b1-75c:~ esalinas$ gsutil cat gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9.log 2>/dev/null |tail -1 2>/dev/null
2017/08/11 13:46:05 I: Done copying files.
wm8b1-75c:~ esalinas$ 

Answers

  • KateNKateN admin Cambridge, MAMember, Broadie, Moderator admin

    Could you please share the full JES log, including the content between the head & tail for that particular shard?

  • esalinasesalinas ✭✭✭ BroadMember, Broadie ✭✭✭

    @KateN I ommited that because I wanted to focus on the timings I thought most relevant to the post. By request, here is the whole log:

    wm8b1-75c:m2_up esalinas$ gsutil cat gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9.log
    2017/08/11 13:25:11 I: Switching to status: pulling-image
    2017/08/11 13:25:11 I: Calling SetOperationStatus(pulling-image)
    2017/08/11 13:25:11 I: SetOperationStatus(pulling-image) succeeded
    2017/08/11 13:25:11 I: Writing new Docker configuration file
    2017/08/11 13:25:11 I: Pulling image "broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb"
    2017/08/11 13:26:12 I: Pulled image "broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb" successfully.
    2017/08/11 13:26:12 I: Switching to status: localizing-files
    2017/08/11 13:26:12 I: Calling SetOperationStatus(localizing-files)
    2017/08/11 13:26:12 I: SetOperationStatus(localizing-files) succeeded
    2017/08/11 13:26:12 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam.
    2017/08/11 13:26:12 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam
    2017/08/11 13:28:04 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list.
    2017/08/11 13:28:04 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list
    2017/08/11 13:28:07 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf.
    2017/08/11 13:28:07 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf
    2017/08/11 13:30:32 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai.
    2017/08/11 13:30:32 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai
    2017/08/11 13:30:34 I: Docker file /cromwell_root/firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt maps to host location /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt.
    2017/08/11 13:30:34 I: Running command: sudo gsutil -q -m cp gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt /mnt/local-disk/firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt
    2017/08/11 13:30:35 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf.
    2017/08/11 13:30:35 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf
    2017/08/11 13:30:35 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam.
    2017/08/11 13:30:35 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam
    2017/08/11 13:32:34 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.
    2017/08/11 13:32:34 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta
    2017/08/11 13:33:16 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai.
    2017/08/11 13:33:16 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai
    2017/08/11 13:33:18 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai.
    2017/08/11 13:33:18 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai
    2017/08/11 13:33:19 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf.
    2017/08/11 13:33:19 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf
    2017/08/11 13:33:23 I: Docker file /cromwell_root/exec.sh maps to host location /mnt/local-disk/exec.sh.
    2017/08/11 13:33:23 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/exec.sh /mnt/local-disk/exec.sh
    2017/08/11 13:33:25 I: Docker file /cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict maps to host location /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict.
    2017/08/11 13:33:25 I: Running command: sudo gsutil -q -m cp gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict /mnt/local-disk/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict
    2017/08/11 13:33:25 I: Done copying files.
    2017/08/11 13:33:25 I: Switching to status: running-docker
    2017/08/11 13:33:25 I: Calling SetOperationStatus(running-docker)
    2017/08/11 13:33:25 I: SetOperationStatus(running-docker) succeeded
    2017/08/11 13:33:25 I: Running command: iptables -I FORWARD -d metadata.google.internal -p tcp --dport 80 -j DROP
    2017/08/11 13:33:25 I: Setting these data volumes on the docker container: [-v /tmp/ggp-634020541:/tmp/ggp-634020541 -v /mnt/local-disk:/cromwell_root]
    2017/08/11 13:33:25 I: Running command: docker run -v /tmp/ggp-634020541:/tmp/ggp-634020541 -v /mnt/local-disk:/cromwell_root -e Mutect1_Task.cosmicVCF-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf -e __extra_config_gcs_path=gs://cromwell-auth-nci-esalinas-bi-org/dccb19c9-7e0f-4d94-946b-f0127c1794a3_auth.json -e a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt=/cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt -e Mutect1_Task.refFastaIdx-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai -e Mutect1_Task.dbSNPVCF-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf -e exec=/cromwell_root/exec.sh -e Mutect1_Task.refFasta-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta -e Mutect1_Task.normalBamIdx-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai -e a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt=/cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt -e Mutect1_Task.tumorBamIdx-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai -e Mutect1_Task.refFastaDict-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict -e Mutect1_Task-9-rc.txt=/cromwell_root/Mutect1_Task-9-rc.txt -e Mutect1_Task.mutectIntervals-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list -e Mutect1_Task.readGroupBlackList-0=/cromwell_root/firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt -e a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt=/cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt -e Mutect1_Task.normalBam-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam -e Mutect1_Task.tumorBam-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam -e Mutect1_Task.normalPanel-0=/cromwell_root/fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb /tmp/ggp-634020541
    2017/08/11 13:45:59 I: Switching to status: delocalizing-files
    2017/08/11 13:45:59 I: Calling SetOperationStatus(delocalizing-files)
    2017/08/11 13:45:59 I: SetOperationStatus(delocalizing-files) succeeded
    2017/08/11 13:45:59 I: Docker file /cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt maps to host location /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt.
    2017/08/11 13:45:59 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt
    2017/08/11 13:46:01 I: Deleting log file
    2017/08/11 13:46:01 I: Running command: sudo rm -f /var/log/google-genomics/out.log
    2017/08/11 13:46:01 I: Switching to status: copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt"
    2017/08/11 13:46:01 I: Calling SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt")
    2017/08/11 13:46:01 I: SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt") succeeded
    2017/08/11 13:46:01 I: Docker file /cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt maps to host location /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt.
    2017/08/11 13:46:01 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt
    2017/08/11 13:46:02 I: Deleting log file
    2017/08/11 13:46:02 I: Running command: sudo rm -f /var/log/google-genomics/out.log
    2017/08/11 13:46:02 I: Switching to status: copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt"
    2017/08/11 13:46:02 I: Calling SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt")
    2017/08/11 13:46:02 I: SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt") succeeded
    2017/08/11 13:46:02 I: Docker file /cromwell_root/Mutect1_Task-9-rc.txt maps to host location /mnt/local-disk/Mutect1_Task-9-rc.txt.
    2017/08/11 13:46:02 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/local-disk/Mutect1_Task-9-rc.txt gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt
    2017/08/11 13:46:04 I: Deleting log file
    2017/08/11 13:46:04 I: Running command: sudo rm -f /var/log/google-genomics/out.log
    2017/08/11 13:46:04 I: Switching to status: copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt"
    2017/08/11 13:46:04 I: Calling SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt")
    2017/08/11 13:46:04 I: SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt") succeeded
    2017/08/11 13:46:04 I: Docker file /cromwell_root/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt maps to host location /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt.
    2017/08/11 13:46:04 I: Running command: sudo gsutil -q -m cp -L /var/log/google-genomics/out.log /mnt/local-disk/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt
    2017/08/11 13:46:05 I: Deleting log file
    2017/08/11 13:46:05 I: Running command: sudo rm -f /var/log/google-genomics/out.log
    2017/08/11 13:46:05 I: Switching to status: copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt"
    2017/08/11 13:46:05 I: Calling SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt")
    2017/08/11 13:46:05 I: SetOperationStatus(copied 1 file(s) to "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt") succeeded
    2017/08/11 13:46:05 I: Done copying files.
    wm8b1-75c:m2_up esalinas$ 
    
    
  • ThibThib ✭✭ CambridgeMember, Broadie, Dev ✭✭

    Hey Eddie,

    can you post the output of gcloud alpha genomics operations describe operations/ENn-3IvdKxjf_PKryqzyxb4BINj4y-ORBSoPcHJvZHVjdGlvblF1ZXVl to see what JES thinks the start / end times are ?

  • esalinasesalinas ✭✭✭ BroadMember, Broadie ✭✭✭

    @Thib by request:

    {
      "done": true,
      "metadata": {
        "@type": "type.googleapis.com/google.genomics.v1.OperationMetadata",
        "clientId": "",
        "createTime": "2017-08-11T13:23:59Z",
        "endTime": "2017-08-11T13:46:06Z",
        "events": [
          {
            "description": "start",
            "startTime": "2017-08-11T13:25:09.857814845Z"
          },
          {
            "description": "pulling-image",
            "startTime": "2017-08-11T13:25:11.096012625Z"
          },
          {
            "description": "localizing-files",
            "startTime": "2017-08-11T13:26:12.149079112Z"
          },
          {
            "description": "running-docker",
            "startTime": "2017-08-11T13:33:25.791717528Z"
          },
          {
            "description": "delocalizing-files",
            "startTime": "2017-08-11T13:45:59.494861094Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt\"",
            "startTime": "2017-08-11T13:46:01.247230653Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt\"",
            "startTime": "2017-08-11T13:46:02.851545226Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt\"",
            "startTime": "2017-08-11T13:46:04.104699977Z"
          },
          {
            "description": "copied 1 file(s) to \"gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt\"",
            "startTime": "2017-08-11T13:46:05.202177253Z"
          },
          {
            "description": "ok",
            "startTime": "2017-08-11T13:46:06.311337005Z"
          }
        ],
        "labels": {
          "cromwell-workflow-id": "cromwell-dccb19c9-7e0f-4d94-946b-f0127c1794a3",
          "wdl-task-name": "mutect1-task"
        },
        "projectId": "nci-esalinas-bi-org",
        "request": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RunPipelineRequest",
          "ephemeralPipeline": {
            "description": "",
            "docker": {
              "cmd": "/bin/bash /cromwell_root/exec.sh",
              "imageName": "broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb"
            },
            "inputParameters": [
              {
                "defaultValue": "",
                "description": "",
                "name": "__extra_config_gcs_path"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam"
                },
                "name": "Mutect1_Task.tumorBam-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai"
                },
                "name": "Mutect1_Task.normalBamIdx-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai"
                },
                "name": "Mutect1_Task.refFastaIdx-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta"
                },
                "name": "Mutect1_Task.refFasta-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf"
                },
                "name": "Mutect1_Task.dbSNPVCF-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai"
                },
                "name": "Mutect1_Task.tumorBamIdx-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam"
                },
                "name": "Mutect1_Task.normalBam-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "exec.sh"
                },
                "name": "exec"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf"
                },
                "name": "Mutect1_Task.normalPanel-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt"
                },
                "name": "Mutect1_Task.readGroupBlackList-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict"
                },
                "name": "Mutect1_Task.refFastaDict-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf"
                },
                "name": "Mutect1_Task.cosmicVCF-0"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list"
                },
                "name": "Mutect1_Task.mutectIntervals-0"
              }
            ],
            "name": "CallingGroup_Workflow",
            "outputParameters": [
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "Mutect1_Task-9-rc.txt"
                },
                "name": "Mutect1_Task-9-rc.txt"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt"
                },
                "name": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt"
                },
                "name": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt"
              },
              {
                "defaultValue": "",
                "description": "",
                "localCopy": {
                  "disk": "local-disk",
                  "path": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt"
                },
                "name": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt"
              }
            ],
            "pipelineId": "",
            "projectId": "nci-esalinas-bi-org",
            "resources": {
              "bootDiskSizeGb": 10,
              "disks": [
                {
                  "autoDelete": false,
                  "mountPoint": "/cromwell_root",
                  "name": "local-disk",
                  "readOnly": false,
                  "sizeGb": 41,
                  "source": "",
                  "type": "PERSISTENT_HDD"
                }
              ],
              "minimumCpuCores": 1,
              "minimumRamGb": 3,
              "noAddress": false,
              "preemptible": true,
              "zones": [
                "us-central1-b",
                "us-central1-c",
                "us-central1-f"
              ]
            }
          },
          "pipelineArgs": {
            "clientId": "",
            "inputs": {
              "Mutect1_Task.cosmicVCF-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf",
              "Mutect1_Task.dbSNPVCF-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf",
              "Mutect1_Task.mutectIntervals-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list",
              "Mutect1_Task.normalBam-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam",
              "Mutect1_Task.normalBamIdx-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai",
              "Mutect1_Task.normalPanel-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf",
              "Mutect1_Task.readGroupBlackList-0": "gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt",
              "Mutect1_Task.refFasta-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta",
              "Mutect1_Task.refFastaDict-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict",
              "Mutect1_Task.refFastaIdx-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai",
              "Mutect1_Task.tumorBam-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam",
              "Mutect1_Task.tumorBamIdx-0": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai",
              "__extra_config_gcs_path": "gs://cromwell-auth-nci-esalinas-bi-org/dccb19c9-7e0f-4d94-946b-f0127c1794a3_auth.json",
              "exec": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/exec.sh"
            },
            "labels": {
              "cromwell-workflow-id": "cromwell-dccb19c9-7e0f-4d94-946b-f0127c1794a3",
              "wdl-task-name": "mutect1-task"
            },
            "logging": {
              "gcsPath": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9.log"
            },
            "outputs": {
              "Mutect1_Task-9-rc.txt": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-rc.txt",
              "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt",
              "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt",
              "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt"
            },
            "projectId": "nci-esalinas-bi-org",
            "resources": {
              "bootDiskSizeGb": 10,
              "disks": [
                {
                  "autoDelete": false,
                  "mountPoint": "",
                  "name": "local-disk",
                  "readOnly": false,
                  "sizeGb": 41,
                  "source": "",
                  "type": "PERSISTENT_HDD"
                }
              ],
              "minimumCpuCores": 1,
              "minimumRamGb": 3,
              "noAddress": false,
              "preemptible": true,
              "zones": [
                "us-central1-b",
                "us-central1-c",
                "us-central1-f"
              ]
            },
            "serviceAccount": {
              "email": "default",
              "scopes": [
                "https://www.googleapis.com/auth/genomics",
                "https://www.googleapis.com/auth/compute"
              ]
            }
          }
        },
        "runtimeMetadata": {
          "@type": "type.googleapis.com/google.genomics.v1alpha2.RuntimeMetadata",
          "computeEngine": {
            "diskNames": [
              "local-disk-13730289323079417439"
            ],
            "instanceName": "ggp-13730289323079417439",
            "machineType": "us-central1-b/n1-standard-1",
            "zone": "us-central1-b"
          }
        },
        "startTime": "2017-08-11T13:24:07Z"
      },
      "name": "operations/ENn-3IvdKxjf_PKryqzyxb4BINj4y-ORBSoPcHJvZHVjdGlvblF1ZXVl"
    }
    wm8b1-75c:workflows esalinas$ 
    
  • ChrisLChrisL admin Cambridge, MAMember, Broadie, Moderator, Dev admin

    Is it possible that FireCloud's Cromwell restarted around that time? If so, that start time could be when Cromwell reconnected to the workflow.

    Does FireCloud give you access to the events in Cromwell's metadata?

  • abaumannabaumann ✭✭✭ Broad DSDEMember, Broadie ✭✭✭

    You can't see it from the UI, but you can access the metadata via the API or via inspecting the JSON returned in the UI

  • esalinasesalinas ✭✭✭ BroadMember, Broadie ✭✭✭

    @abaumann @ChrisL
    I looked in the network tab and extracted by copy/paste the metadata for the task:

     {
          "preemptible": true,
          "executionStatus": "Done",
          "stdout": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-stdout.log",
          "backendStatus": "Success",
          "shardIndex": 9,
          "jes": {
            "endpointUrl": "https://genomics.googleapis.com/",
            "machineType": "us-central1-b/n1-standard-1",
            "googleProject": "broad-dsde-prod",
            "executionBucket": "gs://cromwell-prod/cromwell-executions",
            "zone": "us-central1-b",
            "instanceName": "ggp-13730289323079417439"
          },
          "outputs": {
            "mutect1_pw": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.power.wig.txt",
            "mutect1_cs": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.call_stats.txt",
            "mutect1_cw": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9.MuTect1.coverage.wig.txt"
          },
          "runtimeAttributes": {
            "preemptible": "1",
            "failOnStderr": "false",
            "bootDiskSizeGb": "10",
            "disks": "local-disk 41 HDD",
            "continueOnReturnCode": "0",
            "docker": "broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb",
            "cpu": "1",
            "noAddress": "false",
            "zones": "us-central1-b,us-central1-c,us-central1-f",
            "memory": "3 GB"
          },
          "callCaching": {
            "allowResultReuse": true,
            "hit": false,
            "result": "Cache Miss",
            "hashes": {
              "output count": "ECCBC87E4B5CE2FE28308FD9F2A7BAF3",
              "runtime attribute": {
                "docker": "BC1BE29F443C8328CD84F26939F06B21",
                "continueOnReturnCode": "CFCD208495D565EF66E7DFF9F98764DA",
                "failOnStderr": "68934A3E9455FA72420237EB05902327"
              },
              "output expression": {
                "File mutect1_cw": "7C091221F0A599402CDC228153D9B5BE",
                "File mutect1_pw": "00C7B257268D299953D21DD2A17A7408",
                "File mutect1_cs": "A7FFDC381A22E37B3A91EE12DC99C20C"
              },
              "input count": "98F13708210194C475687BE6106A3B84",
              "backend name": "5BAA79C7C5A573C899A61D342AA00484",
              "command template": "BBC470850A76D156F512E1C0C4E764F6",
              "input": {
                "String ctrlName": "D64237DC8412DD82165A855BFB6A878D",
                "File tumorBam": "9vbMXg==",
                "File tumorBamIdx": "LUWejw==",
                "File readGroupBlackList": "ll6rwA==",
                "File normalBamIdx": "q6gRow==",
                "Float contamFloor": "04817EFD11C15364A6EC239780038862",
                "String pairName": "3DC056496EAFA2745D2AA876B8BE53A1",
                "Float fracContam": "0D16A6CB2CA34D65DA949608A7BC01D1",
                "File refFasta": "CPQf5g==",
                "Int preemptible": "C4CA4238A0B923820DCC509A6F75849B",
                "Int downsampleToCoverage": "B7A782741F667201B54880C925FAEC4B",
                "String caseName": "099570342BCFD547B107DBCFEF0C25CA",
                "File refFastaDict": "/yGKsA==",
                "File dbSNPVCF": "RqUhnw==",
                "File mutectIntervals": "69zrBQ==",
                "File normalPanel": "iPyxFg==",
                "File refFastaIdx": "xQE6Ig==",
                "Int diskGB": "3416A75F4CEA9109507CACD8E2F2AEFC",
                "File cosmicVCF": "bgFFew==",
                "File normalBam": "GifcBA=="
              }
            },
            "effectiveCallCachingMode": "ReadAndWriteCache"
          },
          "inputs": {
            "diskGB": 41,
            "caseName": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99",
            "refFastaDict": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.dict",
            "refFastaIdx": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta.fai",
            "ctrlName": "eb394344-96b4-4f36-b639-0b4b4728eaa9",
            "preemptible": 1,
            "normalBam": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bam",
            "mutectIntervals": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-CallSomaticMutations_131_Prepare_Task/attempt-2/glob-bcd274f6f6e2bd6e474e81d4a4949f55/split_base.10.interval_list",
            "fracContam": 0.02,
            "readGroupBlackList": "gs://firecloud-tcga-open-access/tutorial/reference/read_group_blacklist.v1.txt",
            "normalPanel": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/hg38_kgenomes_dummy.vcf",
            "cosmicVCF": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/CosmicCodingMuts.vcf.chr.vcf",
            "tumorBam": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bam",
            "downsampleToCoverage": 10000,
            "normalBamIdx": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/2b3fdc12-38c3-471f-a495-38b3cc71587a/gdc_bam_downloader_workflow/f425b09b-089e-4cc1-a09a-227bd3e6f807/call-gdc_bam_downloader/58661e9d-4448-4632-a6cf-f8f4bd9ebf3a/C347.TCGA-44-2666-10A-01D-A271-08.5_gdc_realn.bai",
            "contamFloor": 0.01,
            "tumorBamIdx": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/182be849-187f-4655-b1c0-dbc5700625af/gdc_bam_downloader_workflow/cf5b1875-dab2-4490-a203-d420149acb92/call-gdc_bam_downloader/a70cb15c-b06c-4d46-b153-edd4fb3fddbe/C347.TCGA-44-2666-01B-02D-A271-08.5_gdc_realn.bai",
            "dbSNPVCF": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/Homo_sapiens_assembly38.dbsnp138.vcf.prepped.vcf",
            "pairName": "a0a486e0-a85c-415a-b7c0-399d6b2e6e99_eb394344-96b4-4f36-b639-0b4b4728eaa9",
            "refFasta": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/GRCh38.d1.vd1.fasta"
          },
          "backendLabels": {
            "cromwell-workflow-id": "cromwell-dccb19c9-7e0f-4d94-946b-f0127c1794a3",
            "wdl-task-name": "mutect1-task"
          },
          "returnCode": 0,
          "labels": {
            "wdl-task-name": "Mutect1_Task",
            "cromwell-workflow-id": "cromwell-dccb19c9-7e0f-4d94-946b-f0127c1794a3"
          },
          "jobId": "operations/ENn-3IvdKxjf_PKryqzyxb4BINj4y-ORBSoPcHJvZHVjdGlvblF1ZXVl",
          "backend": "JES",
          "end": "2017-08-11T13:48:28.295Z",
          "dockerImageUsed": "broadinstitute/[email protected]:2285809244dabf3aefe11f4e16e91fe3d541b894f8325e4f6c9c98a8dfc81deb",
          "stderr": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9-stderr.log",
          "callRoot": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9",
          "attempt": 1,
          "executionEvents": [{
            "startTime": "2017-08-11T13:24:07Z",
            "description": "initializing VM",
            "endTime": "2017-08-11T13:25:09.857814845Z"
          }, {
            "startTime": "2017-08-11T13:46:06.311337005Z",
            "description": "ok",
            "endTime": "2017-08-11T13:46:06.311337005Z"
          }, {
            "startTime": "2017-08-11T13:46:06.311337005Z",
            "description": "cromwell poll interval",
            "endTime": "2017-08-11T13:48:25.355Z"
          }, {
            "startTime": "2017-08-11T13:45:59.494861094Z",
            "description": "delocalizing-files",
            "endTime": "2017-08-11T13:46:06.311337005Z"
          }, {
            "startTime": "2017-08-11T13:48:23.254Z",
            "description": "CheckingJobStore",
            "endTime": "2017-08-11T13:48:23.262Z"
          }, {
            "startTime": "2017-08-11T13:48:25.355Z",
            "description": "UpdatingCallCache",
            "endTime": "2017-08-11T13:48:27.339Z"
          }, {
            "startTime": "2017-08-11T13:33:25.791717528Z",
            "description": "running-docker",
            "endTime": "2017-08-11T13:45:59.494861094Z"
          }, {
            "startTime": "2017-08-11T13:26:12.149079112Z",
            "description": "localizing-files",
            "endTime": "2017-08-11T13:33:25.791717528Z"
          }, {
            "startTime": "2017-08-11T13:25:09.857814845Z",
            "description": "start",
            "endTime": "2017-08-11T13:25:11.096012625Z"
          }, {
            "startTime": "2017-08-11T13:48:23.322Z",
            "description": "CheckingCallCache",
            "endTime": "2017-08-11T13:48:23.334Z"
          }, {
            "startTime": "2017-08-11T13:48:27.339Z",
            "description": "UpdatingJobStore",
            "endTime": "2017-08-11T13:48:28.295Z"
          }, {
            "startTime": "2017-08-11T13:48:23.254Z",
            "description": "Pending",
            "endTime": "2017-08-11T13:48:23.254Z"
          }, {
            "startTime": "2017-08-11T13:48:23.262Z",
            "description": "CheckingCacheEntryExistence",
            "endTime": "2017-08-11T13:48:23.267Z"
          }, {
            "startTime": "2017-08-11T13:48:23.254Z",
            "description": "RequestingExecutionToken",
            "endTime": "2017-08-11T13:48:23.254Z"
          }, {
            "startTime": "2017-08-11T13:23:59Z",
            "description": "waiting for quota",
            "endTime": "2017-08-11T13:24:07Z"
          }, {
            "startTime": "2017-08-11T13:25:11.096012625Z",
            "description": "pulling-image",
            "endTime": "2017-08-11T13:26:12.149079112Z"
          }, {
            "startTime": "2017-08-11T13:48:23.267Z",
            "description": "PreparingJob",
            "endTime": "2017-08-11T13:48:23.322Z"
          }, {
            "startTime": "2017-08-11T13:48:23.334Z",
            "description": "RunningJob",
            "endTime": "2017-08-11T13:23:59Z"
          }],
          "backendLogs": {
            "log": "gs://fc-9c9d440a-1b11-4757-bf46-3e7ff46805b8/27a2e45a-cee2-4e91-8d22-8e65054657b2/CallingGroup_Workflow/dccb19c9-7e0f-4d94-946b-f0127c1794a3/call-Mutect1_Task/shard-9/Mutect1_Task-9.log"
          },
          "start": "2017-08-11T13:48:23.253Z"
        }],
    
    
    
    
Sign In or Register to comment.