LATEST RELEASE: July 17, 2018
Release Notes can be found here.

seemingly whimsical dbGAP linkage causes workflow failures?

esalinasesalinas BroadMember, Broadie

my dbGAP access has seemingly been whimsical. Going out unexpectedly.
See also http://gatkforums.broadinstitute.org/firecloud/discussion/8894/cannot-access-controlled-access-workspaces-on-firecloud#latest

I started workflows in FC in a controlled access workspace. Then, in the middle of the day, my dbGAP link was lost. It was maybe an hour or two or so ago. After I link my dbGAP back (by going to my profile page and several times trying to link) my linkage is restored I find a workflow failed!

Two scatter-tasks of a 25-way scatter failed. In the terminal transcript below you can see how they both started having permission denied errors at around the same time 18:04!

wm8b1-75c:dm esalinas$ for LOG in `gsutil ls gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/**|ggrep -Pi 'log$'|ggrep -Piv 'std'   `; do echo $LOG ; gsutil cat $LOG | egrep -ic 'forbidden' ;  gsutil cat $LOG | egrep -C 5 -m 3 -i 'forbidden' ;      done ;
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-0/Mutect1_Task-0.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-1/Mutect1_Task-1.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-10/Mutect1_Task-10.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-11/Mutect1_Task-11.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-12/Mutect1_Task-12.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-13/Mutect1_Task-13.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-14/Mutect1_Task-14.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-15/Mutect1_Task-15.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-16/Mutect1_Task-16.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-17/Mutect1_Task-17.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-18/Mutect1_Task-18.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-19/Mutect1_Task-19.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-2/Mutect1_Task-2.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/Mutect1_Task-20.log
632
2017/01/26 17:54:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 17:59:57 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 17:59:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:57 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:57 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-20/
2017/01/26 18:04:58 E: command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-21/Mutect1_Task-21.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-22/Mutect1_Task-22.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-23/Mutect1_Task-23.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/Mutect1_Task-24.log
476
2017/01/26 17:54:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 17:59:46 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 17:59:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:46 I: Copying /var/log/google-genomics/*.log to gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:46 I: Running command: sudo gsutil -h Content-type:text/plain -q -m cp /var/log/google-genomics/*.log gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/
2017/01/26 18:04:47 E: command failed: AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
AccessDeniedException: 403 Forbidden
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-24/out.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-3/Mutect1_Task-3.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-4/Mutect1_Task-4.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-5/Mutect1_Task-5.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-6/Mutect1_Task-6.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-7/Mutect1_Task-7.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-8/Mutect1_Task-8.log
0
gs://fc-6713bc28-84a3-4309-adc1-f4d8fb7be7d0/337af052-c653-48ce-9c7b-bd29174bc20f/CallingGroup_Workflow/ea246848-2a7c-4cca-bb7e-8e852fc90cda/call-Mutect1_Task/shard-9/Mutect1_Task-9.log
0
wm8b1-75c:dm esalinas$ 

I noticed previous workflows (see http://gatkforums.broadinstitute.org/firecloud/discussion/8907/workflow-wf-running-in-controlled-access-ca-workspace-fail-if-ca-turned-off-during-wf-execution#latest ) also seemingly have permission issue at the same time-of-day (near 18:00) but two days before (on the 24, Tuesday, not 26th, today).

What's going on? The seemingly whimsical linkage makes workflow success harder to achieve

Sign In or Register to comment.