Ever wish you could automatically remove your unwanted output files from a submission without having to manually review them? If so, take this two minute survey and tell us more.
Latest Release: 1/17/19
Release Notes can be found here.

FISS within a wdl task - Application Default Credentials

Hi-

I have a wdl task that uses FISS python bindings to access the workspace data model. While this works fine locally, I receive a credential error when running in Firecloud:

Traceback (most recent call last):
  File "/cromwell_root/fc-adaae650-a458-4c56-8a55-d96fa463a5c6/fdfd7400-ad04-4c26-81ed-7b2d6cf65723/w/2c60c47a-86f1-4a70-bcac-fef65af20fbe/call-getScript/cohortToSamples.py", line 12, in <module>
    samples = fiss.fapi.get_entities('topmed-shared','topmed-shared', 'sample').json()
  File "/usr/local/lib/python2.7/site-packages/firecloud/api.py", line 193, in get_entities
    headers = _fiss_access_headers()
  File "/usr/local/lib/python2.7/site-packages/firecloud/api.py", line 34, in _fiss_access_headers
    credentials = GoogleCredentials.get_application_default()
  File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1288, in get_application_default
    return GoogleCredentials._get_implicit_credentials()
  File "/usr/local/lib/python2.7/site-packages/oauth2client/client.py", line 1278, in _get_implicit_credentials
    raise ApplicationDefaultCredentialsError(ADC_HELP_MSG)
oauth2client.client.ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Here's the python script and wdl:

import argparse
parser = argparse.ArgumentParser(description='Get a list of sample IDs for all input cohorts within a firecloud data model and print to a text file, one sample id per line',formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--cohorts', nargs=1, default='', help='common separated list of cohorts that you would like the sample ids from')
parser.add_argument('--outfile_pref', nargs=1, default='', help='label for output file of sample ids')

args = parser.parse_args()
args.cohorts = args.cohorts[0].split(',')
out_file = args.outfile_pref[0]+'.txt'


from firecloud import fiss
samples = fiss.fapi.get_entities('topmed-shared','topmed-shared', 'sample').json()
sample_study_gen = (s['attributes']['participant']['entityName'] for s in samples if s['attributes']['study'] in cohorts)

with open(out_file, 'w') as f:
    for p in sample_study_gen:
        f.write(p+'\n')
task getScript {
    command {
        wget "https://raw.githubusercontent.com/manning-lab/topmed-t2d-glycemia-public/dev/methods/dataModel/cohortToSamples.py"
    }

    runtime {
        docker: "tmajarian/[email protected]:f3402d7cb7c5ea864044b91cfbdea20ebe98fc1536292be657e05056dbe5e3a4"
    }

    output {
        File outscript = "cohortToSamples.py"
    }
}

task runScript {
    String cohorts
    String label
    File script

    command {
        python ${script} --cohorts ${cohorts} --outfile_pref ${label}
    }

    runtime {
        docker: "broadgdac/[email protected]:a65324c8cf1edc769bee3195c798468defacefece3a3d551143706cd412e4c39"
        disks: "local-disk 10 SSD"
        memory: "2G"
    }

    output { 
        File out_file = "${label}.txt"
    }
}

workflow w {
    String these_cohorts
    String this_label

    call getScript

    call runScript { 
        input: cohorts=this_cohorts, label=this_label, script=getScript.outscript
    }

    output {
        File sample_list = runScript.out_file
    }
}

I get why this error exists as far as authentication within each compute node goes but am wondering if there is a known workaround for this problem. It seems reasonable that a job launched from a given workspace should inherit the credentials necessary to access the workspace metadata. Depending on the answer, this might be a feature request.

Thanks

Tagged:

Best Answer

Answers

Sign In or Register to comment.