To celebrate the release of GATK 4.0, we are giving away free credits for running the GATK4 Best Practices pipelines in FireCloud, our secure online analysis portal. It’s first come first serve, so sign up now to claim your free credits worth $250. Sponsored by Google Cloud. Learn more at https://software.broadinstitute.org/firecloud/documentation/freecredits

read_tsv() with a path to Google Bucket as an argument

EgorEgor Saint PetersburgMember, Broadie

Hello everyone,
I use the read_tsv() function to read an Array[Array[String]] from a .tsv file located in Google Storage bucket. This how the related part of our WDL script looks like:

   File inputSamplesFile 
   Array[Array[String]] inputSamples = read_tsv(inputSamplesFile)

It works fine in Firecloud, where the inputSamplesFile is passed as a workspace attribute and then as an input parameter to a method. But when my colleague tried to use Cromwell JES and the path to the file was written in the JSON input file, here is what happened:

java.lang.IllegalArgumentException: Could not find suitable filesystem among Default to parse gs://bucket-name/folder-name/file-name.tsv

If the same .tsv file is placed next to the Cromwell jar and the path to it is written in the JSON input file, all is working well. However, we would really like to be able to place the .tsv to Google Cloud Storage and use it with Cromwell JES. So, my questions are:
1) What might be causing the exception?
2) How can it be fixed?

Thank you!

Tagged:

Answers

  • RuchiRuchi Member, Broadie, Dev
    edited May 2017

    Hello,

    The reason Cromwell is giving this exception is because of the credentials provided to the Jes Backend in this specific instance of Cromwell. When your colleague set up their own personal Cromwell Jes, they are required to define GCS credentails to be used by the Jes backend and the Cromwell engine.

    Can they confirm they have some google auth scheme defined for both these configuration keys below?

    engine.filesystems.gcs.auth = "application-default" 
    backend.providers.Jes.config.filesystems.gcs.auth = "application-default"
    }
    
  • EgorEgor Saint PetersburgMember, Broadie

    Thank you, Ruchi. We tried to apply your suggestion, but unfortunately, it does not work for us.
    We used the following configuration (sensitive information replaced with ***):

    "google {
    
      application-name = "cromwell"
    
      auths = [
         {
           name = "application-default"
           scheme = "application_default"
         },
        {
          name = "user-via-refresh"
          scheme = "refresh_token"
          client-id = ***
          client-secret = ***
        },
        {
          name = "service-account"
          scheme = "service_account"
          service-account-id = ***
          pem-file =***
        }
      ]
    }"
    
    engine {
      filesystems {
        gcs {
          auth = "service-account"
        }
      }
    } 
    backend.providers.Jes.config.filesystems.gcs.auth = "service-account"
    

    @Ruchi said:

    Hello,

    The reason Cromwell is giving this exception is because of the credentials provided to the Jes Backend in this specific instance of Cromwell. When your colleague set up their own personal Cromwell Jes, they are required to define GCS credentails to be used by the Jes backend and the Cromwell engine.

    Can they confirm they have some google auth scheme defined for both these configuration keys below?

    engine.filesystems.gcs.auth = "application-default" 
    backend.providers.Jes.config.filesystems.gcs.auth = "application-default"
    }
    
  • RuchiRuchi Member, Broadie, Dev

    A few more questions for you:

    1. Is Cromwell giving the exact same error with the changes to your filesystems configuration?
    2. Have you confirmed that the service-account you're using has access to the tsv file?
    3. Have you tried running Cromwell JES with alternate credentials, such as application-default?

    Thank you for your patience!

  • RuchiRuchi Member, Broadie, Dev

    @Egor Are you still facing the same problem? Please let us know if you require further assistance with this issue.

  • EgorEgor Saint PetersburgMember, Broadie

    @Ruchi said:
    @Egor Are you still facing the same problem? Please let us know if you require further assistance with this issue.

    Sorry, we moved to other issues for a few days. Thank you for your suggestions, we will check them and give you an answer tomorrow.

Sign In or Register to comment.