Update: July 26, 2019
This section of the forum is now closed; we are working on a new support model for WDL that we will share here shortly. For Cromwell-specific issues, see the Cromwell docs and post questions on Github.

Working with Requester Pays S3 buckets

At Fred Hutch we'd like to set up a public AWS S3 bucket containing reference data, analogous to the broad-references bucket in GCP.

We would like to take advantage of the Requester Pays feature in S3, so that requesters pay for the download of objects. That way we can share this data with anyone who may want to use it, without being charged money each time.

The problem is, that to access a Requester Pays bucket, the client needs to send a special header (in the AWS CLI this is done by adding the --request-payer flag) to acknowledge that they understand they will be paying for the download. Without that header, the download will fail.

I am assuming that Cromwell does not send this header by default. Is there a way in the conf file to configure Cromwell to send that header, at least when downloading from a particular bucket? If not, then please consider this a feature request. This would be hugely useful not just to us but to the wider community who would benefit by being able to access this reference data and run their own workflows against it.

Thank you!

Sign In or Register to comment.