1) Did you compare the coverages before and after BQSR? Did BQSR have a major impact?
2) What was the command you ran? Can you check how much of a difference there is if you lower the quality thresholds?
3) Yes, it would be nice to have a comparison.
4) It is possible that may be necessary after you do some additional testing (if the sequence quality is low, you can probably let them know and hope they can meet the coverage you need).
I think this thread will help.
You can keep track of the status of the fix here.
First, we want to apologize for the upgrade stopping your existing workflows. In Cromwell 34 , the file naming structure was altered just enough to get past the auto-restart we had in place, thus resulting in you having to manually restart your workflows. The transition was not handled as smoothly as we would have hoped, and work is underway to improve things for future possible breaking changes.
In regards to your first question, yes, the runtime attribute error you see is due to the Cromwell 34 upgrade. They introduced some code to ensure that the runtime attributes* which use size units (eg "GB", "KB", etc.) do so using the full number-space-unit format desired for best practices. Unfortunately this was not communicated to you, which it certainly should have been. In either case, you have two options for going forward. If you'd like, you can update your pipelines to the new format, though I understand that this is inconvenient. Alternatively, you can wait for a hotfix. We are working to enable use of either the "1G" or "1 GB" formats, and a release should go out by end of day today.
*To my knowledge, the runtime attributes affected include memory, disk, and gpu
Your last question asks about introducing a feature to allow you to freeze the version of Cromwell you are running in FireCloud. I've been told that this is a very large feature with a number of complexities around it that make it prohibitively difficult to implement. I'm not saying that it would never be implemented, but I do want to set your expectations with it. If you'd like to request that as a feature, please post a feature request in this section.
You could output all intervals you are interested in, then attach them together.
bam files are no feature files. This tool is only meant for vcf or bed files.
Bam files should be indexed automatically if you use them in a GATK tool. You can also use index from samtools to index bam files.
samtools index <file.bam>
The dev team mentioned this could be related to a bug that will be fixed in the next gatk release.
A possible workaround would be adding -AX ReferenceBases in the optional m2_extra_args parameter. This turns off the annotation with the bug, which is only needed when running the new Bayesian orientation bias filter.
Let us know if this fixes the issue.
Edit: Corrected the parameter
It will not cause issues for Cromwell if you put an exit in your command block. I believe your command actually gets run in a subshell so should be good.
According to this similar post, this would be a transient error. Would you please retry the workflow and let us know what happens.
I looked into your specific case and it seems like Cromwell spent a long time the Checking Call Cache state:
Checking Call Cache
There are two main phases in this stage, Cromwell generates hashes for your input files and then searches for a match. Nothing seems to indicate database issues so it's likely not the search for a match that was the bottleneck, it seems to be the file hashing. It seems like the task with 1 file input (cluster_cnmf) took about 4 seconds checking for a call cache while a task with 16 files (report_cluster) took the longest. There's a centralize I/O queue that handles all I/O caching -- my guess is that when Cromwell has more I/O throughput, the I/O jobs spend more time in the queue as there is quota throttling how fast they are completed. I'll bring this back to the team and start having it addressed.
Thank you for bringing this up
FireCloud does not yet support WDL 1.0. If you adjust the syntax to $ instead of ~, it should work.