We've moved!
For WDL questions, see the WDL specification and WDL docs.
For Cromwell questions, see the Cromwell docs and please post any issues on Github.

globbing output directory? know paths?

esalinasesalinas BroadMember, Broadie ✭✭✭

I have in my WDL output a glob of an array of files like this :

    Array[File] ais=glob("*.png")

Then in my output (in the bucket) I see a list file and a "directory" (both having a hash) listing and containing files picked up from the glob.

gsutil cat gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4.list
THCA-BJ-A0Z2-TP-NB.maf.mutation_CCG_histogram.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_coverage.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_histogram.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_orientation_alt_counts.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_orientation_weighted_mutations.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_profile.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_AF.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_counts.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_normalized.png
THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_samples_1-1.png

(here are the PNGs)

wm8b1-75c:lp_glob esalinas$ gsutil ls   gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_CCG_histogram.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_coverage.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_histogram.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_orientation_alt_counts.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_orientation_weighted_mutations.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_profile.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_AF.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_counts.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_normalized.png
gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/glob-fc36854b6867c1581ab159b09dd7e2f4/THCA-BJ-A0Z2-TP-NB.maf.mutation_profile_samples_1-1.png
wm8b1-75c:lp_glob esalinas$ 

An additional output is an HTML file. The HTML file has references to the created PNGs.

wm8b1-75c:lp_glob esalinas$ gsutil cat    gs://fc-9c84e685-79f8-4d84-9e52-640943257a9b/3beb06ee-6f01-4a9d-8bb7-e8eacd214561/lego_plotter_workflow/f525deea-91b9-4365-b31d-2a2a3ae3346d/call-lego_plotter_task/*.html|egrep -m 1   -i '\.png'
<img border="0" src="THCA-BJ-A0Z2-TP-NB.maf.mutation_profile.png" type="image/png" alt="THCA-BJ-A0Z2-TP-NB.maf" width="924" height="604" />
wm8b1-75c:lp_glob esalinas$ 

However, when the HTML is served from the bucket, the PNGS don't show up because the "glob-....." is not in the reference of the IMG. The desired behavior is for the PNGS to show up.

I can see at least two ways to achieve the objective:

1) zip up the HTML and the PNGS and let the viewer deal with the issue
2) have a follow-up task that takes in the HTML and all the arrays of files and then edits the HTML and prepends the src for the images with "glob-fc36854b6867c1581ab159b09dd7e2f4"

My question(s):

1) can anyone see any alternatives where either a) the viewer wouldn't have to deal with the issue or b) does not need a follow-up task (VM) to do a search-and-replace style procedure
2) is there a plan to delocalize globs NOT in a "glob-...." directory but all in the same directory?

Issue · Github
by Geraldine_VdAuwera

Issue Number
1925
State
open
Last Updated
Assignee
Array
Milestone
Array

Answers

Sign In or Register to comment.