successfully ran wdl locally, but get this error on cromshell/firecloud: 'sh: -q: unknown operand'

bw2bw2 Member, Broadie
edited December 2018 in Ask the Cromwell + WDL Team

I'm just starting to use wdl/firecloud, and have things working locally, but am having trouble getting it to run successfully on cromshell/firecloud.
My wdl file is:

task gangstr {
    File ref_fasta
    File ref_fasta_fai
    File input_bam
    File input_bam_bai
    File repeat_regions_bed
    String output_prefix

    command {
        GangSTR --ref ${ref_fasta} --bam ${input_bam} --regions ${repeat_regions_bed} --out ${output_prefix}
    }

    output {
        File output_vcf="${output_prefix}.vcf"
    }

    runtime {
        docker: "weisburd/[email protected]:0d97d600e5b8eb0350ddaf4ee4d6b9433e101223e63620e7378ed3a6cb737f54"
    }
}

workflow GangSTR {
    File ref_fasta
    File ref_fasta_fai
    File input_bam
    File input_bam_bai
    File repeat_regions_bed
    String output_prefix

    call gangstr {
        input:
            ref_fasta=ref_fasta,
            ref_fasta_fai=ref_fasta_fai,
            input_bam=input_bam,
            input_bam_bai=input_bam_bai,
            repeat_regions_bed=repeat_regions_bed,
            output_prefix=output_prefix
    }
}

(from: https://github.com/bw2/str-callers/blob/master/wdl/gangstr/gangstr.wdl)

The docker container is: https://hub.docker.com/r/weisburd/gangstr/dockerfile

java -jar ~/code/cromwell/womtool-36.jar validate gangstr.wdl validates successfully.

When I run locally with
java -jar ~/code/cromwell/cromwell-36.jar run gangstr.wdl -i local-inputs-chr22-minimal.json
it completes successfully, but if I submit with
cromshell submit ./gangstr.wdl gcs-inputs-chr22-minimal.json

the workflow fails with:

"message": "Task GangSTR.gangstr:NA:1 failed. The job was stopped before the command finished. PAPI error code 9. Execution failed: action 11: unexpected exit status 1 was not ignored\n[UserAction] Unexpected exit status 1 while running \"/bin/bash /cromwell_root/script\": [GangSTR-1.4] ERROR: Unnecessary leftover arguments\n"

The input configs are - gcs-inputs-chr22-minimal.json:

{
  "GangSTR.ref_fasta": "gs://broad-dsp-spec-ops/scratch/weisburd/ref/GRCh37/hg19.fa",
  "GangSTR.ref_fasta_fai": "gs://broad-dsp-spec-ops/scratch/weisburd/ref/GRCh37/hg19.fa.fai",
  "GangSTR.repeat_regions_bed": "gs://broad-dsp-spec-ops/scratch/weisburd/ref/GRCh37/repeats/chr22_only/gangstr/hg19_ver8_without_chr_prefix.bed",
  "GangSTR.input_bam": "gs://broad-dsp-spec-ops/scratch/weisburd/data/GRCh37/syndip/genomes/chr22_only/CHM1_CHM13_2_chr22_minimal.bam",
  "GangSTR.input_bam_bai": "gs://broad-dsp-spec-ops/scratch/weisburd/data/GRCh37/syndip/genomes/chr22_only/CHM1_CHM13_2_chr22_minimal.bam.bai",
  "GangSTR.output_prefix": "CHM1_CHM13_2_chr22_minimal_output"

(from https://github.com/bw2/str-callers/blob/master/wdl/gangstr/gcs-inputs-chr22-minimal.json)

and local-inputs-chr22-minimal.json:

{
  "GangSTR.ref_fasta": "/Users/weisburd/project1/ref/GRCh37/hg19.fa",
  "GangSTR.ref_fasta_fai": "/Users/weisburd/project1/ref/GRCh37/hg19.fa.fai",
  "GangSTR.repeat_regions_bed": "/Users/weisburd/project1/ref/GRCh37/repeats/chr22_only/gangstr/hg19_ver8_without_chr_prefix.bed",
  "GangSTR.input_bam": "/Users/weisburd/project1/data/GRCh37/syndip/genomes/chr22_only/CHM1_CHM13_2_chr22_minimal.bam",
  "GangSTR.input_bam_bai": "/Users/weisburd/project1/data/GRCh37/syndip/genomes/chr22_only/CHM1_CHM13_2_chr22_minimal.bam.bai",
  "GangSTR.output_prefix": "CHM1_CHM13_2_chr22_minimal_output"
}

(from https://github.com/bw2/str-callers/blob/master/wdl/gangstr/local-inputs-chr22-minimal.json)

Also,

 $ gs cat gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/gangstr.log
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand
[GangSTR-1.4] ERROR: Unnecessary leftover arguments
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand
sh: -q: unknown operand

and

$ gs cat gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/script
#!/bin/bash

cd /cromwell_root
tmpDir=$(mkdir -p "/cromwell_root/tmp.d4a7e6e9" && echo "/cromwell_root/tmp.d4a7e6e9")
chmod 777 "$tmpDir"
export _JAVA_OPTIONS=-Djava.io.tmpdir="$tmpDir"
export TMPDIR="$tmpDir"
export HOME="$HOME"
(
cd /cromwell_root

)
out0d3ee78e="${tmpDir}/out.$$" err0d3ee78e="${tmpDir}/err.$$"
mkfifo "$out0d3ee78e" "$err0d3ee78e"
trap 'rm "$out0d3ee78e" "$err0d3ee78e"' EXIT
tee '/cromwell_root/stdout' < "$out0d3ee78e" &
tee '/cromwell_root/stderr' < "$err0d3ee78e" >&2 &
(
cd /cromwell_root


GangSTR --ref /cromwell_root/broad-dsp-spec-ops/scratch/weisburd/ref/GRCh37/hg19.fa --bam /cromwell_root/broad-dsp-spec-ops/scratch/weisburd/data/GRCh37/syndip/genomes/chr22_only/CHM1_CHM13_2_chr22_minimal.bam --regions /cromwell_root/broad-dsp-spec-ops/scratch/weisburd/ref/GRCh37/repeats/chr22_only/gangstr/hg19_ver8_without_chr_prefix.bed --out CHM1_CHM13_2_chr22_minimal_output
)  > "$out0d3ee78e" 2> "$err0d3ee78e"
echo $? > /cromwell_root/rc.tmp
(
# add a .file in every empty directory to facilitate directory delocalization on the cloud
cd /cromwell_root
find . -type d -empty -print0 | xargs -0 -I % touch %/.file
)
(
cd /cromwell_root
sync


)
mv /cromwell_root/rc.tmp /cromwell_root/rc

Also, the stdout and stderr files are never generated:

wm4d1-03b:~/project1/wdl/gangstr 716 1 $ gs cat gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/stderr
CommandException: No URLs matched: gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/stderr
wm4d1-03b:~/project1/wdl/gangstr 717 1 $ gs cat gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/stdout
CommandException: No URLs matched: gs://broad-dsde-methods/cromwell-execution-34/GangSTR/0d3ee78e-2291-40f2-a52c-23cf759fc7c5/call-gangstr/stdout

Does anyone know where sh is being used, and where the extra -q arg could be coming from?

.

Tagged:

Answers

  • bw2bw2 Member, Broadie

    Turns out the issue was that
    1) my docker image defined ENTRYPOINT - and removing that got rid of the GangSTR leftover arguments error
    2) a second error than appeared

    BusyBox v1.28.4 (2018-07-17 15:21:40 UTC) multi-call binary.
    
    Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]
    

    and the issue there was that I'm using alpine:3.8 as my docker base image. The script that cromwell runs inside the docker container has

    gs cat gs://broad-dsde-methods/cromwell-execution-34/GangSTR/a4618bef-9c16-41db-8748-013a9e9a6af4/call-gangstr/script
    
    #!/bin/bash
    ...
    echo $? > /cromwell_root/rc.tmp
    (
    # add a .file in every empty directory to facilitate directory delocalization on the cloud
    cd /cromwell_root
    find . -type d -empty -print0 | xargs -0 -I % touch %/.file
    )
    (
    cd /cromwell_root
    sync
    
    
    )
    mv /cromwell_root/rc.tmp /cromwell_root/rc
    

    but the find command in Alpine doesn't support -empty

    So my question is - what's the recommended docker base image for cromwell commands?

  • bw2bw2 Member, Broadie

    I switched to bitnami/minideb:stretch as the docker base image, so now it runs on firecloud.
    cromshell still fails with

    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    [GangSTR-1.4] ERROR: Could not open regions file
    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    sh: -q: unknown operand
    
  • RuchiRuchi Member, Broadie, Moderator, Dev admin

    Hey @bw2

    That's a really good question. I'm struggling to get an exact answer as it's really just that the bash jargon around one's command that needs to work. Most of our internal testing uses ubuntu, but its a good point that we should consistently stick to a specific base requirement.

Sign In or Register to comment.