PublicPairedSingleSampleWf_160927 Error while running locally @ SamToFastqAndBwaMem

Hi,

i tried to run your PublicPairedSingleSampleWf locally but run in some trouble...
The pipeline stopps with java.lang.Exception: Call PairedEndSingleSampleWorkflow.SamToFastqAndBwaMem: return code was 1,
When i look at the stderr in call-SamToFastqAndBwaMem execution Folder i find the following lines:
<br /> /bin/bash: line 16: /data/reference/TestData/hiseq2500/NA12878/H06HDADXX130110.1.ATCACGAT.20k_reads.unmerged.bam: No such file or directory<br /> tee: /data/reference/TestData/hiseq2500/NA12878/H06HDADXX130110.1.ATCACGAT.20k_reads.unmerged.bwa.stderr.log: No such file or directory<br />
And also an Error from picard:
<br /> Exception in thread "main" htsjdk.samtools.SAMException: Error in writing fastq file /dev/stdout<br /> at htsjdk.samtools.fastq.BasicFastqWriter.write(BasicFastqWriter.java:68)<br /> at picard.sam.SamToFastq.writeRecord(SamToFastq.java:350)<br /> at picard.sam.SamToFastq.doWork(SamToFastq.java:195)<br /> at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:208)<br /> at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:95)<br /> at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:105)<br />
It seems that SamTOFastq he isnt able to write the output into stdout. But I dunno why, any clue ?

Thanks for your help,
EADG

Best Answer

Answers

  • KateNKateN Cambridge, MAMember, Broadie, Moderator

    You should be able to store your unmapped bams anywhere, as Cromwell should localize them to the working directory when you run the workflow. What does your inputs json look like?

  • EADGEADG KielMember
    edited December 2016

    hm I think this overlap with my question about the sub-function...If I'm right the sub-function strips off the path from the unmapped bams.
    String sub_strip_path = "gs://.*/"<br /> String sub_strip_unmapped = unmapped_bam_suffix + "$"
    So I have to adjust this command to my localfile-System.

  • KateNKateN Cambridge, MAMember, Broadie, Moderator

    Yes! We do use the sub() function to strip paths (gs:// in your example) and file types (.txt, .vcf, etc.) in our published pipeline.

  • EADGEADG KielMember

    Ok thank you...in the end I feel a little stupid because retrospective it seems obviously how the sub-function works and how the error is related too.

    But maybe other user will have a benefit from it when troubleshooting the local version of the pipeline....

  • I keep getting the exact same error when running the pipeline locally:
    Error in writing fastq file /dev/stdout
    Can someone PLEASE tell me what the fix is ?

  • here are the details:
    Picked up JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell-executions/germline_single_sample_workflow/0c7fec27-ffba-4132-bf87-a3ff2823586a/call-SamToFastqAndBwaMemAndMba/
    shard-0/tmp.70736ade
    Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/cromwell-executions/germline_single_sample_workflow/0c7fec27-ffba-4132-bf87-a3ff2823586a/call-SamToFastqAndBwaMemAndMba/
    shard-0/tmp.70736ade
    17:44:02.776 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/gitc/picard.jar!/com/intel/gkl/native/libgkl_compression.so
    17:44:02.785 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/usr/gitc/picard.jar!/com/intel/gkl/native/libgkl_compression.so
    [Fri Apr 27 17:44:02 UTC 2018] SamToFastq INPUT=/cromwell-executions/germline_single_sample_workflow/0c7fec27-ffba-4132-bf87-a3ff2823586a/call-SamToFastqAndBwaMemA
    ndMba/shard-0/inputs/mapr/jjoshi.mapr.com/gatk4/five-dollar-genome-analysis-pipeline/NA12878-small/NA12878-small-input-data/HJYFJ.4.NA12878.downsampled.query.sorte
    d.unmapped.bam FASTQ=/proc/self/fd/1 INTERLEAVE=true INCLUDE_NON_PF_READS=true OUTPUT_PER_RG=false COMPRESS_OUTPUTS_PER_RG=false RG_TAG=PU RE_REVERSE=true CLIPP
    ING_MIN_LENGTH=0 READ1_TRIM=0 READ2_TRIM=0 INCLUDE_NON_PRIMARY_ALIGNMENTS=false VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_REC
    ORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
    [Fri Apr 27 17:44:02 UTC 2018] Executing as root@9a5e098f97bc on Linux 3.10.0-693.5.2.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2~bpo8+1-b14;
    Deflater: Intel; Inflater: Intel; Picard version: 2.15.0-SNAPSHOT
    [Fri Apr 27 17:44:02 UTC 2018] MergeBamAlignment ADD_PG_TAG_TO_READS=false UNMAPPED_BAM=/cromwell-executions/germline_single_sample_workflow/0c7fec27-ffba-4132-bf8
    7-a3ff2823586a/call-SamToFastqAndBwaMemAndMba/shard-0/inputs/mapr/jjoshi.mapr.com/gatk4/five-dollar-genome-analysis-pipeline/NA12878-small/NA12878-small-input-data
    /HJYFJ.4.NA12878.downsampled.query.sorted.unmapped.bam ALIGNED_BAM=[/dev/stdin] OUTPUT=HJYFJ.4.NA12878.downsampled.query.sorted..aligned.unsorted.bam PROGRAM_RECOR
    D_ID=bwamem PROGRAM_GROUP_VERSION=0.7.15-r1140 PROGRAM_GROUP_COMMAND_LINE=bwa mem -K 100000000 -p -v 3 -t 16 -Y /cromwell-executions/germline_single_sample_workflo
    w/0c7fec27-ffba-4132-bf87-a3ff2823586a/call-SamToFastqAndBwaMemAndMba/shard-0/inputs/mapr/jjoshi.mapr.com/gatk4/five-dollar-genome-analysis-pipeline/NA12878-small/
    NA12878-small-reference-data/Homo_sapiens_assembly38.fasta PROGRAM_GROUP_NAME=bwamem PAIRED_RUN=true CLIP_ADAPTERS=false IS_BISULFITE_SEQUENCE=false ALIGNED_READS

    ONLY=false MAX_INSERTIONS_OR_DELETIONS=-1 ATTRIBUTES_TO_RETAIN=[X0] ATTRIBUTES_TO_REMOVE=[NM, MD] EXPECTED_ORIENTATIONS=[FR] ALIGNER_PROPER_PAIR_FLAGS=true SORT_OR
    DER=unsorted PRIMARY_ALIGNMENT_STRATEGY=MostDistant ADD_MATE_CIGAR=true UNMAP_CONTAMINANT_READS=true UNMAPPED_READ_STRATEGY=COPY_TO_TAG VALIDATION_STRINGENCY=SILEN
    T MAX_RECORDS_IN_RAM=2000000 REFERENCE_SEQUENCE=/cromwell-executions/germline_single_sample_workflow/0c7fec27-ffba-4132-bf87-a3ff2823586a/call-SamToFastqAndBwaMemA
    ndMba/shard-0/inputs/mapr/jjoshi.mapr.com/gatk4/five-dollar-genome-analysis-pipeline/NA12878-small/NA12878-small-reference-data/Homo_sapiens_assembly38.fasta AT
    TRIBUTES_TO_REVERSE=[OQ, U2] ATTRIBUTES_TO_REVERSE_COMPLEMENT=[E2, SQ] READ1_TRIM=0 READ2_TRIM=0 CLIP_OVERLAPPING_READS=true INCLUDE_SECONDARY_ALIGNMENTS=true MIN_
    UNCLIPPED_BASES=32 MATCHING_DICTIONARY_TAGS=[M5, LN] VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=2 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=c
    lient_secrets.json USE_JDK_DEFLATER=false USE_JDK_INFLATER=false
    [Fri Apr 27 17:44:02 UTC 2018] Executing as root@9a5e098f97bc on Linux 3.10.0-693.5.2.el7.x86_64 amd64; OpenJDK 64-Bit Server VM 1.8.0_111-8u111-b14-2~bpo8+1-b14;
    Deflater: Intel; Inflater: Intel; Picard version: 2.15.0-SNAPSHOT
    INFO 2018-04-27 17:44:03 SamAlignmentMerger Processing SAM file(s): [/dev/stdin]
    [M::bwa_idx_load_from_disk] read 3171 ALT contigs
    [W::main_mem] when '-p' is in use, the second query file is ignored.
    [M::process] read 662252 sequences (100000052 bp)...
    [M::process] 0 single-end sequences; 662252 paired-end sequences
    [Fri Apr 27 17:44:35 UTC 2018] picard.sam.SamToFastq done. Elapsed time: 0.55 minutes.
    Runtime.totalMemory()=5024776192
    To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
    Exception in thread "main" htsjdk.samtools.SAMException: Error in writing fastq file /proc/self/fd/1
    at htsjdk.samtools.fastq.BasicFastqWriter.write(BasicFastqWriter.java:66)
    at picard.sam.SamToFastq.writeRecord(SamToFastq.java:356)
    at picard.sam.SamToFastq.doWork(SamToFastq.java:211)
    at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268)
    at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
    at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
    [Fri Apr 27 17:44:35 UTC 2018] picard.sam.MergeBamAlignment done. Elapsed time: 0.55 minutes.
    Runtime.totalMemory()=3014656000
    To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp

  • EADGEADG KielMember

    Hi @jaideepjoshi,

    sry for the late response...did you fix your problem by your self ? When not, can you maybe share your wdl and the input.json? The public pipeline need some adjustments to run locally...

    Greets EADG

Sign In or Register to comment.