Notice:
If you happen to see a question you know the answer to, please do chime in and help your fellow community members. We encourage our fourm members to be more involved, jump in and help out your fellow researchers with their questions. GATK forum is a community forum and helping each other with using GATK tools and research is the cornerstone of our success as a genomics research community.We appreciate your help!

Test-drive the GATK tools and Best Practices pipelines on Terra


Check out this blog post to learn how you can get started with GATK and try out the pipelines in preconfigured workspaces (with a user-friendly interface!) without having to install anything.

(howto) Set up a Virtual Environment and install all dependencies for Oncotator

Geraldine_VdAuweraGeraldine_VdAuwera adminCambridge, MAMember, Administrator, Broadie admin
edited June 2014 in Oncotator documentation

Using the script we provide is the easiest way to set up your Python environment to run Oncotator. It will install all the other software that is required by Oncotator. Note however that the script does not install Oncotator itself; you will still have to do that manually once you have successfully run the script. See this tutorial for instructions on how to install Oncotator and run it for the first time.


1. Install the virtualenv library for Python

First you need to install the virtualenv library for Python, which you can get here. You can either use your favorite Python package management tool, or install it manually.

$ pip install virtualenv  

In this tutorial we use pip, which is a very convenient way to manage Python packages (for more details and instructions on installing pip, see the documentation here).


2. Make sure that you have a C compiler and a FORTRAN compiler installed on your system

The script will install some software that needs to be compiled, so you need to have some compilers installed on your system. See the article on Oncotator's dependencies for details and links to software downloads. If you are on MacOSX, make sure you have the XCode developer tools installed.


3. Run the setup script

Once you have virtualenv installed, you just run the script we provide using the following command:

$ bash scripts/create_oncotator_venv.sh -e <venv_location>

where <venv_location> is the location (filepath) where you want your virtual environment to live. This creates several directories containing all the necessary files for your shiny new virtual environment. If that works right away (which it should in most cases), skip on ahead to the next step. If you run into any issues, see the Troubleshooting section at the end of this document.


4. Activate the virtual environment

Whenever you want to use your virtual environment, use this command to activate it:

$ source <venv_location>/bin/activate

Once the environment is activated, anything you do involving Python (installing packages or running programs) will be executed within that environment. It's like having a bubble universe on your machine. When you're done, just deactivate it using this command:

$ deactivate

If you plan to always do all your Python work inside the virtual environment, you can leave it activated, and it will be ready for you at any time as long as your shell session stays open.

See this document for more details on how to use your new virtual environment.

If you wish to add the oncotator executable to your path:

$ export PATH=<venv_location>/bin/oncotator:$PATH

5. Troubleshooting the setup script

There are a few known issues that can cause the script to fail, depending on your system. See the article on Oncotator's dependencies for details of known issues and solutions. If you are on MacOSX and you run into the XCode 5.1.1 compilation bug (described in the article referenced above), you can apply the workaround to the script by adding the -c flag to your command like this:

$ bash scripts/create_oncotator_venv.sh -e <venv_location> -c

If that still doesn't work, you'll have to skip those packages and install them manually afterward. To skip packages that require compilation, add the -k flag to your command like this:

$ bash scripts/create_oncotator_venv.sh -e <venv_location> -k

Finally, there is one package (pyvcf) that currently can't be installed through pip. The script is set up to retrieve the source (using wget) and install it from another repository, but in some cases (e.g. if you don't have wget installed) it may fail. If so, you'll need to skip that package and install it manually afterward. To skip that package, add the -s flag to your command like this:

$ bash scripts/create_oncotator_venv.sh -e <venv_location> -s

For instructions on how to install packages manually, see the articles on dependencies and the tutorial on installing python libraries.

Post edited by LeeTL1220 on

Comments

  • egeulgenegeulgen USMember
    edited August 2015

    I keep failing when running the create venv script. I tried to install the failing library(ngslib) manually but keep getting the same error.
    A little help would be great! :) @LeeTL1220

    Command and outputs below:

    Ege-Ulgens-MacBook-Pro:oncotator-1.5.1.0 egeulgen$ bash scripts/create_oncotator_venv.sh -e ~/oncotator -c
    Option -c specified -- the ARCHFLAGS workaround will be applied.
    New python executable in /Users/egeulgen/oncotator/bin/python
    Installing setuptools, pip, wheel...done.

    Virtual environment created and activated in /Users/egeulgen/oncotator.
    Now attempting to install packages into the virtual environment.
    Update Pip
    pip 7.1.0 from /Users/egeulgen/oncotator/lib/python2.7/site-packages (python 2.7)
    Requirement already up-to-date: pip in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    pip 7.1.0 from /Users/egeulgen/oncotator/lib/python2.7/site-packages (python 2.7)
    Attempting to install packages that require compilation. If this fails, try again with the flag -c added to the script command. If that still does not work, you will need to install them manually.

    biopython ==========================
    Requirement already satisfied (use --upgrade to upgrade): biopython in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    OK

    cython ==========================
    Requirement already satisfied (use --upgrade to upgrade): cython in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    OK

    numpy ==========================
    Requirement already satisfied (use --upgrade to upgrade): numpy in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    OK

    pandas ==========================
    Requirement already satisfied (use --upgrade to upgrade): pandas in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from pandas)
    Requirement already satisfied (use --upgrade to upgrade): pytz>=2011k in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from pandas)
    Requirement already satisfied (use --upgrade to upgrade): numpy>=1.7.0 in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from pandas)
    Requirement already satisfied (use --upgrade to upgrade): six>=1.5 in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from python-dateutil->pandas)
    OK

    sqlalchemy ==========================
    Requirement already satisfied (use --upgrade to upgrade): sqlalchemy in /Users/egeulgen/oncotator/lib/python2.7/site-packages
    OK

    ngslib ==========================
    Collecting ngslib
    Using cached ngslib-1.1.14.tar.gz
    Requirement already satisfied (use --upgrade to upgrade): numpy>=1.4.1 in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from ngslib)
    Requirement already satisfied (use --upgrade to upgrade): pysam>=0.8.2 in /Users/egeulgen/oncotator/lib/python2.7/site-packages (from ngslib)
    Building wheels for collected packages: ngslib
    Running setup.py bdist_wheel for ngslib
    Complete output from command /Users/egeulgen/oncotator/bin/python -c "import setuptools;file='/private/var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-build-4nJY9u/ngslib/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" bdist_wheel -d /var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/tmpTjc_XRpip-wheel-:
    Compile KentLib ...
    cd lib && /Applications/Xcode.app/Contents/Developer/usr/bin/make
    gcc -c -fPIC -Wall -I../inc -DMACHTYPE_x86_64 pngwrite.c -o pngwrite.o
    pngwrite.c:7:10: fatal error: 'png.h' file not found
    #include "png.h" // MUST come before common.h, due to setjmp checking in pngconf.h
    ^
    1 error generated.
    make[1]: *** [pngwrite.o] Error 1
    make: *** [all] Error 2
    running bdist_wheel
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.6-intel-2.7
    creating build/lib.macosx-10.6-intel-2.7/ngslib
    copying src/init.py -> build/lib.macosx-10.6-intel-2.7/ngslib
    copying src/fdr.py -> build/lib.macosx-10.6-intel-2.7/ngslib
    copying src/lprob.py -> build/lib.macosx-10.6-intel-2.7/ngslib
    copying src/wplot.py -> build/lib.macosx-10.6-intel-2.7/ngslib
    copying src/wStats.py -> build/lib.macosx-10.6-intel-2.7/ngslib
    running build_ext
    building 'wWigIO' extension
    creating build/temp.macosx-10.6-intel-2.7
    creating build/temp.macosx-10.6-intel-2.7/external
    creating build/temp.macosx-10.6-intel-2.7/external/KentLib
    creating build/temp.macosx-10.6-intel-2.7/external/KentLib/wWigIO
    /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wno-error=unused-command-line-argument-hard-error-in-future -Wno-error=unused-command-line-argument-hard-error-in-future -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c external/KentLib/wWigIO/wWigIO.c -o build/temp.macosx-10.6-intel-2.7/external/KentLib/wWigIO/wWigIO.o -w -shared -fPIC -p -Iexternal/KentLib/inc
    /usr/bin/clang -bundle -undefined dynamic_lookup -g -Wno-error=unused-command-line-argument-hard-error-in-future -Wno-error=unused-command-line-argument-hard-error-in-future build/temp.macosx-10.6-intel-2.7/external/KentLib/wWigIO/wWigIO.o -o build/lib.macosx-10.6-intel-2.7/wWigIO.so -DMACHTYPE_x86_64 -lz -lm external/KentLib/lib/jkweb.a
    clang: error: no such file or directory: 'external/KentLib/lib/jkweb.a'
    error: command '/usr/bin/clang' failed with exit status 1


    Failed building wheel for ngslib
    Failed to build ngslib
    Installing collected packages: ngslib
    Running setup.py install for ngslib
    Complete output from command /Users/egeulgen/oncotator/bin/python -c "import setuptools, tokenize;file='/private/var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-build-4nJY9u/ngslib/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-cTfuqK-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/egeulgen/oncotator/bin/../include/site/python2.7/ngslib:
    Compile KentLib ...
    cd lib && /Applications/Xcode.app/Contents/Developer/usr/bin/make
    gcc -c -fPIC -Wall -I../inc -DMACHTYPE_x86_64 pngwrite.c -o pngwrite.o
    pngwrite.c:7:10: fatal error: 'png.h' file not found
    #include "png.h" // MUST come before common.h, due to setjmp checking in pngconf.h
    ^
    1 error generated.
    make[1]: *** [pngwrite.o] Error 1
    make: *** [all] Error 2
    running install
    running build
    running build_py
    running build_ext
    building 'wWigIO' extension
    /usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -g -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Wno-error=unused-command-line-argument-hard-error-in-future -Wno-error=unused-command-line-argument-hard-error-in-future -I/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c external/KentLib/wWigIO/wWigIO.c -o build/temp.macosx-10.6-intel-2.7/external/KentLib/wWigIO/wWigIO.o -w -shared -fPIC -p -Iexternal/KentLib/inc
    /usr/bin/clang -bundle -undefined dynamic_lookup -g -Wno-error=unused-command-line-argument-hard-error-in-future -Wno-error=unused-command-line-argument-hard-error-in-future build/temp.macosx-10.6-intel-2.7/external/KentLib/wWigIO/wWigIO.o -o build/lib.macosx-10.6-intel-2.7/wWigIO.so -DMACHTYPE_x86_64 -lz -lm external/KentLib/lib/jkweb.a
    clang: error: no such file or directory: 'external/KentLib/lib/jkweb.a'
    error: command '/usr/bin/clang' failed with exit status 1

    ----------------------------------------
    Command "/Users/egeulgen/oncotator/bin/python -c "import setuptools, tokenize;file='/private/var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-build-4nJY9u/ngslib/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-cTfuqK-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/egeulgen/oncotator/bin/../include/site/python2.7/ngslib" failed with error code 1 in /private/var/folders/5w/fr7s0y7d2q1g8nph79l06mh00000gn/T/pip-build-4nJY9u/ngslib

  • egeulgenegeulgen USMember

    It finally worked after I unistalled and reinstalled libpng using homebrew

  • LeeTL1220LeeTL1220 ✭✭✭ Arlington, MAMember, Broadie, Dev ✭✭✭
  • Mikko KiventoMikko Kivento HelsinkiMember
    edited November 2015

    I believe I've been having the same problem, getting this output from the script:

    Installing collected packages: pysam, ngslib Running setup.py install for pysam Running setup.py install for ngslib Complete output from command /data/bioinfo/general/python_venv/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-QyQmuW/ngslib/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-r23WZS-record/install-record.txt --single-version-externally-managed --compile --install-headers /data/bioinfo/general/python_venv/include/site/python2.6/ngslib: Compile KentLib ... cd lib && make make[1]: Entering directory '/tmp/pip-build-QyQmuW/ngslib/external/KentLib/lib' gcc -c -fPIC -Wall -I../inc -DMACHTYPE_x86_64 pngwrite.c -o pngwrite.o pngwrite.c:7:87: error: png.h: No such file or directory
    ...plus 100 lines of more errors.

    With
    pip install ngslib
    I get the same error.

    SOLUTION:
    Doing Google searches I stumbled upon the libpng-devel library. So I installed it:
    sudo yum install libpng-devel

    Then both the manual install of ngslib and running the script succeeded.
    And in case you wonder about the Python2.6 tag in there, I'm using Python 2.7.5 via Software Collection instructed in this article.

Sign In or Register to comment.