Using Virtualenv for safeguarding research project dependencies

Why virtualenv

It’s a tragedy whenever a code breaks after a software “update”. I have recently started using virtualenv for python to prevent my macport / apt-get updates from messing up the code of my research projects. Additionally, by providing a list of python package version, it can enhance reproducibility. Before I forget how to set it up correctly, I should write about it.

Prerequisites

I will use python 2.7 and assume the use of macport for this post. You should also have

py27-virtualenv
pip

installed, either with pip or macport or apt-get or if you like, from source. I think you also can replace pip with easy_install if you prefer easy_install instead. I also assume you know some basic shell commands.

How to create virtualenv

If you just want to use virtualenv for the same version python as the system wide version, at a terminal, call the executable of virtualenv:

$ virtualenv-27 --no-site-packages ENVNAME

The command could also be virtualenv for some people. I assumed that virtualenv was installed using macport

creating virtualenv this way will instruct that the paths inside the virtual env not to include path the system wide python packages (using –no-site-packages seems to be the default behavior for virtualenv but let me just mention that) After this command, whatever python packages installed to the PATH variable in the shell configuration file (e.g. .bashrc / .bash_profile) will be ignored when the virtualenv is activated.

Caveats

Note if you have python packages path set as part of PYTHONPATH, your virtualenv will not work properly so you will want to remove any system wide python packages from your PYTHONPATH.

Once the command above is done executing, a folder with ENVNAME will be created that has the structure for installing its own set of python packages.

$ ls ENVNAME

will show you the following directory structure

bin include lib share

Next, you want to figure out what packages to install and what not to install. These packages will be installed within the ENVNAME directory structure, more specifically inside:

${APPROPRAITE_PATH}/${ENVNAME}/lib/python2.7/site-packages

Now let’s try to activate the virtual env:

$ source APPROPRIATE_PATH/ENVNAME/bin/activate

Then you terminal prompt will change to

(ENVNAME)$

check if the pip inside this environment is a local version:

(ENVNAME)$ which pip

should show

(ENVNAME)$ ENVNAME/bin/pip

Now you can feel free to use this local version of pip to install whatever packages you want.

Saving package version and reinstalling packages

Let's say you have installed a bunch of packages for a specific coding project and you want to save the package used and the respective version. We can have the pip list all the python packages you have installed and save it somewhere

(ENVNAME)$ pip freeze > packages.txt

Let's say you want to set up the same virtual environment on another machine. Now you can copy over "packages.txt", set up the machine using the steps above, then reinstall your packages within this local environment:

(ENVNAME)$ pip install -r packages.txt

which may or may not work well depending on if you have external source of packages.

Let’s say you are like me and have installed some python packages from non-conventional sources, one hackish way to make sure those will be installed properly in the virtualenv is to just copy the directory containing installed python packages to the local environment:

(ENVNAME)$ sudo cp -r PATH1 APPROPRAITE_PATH/ENVNAME/python2.7/site-packages

where PATH1 is

$ echo PATH1
"/TIME_MACHINE_BACKUP/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python/2.7/site-packages/"

Now you can check if the library versions are alright:

(ENVNAME)$ ipython
In [1]: import astropy
In [2]: print astropy.__version__
Out[2]: '0.4'

while you can check that the system wide version of the library has a different version after deactivating the virtual env

$ APPROPRAITE_PATH/${ENVNAME}/bin/deactivate  
$ ipython
In [1]: import astropy
In [2]: print astropy.__version__
Out[2]: '0.4.1'

Automating virtualenv activation

Last bit of this post is about being lazy and having the virtualenv automatically activate itself when you switch to the appropriate directory using command line.

You will need the python package autoenv which can be installed by:

$ pip install autoenv
$ echo "source $PATH_TO_AUTOENV/activate.sh" >> ~/.bash_profile
$ cd $APPROPRIATE_PATH/ # this is the parent directory where ENVNAME lives
$ echo "source ABSOLUTE_PATH/ENVNAME/bin/activate" > .env

now whenever you switch to the appropriate directory, the virtual env will be activated for you.

Troubleshooting

If you see any weird behavior for importing modules in the virtualenv or when using pytest, make sure that you are using versions of python packages installed from within the virtualenv and not the system version.

creating virtualenv for a different python version

let's say that I use python 2.7 for most of my projects and would like to play with python 3.4 i can first install python version 3.4 using macport by:

$ sudo port install python34 py34-pip

then the only difference when creating the virtualenv is to specify using python 3.4

$ virtualenv -p $(which python3.4) ENVNAME --distribute