19 Apr 2012

From Lyon

Tag: Python

Embed IPython in a virtualenv

Written by Balthazar

About virtualenv

I recently got introduced to virtualenv: a “tool to create isolated Python evironments”. It allows to have a fine grain control on the dependencies of each of your python project, and separate each project environment from the others.

For example, if a cool project requires the spamspam v1.1 and another cool project requires the version 1.2, performing a system-wide update of the spamspam library from 1.1 to 1.2 would cause the first project to break. To avoid this kind of problem, you can place each project in a separate virtualenv, and force the version of the spamspam library to the required one.

You can read more about virtualenv on their PyPI page, but a important takeaway is that you will have pseudo-root rights in your virtualenv: you will be able to install whatever python packages you need. I strongly recommand to use pip for that.

Combine virtualenv with IPython

If you don’t know IPython, check it out. It adds powerful features like autocompletion, colors, crazy data visualisation tools, help, source code visualisation of packages and functions, etc. It’s just raw awesomness.

Now, the problem is that virtualenv default behavior is to ignore all packages located in your /usr/lib/pythonX.X/site-packages directory when creating a new virtualenv, which is precisely where IPython is installed. You thus won’t be able to use IPython in your virtualenv without a little bit of hacking.

There are two ways to make it work:

  • the good way
  • the bad way

I’ll describe both techniques and you’ll decide which is which.

Install a local IPython in each virtualenv

First, you can install IPython in each new virtualenv, with

$ pip install ipython

That will work well, but the package is quite heavy (~10Mb), and frankly, that could/should be automaded. Another problem I see with this technique is that you might want to push your code on a repository, for other people to be able to contribute. If so, you might want to push the virtualenv content, to simplify them the set-up work. Even if extremely handy, IPython might not be vital for you project, and you should only install required packages in the virtualenv, for the sake of testing.

One IPython to rule them all

Now there’s a trickier solution: we can configure your system IPython, to make it aware of your virtualenv. This has been extremely well done by Chris Lasher, and all instructions and explanations can be found here. The main idea is to check if you’re running in a virtual environment when you launch IPython. If so, add all your virtualenv python environment at the begining of your PYTHONPATH.

I however encountered a small problem with this method. IPython being in /usr/lib/python2.7/site-packages directory, my virtualenv was not even aware of an IPython module being installed, and this error was fired even before executing the code shown in Chris’ cool blogpost:

Traceback (most recent call last):
  File "/usr/bin/ipython", line 30, in <module>
    import IPython.Shell
ImportError: No module named IPython.Shell

Here’s the small hack I did to make everything work. First I added these lines to my /usr/bin/ipython file, before importing IPython.Shell:

import sys
if "/usr/lib/python2.7/dist-packages" not in sys.path:
    sys.path.append("/usr/lib/python2.7/dist-packages")

The append command will only be executed when I’m launching IPython from a virtualenv, otherwise, the if statement would not be True. Now that /usr/lib/python2.7/dist-packages is in my sys.path, I can import IPython.Shell from my virtualenv.

The code contained in ~/.ipython/virtualenv.py (as described in Chris’ article) will now be executed but still need a “counter-hack”, to cancel the effects of what we wrote in /usr/bin/ipython. We need to completely remove /usr/lib/python2.7/dist-packages from our PYTHONPATH.

To do that, we’ll add a small snippet in ~/.ipython/virtualenv.py

for item in sys.path:
    if '/usr/lib/python2.7/dist-packages' in item:
        sys.path.remove(item)

Here is the final ~/.ipython/virtualenv.py code:

import site
from os import environ
from os.path import join
import sys

if 'VIRTUAL_ENV' in environ:
    virtual_env = join(environ.get('VIRTUAL_ENV'),
                       'lib',
                       'python%d.%d' % sys.version_info[:2],
                       'site-packages')

    # Remember original sys.path.
    prev_sys_path = list(sys.path)
    site.addsitedir(virtual_env)

    # Personal hack
    # Remove /usr/lib/python2.7/dist-packages and associates from the path
    # Counter the effects of this code, in /usr/bin/ipython
    # import sys
	# if "/usr/lib/python2.7/dist-packages" not in sys.path:
    #     sys.path.append("/usr/lib/python2.7/dist-packages")
    for item in sys.path:
        if '/usr/lib/python2.7/dist-packages' in item:
            sys.path.remove(item)
            

    # Reorder sys.path so new directories at the front.
    new_sys_path = []
    for item in list(sys.path):
        if item not in prev_sys_path:
            new_sys_path.append(item)
            sys.path.remove(item)
    sys.path[1:1] = new_sys_path

    print 'VIRTUAL_ENV ->', virtual_env
    del virtual_env

Proof by trial

Now, let’s try it, to check everything’s working:

balto $ cd path/to/virtualenv/project
balto $ source bin/activate
(project)balto $ ipython
VIRTUAL_ENV -> /path/to/virtualenv/project/lib/python2.7/site-packages
Python 2.7.2+ (default, Oct  4 2011, 20:06:09) 
Type "copyright", "credits" or "license" for more information.

IPython 0.10.2 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object'. ?object also works, ?? prints more.

In [1]: 

Conlcusion

Yay. Now, you won’t have to worry about installing IPython when creating a new virtualenv. To check that everything is working properly, install a package in a fresh virtualenv that you have not installed in your “system” python. Open two IPython interpreters: one in the virtualenv, one outside, and try to import this newly installed package in both. The virtualenv IPython should be fine, and the second should fire an ImportError.

Packages and version

I’m running IPython 0.10.2, Python 2.7.2+, and virtualenv 1.7.1.2.

Comments