Managing multiple Python projects: Virtual environments

Using a Virtual Environment

When working at the command line, you can put the virtual environment's "bin" directory first on your PATH, what we call "activating" the environment, and from then on, anytime you run python, you'll be running in the environment.

(This is one advantage of starting your scripts with:

#!/usr/bin/env python

rather than:

#!/usr/bin/python

By using the "/usr/bin/env" version, you'll get the first copy of Python that's on your PATH, and if you've activated a virtual environment, your script will run in that environment.)

Virtual environments provide a "bin/activate" script that you can source from your shell to activate them, e.g.:

$ . /path/to/virtualenv/foo/bin/activate
$ python
... runs the python from virtual environment "foo"

(but be sure to notice that you have to source the activate script, using . or source, and running the script normally will not activate your virtual environment as you might expect).

After activating a virtual environment in a shell, a new command deactivate will become available, that will undo the activation.

Activation is just a convenience for use in an interactive shell. If you're scripting something, just invoke Python or whatever other script or executable you need directly from the virtual environment's bin directory. E.g.:

/path/to/env/bin/python myprogram.py

or:

/path/to/env/bin/script

Anything that's been installed in the virtual environment and has executables or scripts in the bin directory can be run from there directly and will run in the virtual environment.

Installing packages into a virtual environment

In an activated virtual environment, you'll have a command pip that you'll use to install, update, and remove packages. E.g.:

$ pip install requests==2.11.1

will make the requests package, version 2.11.1, available to Python programs running in that virtual environment (and no other). It works by installing it in the lib directory within the virtual environment, which will only be on the library path for Python when running under that virtual environment.

When the requests package releases version 2.12, you can upgrade to it with:

$ pip install -U requests==2.12

or just upgrade to the latest version with:

$ pip install -U requests

If you decide that your project no longer needs requests, you can remove it:

$ pip uninstall requests

In practice, we usually put a project's requirements in a file in the project's top directory named requirements.txt, one per line, and then use the -r option of pip to install them all at once. So we might have this in requirements.txt:

requests==2.12
psycopg2==2.6.1

and then install all that with this command:

$ pip install -r requirements.txt

Of course, pip has all sorts of options and other features, which you can read about in the Pip documentation.

Pip outside of virtual environments

Pip is automatically provided for you inside all virtual environments, but you can also install it system-wide and use it to install Python packages for the whole system. But there are some things to be careful of:

• Be sure to be aware whether you're in a virtual environment or not; it's much more likely that you intend to install something in a virtual environment, but if you run pip outside of one, it'll happily install things to the whole system if you have permissions.
• Using pip to install things to the whole system can create conflicts with Python packages provided by your operating system distribution. Pip will typically install things under /usr/local while OS packaging will install things under /usr, which can help you figure out which version of things you're getting.

Creating virtual environments

Python started including direct support for virtual environments in version 3.3, but for versions before that, including Python 2.7, you need to use a third-party tool called virtualenv to create them.

virtualenv still works for all versions of Python, and since I still need to deal with Python 2.7 on a regular basis, and don't want to try to remember the details of two different ways to create virtual environments, I just use virtualenv for everything. Plus, I still think virtualenv is simpler to use and more flexible.

Here's how simple using virtualenv is:

$ virtualenv -p /path/to/python /path/to/new/virtualenv

So if I wanted to create a virtual environment at $HOME/foo that was running Python 2.7, I could run:

$ virtualenv -p /usr/bin/python2.7 $HOME/foo

That just creates the virtual environment. When you're ready to use it, don't forget to activate it first.

Note that whenever virtualenv creates a new virtual environment, it automatically installs pip into it. This avoids the chicken-and-egg problem of how to install pip into the virtual environment so you can install things into the virtual environment.

Installing virtualenv

You can often install virtualenv using a system package, but it will likely be an old version. Since virtualenv bundles pip, that means all virtual environments you create with it will start with an old version of pip, and will complain at you until you upgrade pip to the latest. (That's easy - pip install -U pip - but gets tiresome.)

I prefer to just install the latest version of virtualenv, and ironically, the best way to install the latest version of virtualenv system-wide is using pip.

First, you'll want to install pip on your system - that is, globally. If your distribution provides a package to install pip, use that to avoid possibly breaking your system's installed Python.

I'll try to provide instructions for a variety of systems, but be warned that the only one I'm currently using is Ubuntu.

For Debian or Ubuntu, you'd use sudo apt-get install python-pip. For RPM-based distributions, yum -y install python-pip. For Arch, try pacman -S python-pip. On Mac OS X, brew install python should install pip too.

Only if your distribution does not provide a way to install pip, then you'll need to install it directly. First, securely download get-pip.py. You can click that link and save the file somewhere, or use wget from the command line:

$ wget https://bootstrap.pypa.io/get-pip.py

Then run get-pip.py, with system privileges:

$ sudo python get-pip.py
Collecting pip
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 932kB/s
Collecting wheel
  Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
    100% |████████████████████████████████| 71kB 8.1MB/s
Installing collected packages: pip, wheel
Successfully installed pip-8.1.2 wheel-0.29.0

Now that you have a system pip, you can install virtualenv:

$ sudo pip install virtualenv

and anytime you want to make sure you have the latest version of virtualenv:

$ sudo pip install -U virtualenv

Learning more

virtualenv and pip are just parts of the whole virtual environment ecosystem for Python. They're enough to get a lot done, but here are some other things you might find interesting at some point.

• virtualenvwrapper provides shell shortcuts to make working with virtual environments easier. For example, instead of having to type . path/to/my-venv/bin/activate, you can just type workon my-venv.

• If you want to make your own package installable by pip, see the Python Packaging User Guide.

  (And send a word of thanks to the people behind it, the Python Packaging Authority. For many years, the packaging side of Python was a poorly documented mess of different, conflicting, poorly documented tools (did I mention the poor documentation?). In the last couple of years, the PPA has brought order out of this chaos and the Python development community is so much better for it.)