Automagic Python virtual environments with asdf and direnv

There are a LOT of ways to setup Python in your machine.

You may find it’s already installed, or you can install it from python.org, or your OS’s package manager, or you can build it from source. But if you’ve been working with Python for a while, you’re bound to want more than one version of Python on your system. You may be working on a library and want support for more than one version, or the version used by your employer is not quite the latest and you want to use all the new stuff in your side project.

There are also a LOT of ways to manage your virtual environments in Python. As you know if you’ve worked in Python… well, at all, creating, activating, and deactivating virtual environments is a big part of the workflow. So naturally there are many tools that address this.

This creates a combinatorial explosion of tools that results in basically everyone having their own way of setting up Python for development. In this article I’d like to share my combination and why I like it.

TL;DR I use asdf to install different versions of Python combined with direnv via asdf-direnv for virtual environment management. Sounds complicated? I promise I have valid reasons.

Installing all the Pythons

I work and mantain a few libraries that support many versions of Python, so in order to run the tests I need a way to install each version separately and have all of them available for tox or nox to run the test suite on each version.

This is something you simply cannot solve using your OS package manager since it’s designed to install one specific version. Naturally there are tools that tackle this problem and the most popular one is pyenv.

The idea, if you’re not familiar with it, is to expose a command to install any version of Python and allow the user to set up a global version and any number of local versions that will become active as the user enters a directory. This is done with simple commands like pyenv install 3.10, pyenv global 3.10.4, and pyenv local 3.9.12, the latter will write to a special file in the directory that tells pyenv to use that specific version of Python when entering it.

This principle is, of course, applicable and useful for all programming languages, pyenv is in fact a fork of Ruby’s rbenv, and there are many other tools for other similar programming languages like Node.js’s nvm. So, naturally, a single solution for all languages emerged: asdf.

asdf works the same way as pyenv and similar tools but allows a series of plugins for each language. The commands are essentially the same but you only need to add the language, something like asdf local python 3.10.4. The Python plugin for asdf actually uses pyenv’s python-build to install the versions of Python, so all the helpful building hints described in pyenv’s wiki still apply.

You may be wondering how do these tools know that if you type python in a specific directory, one particular version of Python needs to be executed and not another. After all, when you run any command in your shell the executable needs to be present in your PATH, and if there are two of them surely the first one would be executed which may not be the one you want.

The solution: shims.

pyenv has a great explanation for them, but essentially when you type python you don’t actually run Python itself but small shell script that resolves which version of Python you want and executes it.

Why am I mentioning this? You shall soon find out… 🧐

The virtual environment management spectrum

Managing Python virtual environments is a very personal thing. Some people like no managing at all and have developed muscle memory for creating and (de)activating the enviroments as they move between projects.

Others rely on higher level tools like Poetry or Hatch to manage the virtual environments along with other aspects of your project, like dependencies, packaging or or publishing.

A separate issue is where you like your virtual environments to be located. Many people like having the virtual environments in their project’s directory, others prefer all virtual environments to be all together somewhere else in the file system. Others don’t care as long as it works.

Personally, I like to have them colocated with the project and activate as I enter the project’s directory.

This type of workflow is exactly what the classic virtualenvwrapper library does. But you’ll notice it’s also what pyenv does for Python versions, so the good people behind pyenv created pyenv-virtualenvwrapper for those who are used to the virtualenvwrapper workflow, and pyenv-virtualenv which works very similarly but extends the pyenv command with virtual environment management capabilities.

“Ok, but what if you’re using asdf?”, you ask.

Well, since virtual environments are a very Python thing it doesn’t fit as neatly into its language agnostic API so there’s no plugin to help you with it. But all is not lost…

Environment variables interlude

If you’re a fan of the 12-factor app, or if you use environment variables at all in your code, you may be familiar with direnv. But in case you’re not, direnv an excellent tool that allows you to define enviroment variables in a .envrc file on your project’s directory, and when you cd into it, it will export them automatically. Once you cd out of it, direnv will unset them.

That in itself is amazingly useful, but it turns out you can do much more, including creating and activate virtual environments!¹

This is exactly what I wanted: I can use asdf to set a local Python version for my project, and have direnv create and activate the virtual environment as I enter the directory. Perfect!

But alas, all was not well in the land…

The shim issue

Remember when I mentioned how shims work before? Turns out those small executables can run a fair bit of code and they run on each call of each shimmed command which can be noticable, particularly if you’re on a slow machine.

Additionally, if you’re used to the which command to know location of an executable, e.g. which python, you will always get the same answer: the location of the shim, which is not the one you want². This is because pyenv et al do not change the PATH environment variable.

This may not sound like a big deal, and it wasn’t, until it was. I had a very obscure error when trying to run the integration test suite for lunr.py using tox where the node executable was not correctly resolved by the shim in the tox environment, but manually adding its location to PATH worked.

However, there is a solution!

The solution

Turns out you can install direnv using asdf using asdf-direnv! It will also intregrate them, so that when you enter a directory, it will set the appropriate locations of the executables in your PATH using direnv, removing the shimming step and addressing the issues I mentioned above. asdf-direnv goes into detail about the motivations and describing the solution in more detail.

What does that look like specifically?

1. Clone a repo or create a new directory to work on
2. Set the local Python version, e.g. asdf local python 3.10.4

3. Create a .envrc file with the following:

    use asdf
    layout python python3.10
   

4. direnv will detect the file but require your specific approval via direnv allow

At this point asdf-direnv will resolve and set the appropriate executables in PATH and direnv will create any virtual environments in .direnv and activate them.

As I exit the directory the virtual environment will deactivate and the PATH will be restored. When I enter the directory the virtual environment will activate and PATH will be populated along with any environment variables I have declared in .envrc.

There is one catch though: because the PATH is changed when entering the directory, if you install a new library that exposes a new executable it will not be immediately available. You must either exit and enter the directory, or run direnv reload.

Wrapping it all up

So we have asdf to install Python versions, direnv which takes care of the virtual enviroments per project and enviroment variables, and asdf-direnv connecting the two. It’s blazing fast on any machine and which works as expected. If something goes wrong or you want to start over, simply delete .direnv and run direnv reload.

I’m sure at this point you’re wincing, thinking how you prefer your own setup. That’s fine, this one ticks all the boxes for me and my projects.

---