Having that pip install -r requirements.txt that takes ages to install? Make sure you install all binary packages from wheels.

When “pip install” can’t find a wheel, it falls back to installing from source. In a large project, dependency after dependency, it slows down builds and makes developers unhappy.

What are Python wheels and why they are so fast

Long ago, most Python packages were distributed as source code with .tar.gz. The installed downloaded the archive and ran the setup.py script inside it. This format is known as “sdist”, source distribution.

Source distributions are still a thing, but about ten years ago, PEP 427 defined a so-called “Wheel format.” Wheels are nothing but the zip-files with the .whl extension. With wheels, you don’t run the code inside the package to install it or compile anything. The installer simply unzips the contents in the pre-defined location.

The performance advantage of wheels over sdist is especially apparent with binary packages. Our goal is to eradicate the need for compiling and make the installation process as fast as possible. Let’s see how we can do it.

Pre-requisites

Update pip and setuptools

Make sure you have the latest version of pip, setuptools, and wheel. Outdated versions may not recognize your platform and fail to find matching wheels.

My CI tasks or Docker files usually update these core dependencies before installing the requirements.

pip install -U pip setuptools wheel
pip install -r requirements.txt

How to spot problematic packages?

Run pip install --only-binary=:all: and see where it fails.

There is an option –only-binary to install only wheels.

pip install --force-reinstall -r requirements.txt --only-binary=:all:

Along the way, you may find some unmaintained projects that don’t have wheels.

ERROR: Could not find a version that satisfies the requirement package-name==0.0.1
ERROR: No matching distribution found for package-name==0.0.1

As long as they are pure python packages, it’s OK to keep them in sdist format. Include them in the installation list one by one.

pip install --force-reinstall -r requirements.txt --only-binary=:all: --no-binary=package-name,another-package-name

At some point, you spot a few packages that are not in the wheel format, but you really don’t want to install them from source. It’s time to find out how to fix it.

How to spot problematic packages faster?

Explore the output of pip download.

In one of my projects, there were too many source distributions, and filtering them out one by one was too cumbersome. I found an alternative approach to download everything and see how many .tar.gz files are in the output directory.

The pip download command can download all packages in the given repository.

mkdir -p downloads
pip download -r ./requirements.txt  --dest downloads
ls downloads/*.tar.gz

How does pip know which wheel to install?

It compares available wheels with your platform tags.

You can see the list of wheels on PyPI, the Python package index. For example, for Pandas 1.35, you’ll find more than a dozen wheels for different platforms.

Each wheel file has a so-called compatibility tag that defines the platform where it can be installed. The tag has the format {python tag}-{abi tag}-{platform tag}, for example, cp310-cp310-win_amd64 for the first package in our list.

The installer finds an intersection between this list and the list of tags compatible with your interpreter and platform.

TO the whole (very long) list of wheels compatible with your installation, type pip debug --verbose.

For example, for my Intel macOS machine and Python 3.10, I have the following tags:

$ pip debug --verbose
...
Compatible tags: 1945
  cp310-cp310-macosx_11_0_x86_64
  cp310-cp310-macosx_11_0_intel
  cp310-cp310-macosx_11_0_fat64
  ...
  py33-none-any
  py32-none-any
  py31-none-any
  py30-none-any

If the intersection between the two lists is found, pip will install the package from the wheel. Otherwise, it falls back to installing from source.

Why there’s no wheel for my platform?

There’s no wheel for your platform.

Now you know that somehow pip doesn’t have a wheel to install. Why is that, and what can you do about it?

• The package doesn’t have wheels at all. You may consider politely asking the maintainer to issue a new version with a wheel.
• You updated Python but didn’t update dependencies. The version of Python is newer than the version of the package. Older dependencies know nothing about more recent Python and don’t have wheels.
• The opposite can also happen. If your version of Python is too old, you may not have a wheel for the package since maintainers stopped supporting it.

The list of reasons, of course, is not exhaustive. You may have more exotic reasons for the mismatch. At least, now you know how to debug it.

More about wheels

• PEP 425. Compatibility Tags for Built Distributions
• PEP 427. The Wheel Binary Package Format 1.0
• What Are Python Wheels and Why Should You Care?