How to Fix ‘Could not build wheels for sentencepiece’ pip Error in Python

The pip command makes it a breeze to install Python packages. If you work with NLP, chances are you’ve used one such package called sentencepiece which is a popular tokenizer — a tool central to Natural Language Processing (NLP).

However, users occasionally experience impediments while installing Python packages which are mainly due to dependency issues. For example, it could be that a package is dependent on a specific version of another package, and that the latter itself is not compatible with some versions of Python and so on. This situation is called dependency hell and it can be frustrating to find a way out.

Backstory

This is what happened to me yesterday when I tried to install sentencepiece. First, I created a separate Python environment for my specific project because it’s NEVER a good idea to install packages globally on your system in order to avoid issues later down the road.

cd ~
python -m venv my_venv

Once the environment is ready, it needs to be activated using the provided script.

source ~/my_venv/bin/activate

At this point, any execution of Python inside this Terminal session should relate to this “active” environment. Next, it is important to update pip itself before installing the actual sentencepiece package.

pip install --upgrade pip
pip install sentencepiece

However, contrary to our expectation, the installation fails with the following error message:

subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.99']' returned non-zero exit status 127.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
  Running setup.py clean for sentencepiece
Failed to build sentencepiece
ERROR: Could not build wheels for sentencepiece, which is required to install pyproject.toml-based projects

The actual trace is much longer than this excerpt, but luckily the final part contains some clues to guide us in the right direction. After a quick glance at the log, we can easily rule out any issues with pip itself. The error indicates that the installation script failed because it "could not build wheels for sentencepiece".

Naturally, I installed the required packages while the Python Virtual Environment is active and tried installing sentencepiece again.

pip install --upgrade wheel setuptools
pip list
pip install sentencepiece

While the dependencies wheel and setuptools got installed, the installation for sentencepiece still failed. Therefore, I concluded that the issue should be coming from Python itself.

Solution

Hence, I experimented with older versions of Python until I found one that worked. Below is a complete solution which I highly recommend to follow through in case you might have some unknown missing system packages.

Step 1: Install yay, Compiler Tooling, and Python Tools

For Manjaro and Arch-Linux derivatives:

sudo pacman -Syu
sudo pacman -S --needed yay base-devel cmake
sudo pacman -S --needed python-setuptools python-wheel

For Ubuntu and Debian derivatives:

sudo apt update && sudo apt upgrade
sudo apt install yay build-essential cmake
sudo apt install python-setuptools python-wheel

Step 2: Install an Older Version of Python (v3.10) Using yay

yay -S python310

Step 3: Create a New Python Virtual Environment Using Python 3.10

python3.10 -m venv ~/my_venv_310
source ~/my_venv_310/bin/activate

Step 4: Install sentencepiece and Dependencies

pip install --upgrade pip wheel setuptools
pip install sentencepiece

Conclusion

The above steps should fix your sentencepiece installation issues by ensuring you have all the required dependencies and a proper version of Python on which sentencepiece is supported.

# Footnotes

https://unix.stackexchange.com/questions/788591/how-to-force-install-specific-version-of-python-with-pacman-arch