I've been told that `python` package management is bad. I have seen some really bad practices online, asking you to run commands here and there without an understanding of the bigger picture, what they do and sometimes with escalated privileges.
Along the years, I have compiled a list of practices I follow, and a list of tools I use. I hope to be able to share some of the knowledge I've acquired and show you a different way of doing things. You might learn about a new tool, or a new use for a tool. Come along for the ride !
<!--more-->
## Python {#python}
As most know, [Python](https://www.python.org/) is an interpreted programming language. I am not going to go into the details of the language in this post, I will only talk about management.
If you want to develop in Python, you need to install libraries. You can find _some_ in your package manager but let's face it `pip` is your way.
The majority of _Linux_ distributions will have Python installed as a lot of system packages now rely on it, even some package managers.
Okay, this is the last time I actually use the system's Python. What ? Why ? You ask !
## pyenv {#pyenv}
I introduce you to [pyenv](https://github.com/pyenv/pyenv). Pyenv is a Python version management tool, it allows you to install and manage different versions of Python as a _user_.
Beautiful, music to my ears.
Let's get it from the package manager, this is a great use of the package manager if it offers an up to date version of the package.
```bash
sudo pacman -S pyenv
```
If you're not using an _Archlinux_ based distribution follow the instructions on their [webpage](https://github.com/pyenv/pyenv#installation).
Alright ! Now that we've got ourselves pyenv, let's configure it real quickly.
Following the docs, I created `~/.config/fish/config.d/pyenv.fish` and in it I put the following.
```fish
# Add pyenv executable to PATH by running
# the following interactively:
set -Ux PYENV_ROOT $HOME/.pyenv
set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths
# Load pyenv automatically by appending
# the following to ~/.config/fish/config.fish:
status is-login; and pyenv init --path | source
```
Open a new shell and you're all ready to continue along, you're all locked, loaded and ready to go!
### Setup the environment {#setup-the-environment}
This is the first building block of my environment. We first start by querying for Python versions available for us.
```bash
pyenv install --list
```
Then, we install the latest Python version. Yes, even if it's an upgrade, I'll handle the upgrade, as well, as we go along.
Set everything up to use the new installed version.
First, we set the global Python version for our _user_.
```bash
pyenv global 3.9.5
```
Then, we switch our current shell's Python version, instead of opening a new shell.
```bash
pyenv shell 3.9.5
```
That was easy. We test that everything works as expected by checking the version.
```bash
pyenv version
```
Now, if you do a `which` on the `python` executable, you will find that it is in the `pyenv` shims' directory.
### Upgrade {#upgrade}
In the **future**, the upgrade path is exactly the same as the setup path shown above. You query for the list of Python versions available, choose the latest and move on from there.
Very easy, very simple.
## pip {#pip}
[pip](https://pypi.org/project/pip/) is the package installer for Python.
At this stage, you have to understand that you are using a Python version installed by _pyenv_ as your _user_. The pip provided, if you do a `which`, is also in the same shims directory.
Using `pip` at this stage as a _user_ is better than running it as _root_ but it is also not touching your system; just your user. We can do **one** better. I'm going to use `pip` as a _user_ once !
I know, you will have a lot of questions at this point as to why. You will see, patience is a virtue.
## pipx {#pipx}
Meet [pipx](https://github.com/pypa/pipx), this tool is the **amazing** companion for a _DevOps_, and _developer_ alike. Why ? You would ask.
It, basically, creates Python _virtual environments_ for packages you want to have access to _globally_. For example, I'd like to have access to a Python **LSP** server on the go.
This way my text editor has access to it too and, of course, can make use of it freely. Anyway, let's cut this short and show you. You will understand better.
Let's use the only `pip` command as a _user_ to install `pipx`.
```bash
pip install --user pipx
```
<divclass="admonition warning">
<pclass="admonition-title">warning</p>
You are setting yourself up for a **world of hurt** if you use `sudo` with `pip` or run it as `root`. **ONLY** run commands as `root` or with escalated privileges when you know what you're doing.
</div>
### LSP Server {#lsp-server}
As I gave the **LSP** server as an example, let's go ahead and install it with some other Python packages needed for global things like _emacs_.
```bash
pipx install black
pipx install ipython
pipx install isort
pipx install nose
pipx install pytest
pipx install python-lsp-server
```
Now each one is in it's own happy little _virtual environment_ separated from any other dependency but its own. Isn't that lovely ?
If you try to run `ipython`, you will see that it will actually work. If you look deeper at it, you will see that it is pointing to `~/.local/bin/ipython` which is a symlink to the actual package in a _pipx__virtual environment_.
### Upgrade {#upgrade}
After you **set** a new Python version with _pyenv_, you simply reinstall everything.
```bash
pipx reinstall-all
```
And like magic, everything get recreated using the new version of Python _newly_ set.
## virtualfish {#virtualfish}
Now that _pipx_ is installed, let's go head and install something to manage our Python _virtual environments_ on-demand, for use whenever we need to, for targeted projects.
Some popular choices people use are [Pipenv](https://pipenv.pypa.io/en/latest/), [Poetry](https://python-poetry.org/), [virtualenv](https://virtualenv.pypa.io/en/latest/) and plain and simple python with the `venv` module.
You're welcome to play with all of them. Considering I use _fish_ as my default _shell_, I like to use [virtualfish](https://virtualfish.readthedocs.io/en/latest/).
Let's install it.
```bash
pipx install virtualfish
```
This offers me a new command; `vf`. With `vf`, I can create Python _virtual environments_ and they will all be saved in a directory of my choosing.
### Setup {#setup}
Let's create one for [Ansible](https://docs.ansible.com/ansible/latest/index.html).
```bash
vf new ansible
```
This should **activate** it. Then, we install _Ansible_.
```bash
pip install ansible molecule docker
```
At this stage, you will notice that you have `ansible` installed. You will also notice that all the _pipx_ packages are also still available.
If you want to tie _virtualfish_ to a specific directory, use `vf connect`.
### Upgrade {#upgrade}
To _upgrade_ the Python version of all of our _virtual environments_, _virtualfish_ makes it as easy as
```bash
vf upgrade
```
And we're done !
## Workflow {#workflow}
At this stage, you have an idea about the tools I use and where their scope falls. I like them because they are _limited_ to their own scope, each has its own little domain where it reigns.
- I use **pyenv** to install and manage different versions of Python for testing purposes while I stay on the latest.
- I use **pipx** for the commands that I need access to _globally_ as a user.
- I use **virtualfish** to create one or more _virtual environment_ per project I work on.
With this setup, I can test with different versions of Python by creating different _virtual environments_ with different version each, or two versions of the tool you're testing as you keep the Python version static.
It could also be different versions of a library, testing forward compatibility for example.
At each step, I have an upgrade path to keep all my environments running the latest versions. I also have a lot of flexibility by using `requirements.txt` files and others for _development_ sometimes or even _testing_.
## Conclusion {#conclusion}
As you can see, with a little bit of knowledge and by standing on the shoulders of giants, you can easily manage a Python environment entirely as a _user_.
You have full access to a wide array of Python distributions to play with. Endless different versions of packages, _globally_ and _locally_ installed.
If you create _virtual environments_ for each of your projects, you won't fall in the common pitfalls of versioning hell.
Keep your _virtual environments_ numerous and dedicated to projects, small sets, and you won't face any major problems with keeping your system clean yet up to date.
