Clarification regarding PEP 668?

[ckp95]

https://peps.python.org/pep-0668

Distributions may opt-in to PEP 668, to mark the system Python as externally managed. Debian already has. I think this means pipx's current Linux installation instructions won't work:

On Linux, install via pip (requires pip 19.0 or later)

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Upgrade pipx with python3 -m pip install --user --upgrade pipx

The distro I'm on right now hasn't opted in so I can't test it myself.

Furthermore, the PEP itself recommends distro maintainers include pipx in their repositories and ship it along with Python:

Consider packaging pipx, a tool for installing Python-language applications, and suggesting it in the error. pipx automatically creates a virtual environment for that application alone, which is a much better default for end users who want to install some Python-language software (which isn’t available in the distro) but are not themselves Python users. Packaging pipx in the distro avoids the irony of instructing users to pip install --user --break-system-packages pipx to avoid breaking system packages. Consider arranging things so your distro’s package / environment for Python for end users (e.g., python3 on Fedora or python3-full on Debian) depends on pipx.

So I think the installation instructions should instead say that

1. the user may already have pipx installed via the distro's package manager, in which case they don't need to do anything
2. if they don't, they should install pipx by the distro's package manager if it happens to be available
3. if it's not available, to make a virtual environment for pipx and install it there.

Some people have already run into this confusion, e.g. here.

[dukecat0]

See #972

[TomiBelan]

Hi. I just found pipx. Nice project!

I agree this is a problem. The PEP 668 authors have good reasons to forbid "sudo pip install" and "pip install --user", but it's harder to install pipx itself. At least I'm glad they're aware of the irony.

My recommendation for users installing pipx today:

1. If pipx is already installed, you may not need to do anything.
2. If you have root on this computer, and your Linux distro has a pipx package, and it's recent enough for you, use that.
3. If you don't have root, or you want the latest pipx version, run python3 -m pip install --user --break-system-packages pipx. Don't worry - the chance this actually breaks some system package is very small.

The main reason why the last step is different than @ckp95's suggestion is that I wanted to keep it simple and limit it to only 1 command. This recipe is not the ideal final state, just a pragmatic compromise for the time being.

How could "pip install --user" break anything, anyway? What's the problem?

Here's an example with the magic-wormhole tool and the click library. Suppose you're still using Ubuntu 18.04 (bionic). You run sudo apt install python3 python3-pip python3-venv magic-wormhole. It installs magic-wormhole 0.10.3 and its dependency python3-click 6.7 in /usr/, among other things. You run python3 -m pip install --user pipx. It installs pipx 1.0.0 and its dependency click 8.0.4 in your $HOME, among other things. You run wormhole. It was supposed to use click 6.7, but actually it'll use click 8.0.4 from your home! A different major version! Oh no!

This is just a theoretical example because click 6.x and click 8.x are quite compatible in practice. I couldn't find any specific practical breakage for pipx and its dependencies. That is why I say the chance is very small.

By the way, pipx doesn't even use click directly. Only userpath/cli.py needs it.

How to install pipx in a venv, if you really want to:

@ckp95 didn't write the full details. Here's what I came up with.

rm -rf ~/.local/pipx/self
python3 -m venv --upgrade-deps ~/.local/pipx/self
~/.local/pipx/self/bin/pip install pipx
mkdir -p ~/.local/bin
ln -s ~/.local/pipx/self/bin/{pipx,activate-global-python-argcomplete,python-argcomplete-check-easy-install-script,register-python-argcomplete,userpath} ~/.local/bin

That last line is the list of all bin scripts that would be created by pip install --user pipx.

Is it OK to install pipx in a venv?

I noticed that when pipx is installed in a venv, it uses a different default interpreter for the venvs it creates. E.g. pipx run --verbose pycowsay hello prints: pipx >(setup:802): Default python interpreter is '/home/user/.local/pipx/self/bin/python3'. When you install pipx normally, it's just '/usr/bin/python3'. The path is stored in pyvenv.cfg of created venvs. I'm a bit worried about this. Is this really OK? It seems to work, but could it cause some subtle issues in the future?

Possible dirty hack to change the default interpreter: edit .local/pipx/self/bin/pipx, replace the first line with #!/usr/bin/python3, and add "if __name__ == '__main__': sys.path[0] = '/home/user/.local/pipx/self/lib/python3.11/site-packages'" just below import sys.

Maybe pipx should add some special code to pipx/interpreter.py to handle this case. If pipx notices that it was installed in a venv, use os.readlink(sys.executable). (Or os.path.realpath. FWIW that's '/usr/bin/python3.11'.) I'm not sure about the details.

I think pipx should have an install script.

Not everyone is a fan of "curl | sh", but it might just be the best long term solution here. At least for Linux (and perhaps macOS).

In a perfect world, the install script should also:

• Detect popular Linux distros and interactively ask the user if they want to install the distro pipx package instead.
• Seamlessly upgrade existing users who previously installed pipx with pip install --user pipx.
• Have configurable install paths (e.g. systemwide in /usr/local/bin and /opt/pipx/self).
• Allow installing development versions of pipx with --pre or a custom spec.
• Interactively offer to run pipx ensurepath if bin is not in $PATH.
• Be very verbose about everything it's doing.

Well, maybe one day.

Tell me what you think!

[chrysle]

Hi @TomiBelan,

there is an approach for such an install script in #849, however lately focus has shifted to shipping a PEX file. Discussion seems to have ceased though.

Your suggestions sound reasonable to me and I'm sure it would be welcomed if you overtook development!

• Detect popular Linux distros and interactively ask the user if they want to install the distro pipx package instead.

Though, issues have been reported lately concerning distro-provided packages.

[uranusjr]

Instead of trying to wrangle out of the restriction, I’d rather just encourage any distributions enabling EXTERNALLY-MANAGED to start distributing pipx in their native package manager. That is the goal of PEP 668 to begin with. My opinion (as both a maintainer of this project and an author of the PEP) is that pipx should not recommend anything toward this—aside from simply not do it—but to ask the user to encourage their platform of choice (whatever Linux distribution they use, in practice) to start distributing pipx.

[chrysle]

But it must be considered that generally packages installed from the distro's package repositories receive less updates (e. g. are more likely affected by in upstream already fixed issues, or lack added features, which have been released to PyPI). There is a rather small community of downstream packagers, and the process of re-packaging/updating is costly. Which is why the release of new packages is often limited to stable/major versions. Take for example packaging in the Ubuntu package repository, or tox, which are hopelessly outdated.

At least this is what I experienced with Linux systems, and consequently prefer to use pip. Are you generally opposed to adding an install script?

[TomiBelan]

Thanks for your comments and links!

I assume pex is not relevant anymore. The discussion in #849 about pex happened before PR #895 by @dukecat0 added a pipx.pyz file built with shiv. pex and shiv are just two different tools for building a python zipapp (.pyz), right? For the pipx use case they look mostly equivalent.

I do agree that installing pipx from the native distro package manager should be the primary encouraged way. My view is: "If you have root on this computer, and your Linux distro has a pipx package, and it's recent enough for you, use that." But I think the "if you don't have root, or you want the latest pipx version" use case is also important.

But, if you think this will be a minority of users, and hence an install script is overkill, I can understand that.

If not an install script, I suggest at least documenting this case with something like this:

mkdir -p ~/.local/bin
curl -fL https://github.com/pypa/pipx/releases/latest/download/pipx.pyz -o ~/.local/bin/pipx
chmod a+x ~/.local/bin/pipx
~/.local/bin/pipx ensurepath

Would you like me to send it as a documentation PR?

[uranusjr]

A note in documentation on the pyz installation method sounds good to me. Some additional notes may be needed for environments that have python pointing to an unsupported Python version as well.

[TomiBelan]

I did more testing of pipx.pyz and I have some concerns, especially #1074. tl;dr: if you run pipx.pyz while an unrelated venv is active, the supposedly isolated venvs created by pipx.pyz can silently depend on it, and may break when the unrelated venv is deleted.

Personally I think I'll avoid using pipx.pyz for now. And I won't make a docs PR to recommend it. But I could be overblowing it -- feel free to do it if you disagree.

[uranusjr]

I’d recommend setting PIPX_PYTHON instead. Personally I feel this is the best practice anyway, even in some other cases e.g. Homebrew, since the underlying Python installation can be somewhat volatile.

[gaborbernat]

A note in documentation on the pyz installation method sounds good to me. Some additional notes may be needed for environments that have python pointing to an unsupported Python version as well.
I’d recommend setting PIPX_PYTHON instead. Personally I feel this is the best practice anyway, even in some other cases e.g. Homebrew, since the underlying Python installation can be somewhat volatile.

PR welcome.