Using PDM in readthedocs builds

[gjoseph92]

I'm trying to switch a poetry-based library to PDM, but having a hard time getting PDM to install dependencies into the virtualenv that readthedocs creates and uses automatically.

Note that I don't want to just do:

python:
  install:
    - method: pip
      path: .

because a) I want to install dependencies from the lockfile, and b) I want to install specific groups of dev dependencies (pdm install -dG docs).

A few things that might be causing problems:

• Readthedocs doesn't seem to set the VIRTUAL_ENV environment variable that PDM is looking for, so it doesn't detect that it's in already in a virtualenv and tries to use PEP 582 mode. Output of env:

  HOSTNAME=build-18005315-project-645352-stackstac
  HOME=/home/docs
  NO_COLOR=1
  READTHEDOCS=True
  READTHEDOCS_PROJECT=stackstac
  PATH=/home/docs/checkouts/readthedocs.org/user_builds/stackstac/envs/169/bin:/home/docs/.asdf/shims:/home/docs/.asdf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  READTHEDOCS_VERSION_TYPE=external
  LANG=C.UTF-8
  READTHEDOCS_LANGUAGE=en
  DEBIAN_FRONTEND=noninteractive
  READTHEDOCS_VERSION_NAME=169
  READTHEDOCS_VERSION=169
  PWD=/home/docs/checkouts/readthedocs.org/user_builds/stackstac/checkouts/169
  

• I tried pdm use $(which python) to manually override this, but the path to the interpreter is a symlink, which it seems that PDM resolves right now Allow symlinks in the list of pdm use #1351?

  which python
  /home/docs/checkouts/readthedocs.org/user_builds/stackstac/envs/169/bin/python
  
  realpath $(which python)
  /home/docs/.asdf/shims/python
  
  pdm use $(which python)
  Using Python interpreter: /home/docs/.asdf/installs/python/3.8.13/bin/python (3.8)
  

  I don't quite understand why passing an explicit path to pdm use doesn't make it use exactly that path.

Is there a recommended way to do this with PDM right now? (When there is, might be a good thing to put in the docs too.)

[frostming]

Using Python interpreter: /home/docs/.asdf/installs/python/3.8.13/bin/python (3.8)

This is just a display issue, verify the python path by pdm info.

[gjoseph92]

which python
/home/docs/checkouts/readthedocs.org/user_builds/stackstac/envs/169/bin/python

pdm use $(which python)
Using Python interpreter: /home/docs/.asdf/installs/python/3.8.13/bin/python (3.8)

pdm info
PDM version:
  2.1.3
Python Interpreter:
  /home/docs/.asdf/installs/python/3.8.13/bin/python (3.8)
Project Root:
  /home/docs/checkouts/readthedocs.org/user_builds/stackstac/checkouts/169
Project Packages:
  /home/docs/checkouts/readthedocs.org/user_builds/stackstac/checkouts/169/__pypackages__/3.8

[gjoseph92]

This seems to be working for PDM 2.1.3:

version: 2

sphinx:
  configuration: docs/conf.py

build:
  os: ubuntu-20.04
  tools:
    python: "3.8"
  jobs:
    post_install:
      - pip install --upgrade pdm
      - VIRTUAL_ENV=$(dirname $(dirname $(which python))) pdm install -dG docs

The VIRTUAL_ENV=$(dirname $(dirname $(which python))) is a bit of hack to force this code path.

[frostming]

After debugging a bit, I found something weird about the readthedocs env, see https://readthedocs.org/projects/stackstac-fork/builds/18015650/

1. The path to the virtualenv isn't added to the PATH.
2. The which command seems to be patched or wrapped so it can find the executable correctly. What is weird is that if I put it after a && or || operator, it fallbacks to the normal behavior.
3. However, $(which python) can't find executables under venv bin path.
4. So as you can see, which python and echo $(which python) output differently! which python gives you the venv python, while pdm use $(which python) uses the asdf python. The which command is not reliable.

[frostming]

The which command seems to be patched or wrapped so it can find the executable correctly

In fact, every leading command has been replaced with a wrapper, so pdm install works but echo yes && pdm install can't find the command pdm.

[gjoseph92]

Thanks for looking into that. Yeah, clearly readthedocs is doing something weird. And apparently, you shouldn't actually use which in most cases: https://unix.stackexchange.com/a/85250. Glad to know PDM wasn't doing anything wrong here.

[gjoseph92]

@frostming still, it's annoying that the only way to get PDM to use virtualenv mode in this case was to set the hacky VIRTUAL_ENV=$(dirname $(dirname $(which python))) environment variable.

It would be nice to have a CLI flag to force PDM to use venv mode and install packages into the current Python environment, without having to use heuristics like this.

[frostming]

It would be nice to have a CLI flag to force PDM to use venv mode and install packages into the current Python environment, without having to use heuristics like this.

It does have, but given the hack made by readthedocs there is no way for PDM to correctly detect the current Python environment. No VIRTUAL_ENV, no PATH manipulated.

P.S. Maybe you can manually build the path to the python interperter.